Corpus Cleaning Process

The steps below outline the process to prepare corpuses of PDF documents into machine-readable, cleaned text to use as an input for LDA analysis. Starting with a corpus of individual PDF files, the following steps are performed:

1. Convert PDF document files into text files. ABBYY FineReader software was used for this step.
2. Remove punctuation and numbers from text, with the exception of hyphens inside compound words.
3. Remove capitalizations
4. Remove single letters and "stop words", which are frequently used words (such as "the", "is", or "but") that are not considered meaningful to text analysis. Stop words were selected using language-specific lists commonly used in text analysis (using the list from Snowball software), with extra words added if considered helpful.
5. Check all remaining words against a language-specific dictionary, and remove words not matched in the dictionary. [definitely done for english, double-checking this was used for sp/pt]
6. Lemmatize the remaining text by converting all nouns to singular form and all verbs to infinite tense.

The resulting cleaned corpuses in English, Spanish, and Portuguese were then ready to analyze with LDA.