-
Notifications
You must be signed in to change notification settings - Fork 0
Topic Modelling
We conducted an unprecedented literature review of over 30,000 multilingual articles on water resources research through the use of Latent Dirichlet Allocation (LDA, [Blei et al., 2003]). LDA is a state-of-the-art document classification method relying on modelling the topics occurring within a corpus. In the following, we detail (i) the process of corpus collection; (ii) the preprocessing steps required by LDA; and (iii) the LDA algorithm itself.
A quality assessment of the corpus collection was then performed to assess possible bias between query results and PDF retrieval and against distinct sources or time periods. We define the within-corpus bias as the difference in relative frequencies between query and corpus such that:
B(Year) = ∆[f query (Year) − f corpus (Year)] B(Source) = ∆[f query (Source) − f corpus (Source)]
To address the possibility that this automated approach could result in between-corpus bias in terms of return rate, we augmented the corpus by manually downloading articles. One well-behaved corpus was defined as reference based on its quality analysis. Ill-behaved corpuses were then adjusted to reach similar return rates. This adjustment reduced both within- and between-corpus bias by using a systematic sampling with unequal inclusion probabilities of the p_i [Madow, 1949], where p_i was defined using the normalized product of the bias functions and i refers to the i-th non-retrieved article.
Each language corpus was amended to correct for bias in year and presence of DOI. Additional articles were manually downloaded to get closer to the English return rate.
1,158 papers were identified to increase the return rate of Portuguese and Spanish corpuses. During a download party, the Portuguese papers were found at a return fate of 77%, and Spanish papers were found at a return rate of 80%. The final return rate between query and corpus:
- English : 72%
- Portuguese : 63%
- Spanish: 56%
The final number of articles:
- English : 21,197
- Portuguese: 486 (downloaded 186)
- Spanish : 794 (downloaded 672)
300 additional papers would be needed to downloaded and added to the corpuses to reach a consistent 72% return rate between the languages - we chose not to proceed and move forward with the corpus.
Once the full corpus of PDF articles from English, Spanish, and Portuguese was obtained, a cleaning process was performed to prepare the text for LDA. First, the text was converted into a standardized version in which all cases were lowered, words were checked against a dictionary (and removed if not found), and special patterns for words such as emails and URLs were assigned tags. Additional cleaning removed punctuation (except for apostrophes and hyphens inside words) and single letter words, resulting in a cleaned, standardized version of the text. Finally, lemmatization was performed using TreeTagger to reduce all words to their base forms by changing all nouns into singular form and all verbs into the infinite tense. TreeTagger is a software tool developed at the Institute for Computational Linguistics of the University of Stuttgart, which tags text with lemma information and is compatible with a wide variety of languages including English, Spanish, and Portuguese (Schmidt 1995). This final version of each corpus, with lemmatization, was then used as the input for LDA.
Latent Dirichlet Allocation is a Bayesian, generative, probabilistic model conceptualizing each item in a corpus of documents as a random mixture of topics [Blei et al., 2003]. Topics are Bayesian bag-of-words corresponding to a distribution over the vocabulary (i.e. the words appearing within the corpus). In essence, one topic corresponds to words that have a significant probability to co-occur. LDA being a Bayesian method, its parameters are tuned using a Markov Chain Monte-Carlo such as the well-known Metropolis Hastings algorithm (MCMC-MH, [Hastings, 1970]).
The LDA was programed to identify a specific number of topics for each language's corpus based on xxxx. The LDA identified 105 topics in the English corpus, 65 topics in the Spanish corpus, and 65* topics in the Portuguese corpus.
Topic modeling was validated through cross-validation with a training set of human-derived topics for the English Corpus: 1 000 papers were human-read and manually coded for the presence of keywords and main research topics.
The topics from all 3 corpuses were reviewed by experts and it was determined that the LDA successfully identified topics based on the top ~10 occurring words. In addition, as topics are statistical objects, a human label has to be applied to them to make them tractable. In consequence, we evaluate both the significance and label of each topic using expert knowledge. Each topic was labeled individually by research team members, then harmonized by
- specific topic name
- theme
- specific and broad categories of scientific research as defined by NSF
- description: spatial scale, water budget, or methods
The Spanish and Portuguese Corpuses were reviewed by experts to determine if they behave similarly to the English corpus. A cross-comparison of LDAs determined that the NSF specific category is the appropriate level to compare the performance of LDAs.
- Spanish LDA stats
- 52 TRUE topics
- 33 themes shared with english corpus
- 4 methods
- 11 spatial scales (8 are locations specific)
- 3 water budgets
- Portuguese LDA stats
- 32 TRUE topics
- 19 themes shared with english corpus
- 1 method
- 7 location specific
- 3 water budget To further constrain our analysis, additional text mining was performed to extract funding information, location as well as scale and temporal scale.
Filter is built using the DFM only and not the topicDocs as they lead to overfit and reduce TPR.
Tuning highlight that mtry = 6 leads to best result.