-
Notifications
You must be signed in to change notification settings - Fork 0
Topic Modelling
We conducted an unprecedented literature review of over 30,000 multilingual articles on water resources research through the use of Latent Dirichlet Allocation (LDA, [Blei et al., 2003]). LDA is a state-of-the-art document classification method relying on modelling the topics occurring within a corpus. In the following, we detail (i) the process of corpus collection; (ii) the preprocessing steps required by LDA; and (iii) the LDA algorithm itself.
A quality assessment of the corpus collection was then performed to assess possible bias between query results and PDF retrieval and against distinct sources or time periods. We define the within-corpus bias as the difference in relative frequencies between query and corpus such that:
B(Year) = ∆[f query (Year) − f corpus (Year)] B(Source) = ∆[f query (Source) − f corpus (Source)]
To address the possibility that this automated approach could result in between-corpus bias in terms of return rate, we augmented the corpus by manually downloading articles. One well-behaved corpus was defined as reference based on its quality analysis. Ill-behaved corpuses were then adjusted to reach similar return rates. This adjustment reduced both within- and between-corpus bias by using a systematic sampling with unequal inclusion probabilities of the p_i [Madow, 1949], where p_i was defined using the normalized product of the bias functions and i refers to the i-th non-retrieved article.
Lack of DOI in many Spanish and Portuguese articles led to skewed corpus collection. Each language corpus was amended to reflect a 72% return rate between query and corpus. Additional articles were manually downloaded.
The initial return rate between query and corpus:
- English : 72%
- Portuguese : 63%
- Spanish: 56%
The final number of articles to raise each return rate to 72%:
- English : 21,197
- Portuguese: 486 (downloaded 186)
- Spanish : 794 (downloaded 672)
Once the full corpus of PDF articles from English, Spanish, and Portuguese was obtained, a cleaning process was performed to prepare the text for LDA. First, the text was converted into a standardized version in which all cases were lowered, words were checked against a dictionary (and removed if not found), and special patterns for words such as emails and URLs were assigned tags. Additional cleaning removed punctuation (except for apostrophes and hyphens inside words) and single letter words, resulting in a cleaned, standardized version of the text. Finally, lemmatization was performed using TreeTagger to reduce all words to their base forms by changing all nouns into singular form and all verbs into the infinite tense. TreeTagger is a software tool developed at the Institute for Computational Linguistics of the University of Stuttgart, which tags text with lemma information and is compatible with a wide variety of languages including English, Spanish, and Portuguese (Schmidt 1995). This final version of the corpus, with lemmatization, was then used as the input for LDA.
Latent Dirichlet Allocation is a Bayesian, generative, probabilistic model conceptualizing each item in a corpus of documents as a random mixture of topics [Blei et al., 2003]. Topics are Bayesian bag-of-words corresponding to a distribution over the vocabulary (i.e. the words appearing within the corpus). In essence, one topic corresponds to words that have a significant probability to co-occur. LDA being a Bayesian method, its parameters are tuned using a Markov Chain Monte-Carlo such as the well-known Metropolis Hastings algorithm (MCMC-MH, [Hastings, 1970]).
Topic modeling was validated through cross-validation with a training set of human-derived topics: 1 000 papers were human-read and manually coded for the presence of keywords and main research topics.
In addition, as topics are statistical objects, to make them tractable a human label has to be applied to them. In consequence, we evaluate both the significance and label of each topic using expert knowledge and a panel of non-expert working blindly evaluating the topics. To further constrain our analysis, additional text mining was performed to extract funding information, location as well as scale and temporal scale.