Document Metadata

For each document in the corpus, we tried to obtain the following metadata:

• spatial scale
• temporal scale
• country
• funding source

Spatial Scale

After careful investigation of the topics resulting from the LDA, we found that the spatial scale as well as the components of the water budget can both be obtained by using the LDA topics.

Temporal Scale

Problem Definition

The problem at hand is a classification task. Classification can have multiple labels or one label with different values. In a multilabel classification, documents have the spatial scales labels: "Day" and "Week" and "Year" at the same time. A multi-class example for the current problem is one label "Timescale" with values "Day", "Week", "Year" and so on. Multi-class problems are simpler than multi-labels problem.

Two main approaches exist to solve such a multi-label classification problem. The first one is to use algorithm adapted for such a task. The second one is to transform the problem to make it tractable. Such a transformation can be achieved through two schemes: Binary Relevance (BR) and Label Powerset (LP). In the LP approach, the multilabel problem is transformed into a multi-class classification of the labelsets. In the BR approach, the problem is decomposed in a set of binary problems. In addition, we tested more refined BR approaches: classifier chains were the binary classifiers are trained in a given order to predict the labels; nested stacking were the labels are approximated successively by learners and dependent binary relevance where every other labels are used as the input for each label-specific binary learners and stacking which combine dependent binary relevance and nested stacking.

The main performance metric used is the multilabel Hamming loss corresponding to the proportion of incorrectly predicted labels. We also use the subset 0 1 metric corresponding to the proportion of observations for which the complete set of label is correctly predicted.

Results

For deriving the spatial scale metadata we initially tried the following approaches:

labels	input	model
multilabel	DTM from cleaned corpus	algorithm adaptation
multilabel	LDA topics membership	algorithm adaptation
multilabel	DTM from cleaned corpus	binary relevance (and variations)
multilabel	LDA topics membership	binary relevance (and variations)
multilabel	DTM from cleaned corpus	label powerset
multilabel	LDA topics membership	label powerset

As these approaches yielded poor performance, we simplified the labels into three labels corresponding to short term, long term and very long term.

labels	input	model
multilabel (simple)	DTM from cleaned corpus	algorithm adaptation
multilabel (simple)	LDA topics membership	algorithm adaptation
multilabel (simple)	DTM from cleaned corpus	binary relevance (and variations)
multilabel (simple)	LDA topics membership	binary relevance (and variations)
multilabel (simple)	DTM from cleaned corpus	label powerset
multilabel (simple)	LDA topics membership	label powerset
multiclass	DTM from cleaned corpus	algorithm adaptation

As these approaches led to poor performance, we tried to use the latest deep-learning model for natural language processing released by Google: BERT.

labels	input	model
multilabel	raw corpus	Google's BERT
multilabel (simple)	raw corpus	Google's BERT

The model performance remained poor. We hypothetized that this was due to the noise present in the raw corpus: email addresses, formatting from journals, etc. The next iteration of models was thus run on abstracts both for previous traditional approaches and BERT.

labels	input	model
multilabel (simple)	DTM from abstracts	algorithm adaptation
multilabel (simple)	DTM from abstracts	binary relevance (and variations)
multilabel (simple)	raw abstract	Google's BERT
multiclass	raw abstract	Google's BERT

Summary

The poor results of the different approaches applied point toward a rather difficult issue to assign the temporal scales of a study solely based on text-mining techniques or on deep NLP models. This is likely due because the spatial scale information is often tenuous in most studies and because the training set is at least two orders of magnitude lower than what is usually used in the deep NLP literature.

Country Location

As for the spatial scale metadata, the problem at hand is a classification task. Building on the experience from the spatial scale metadata, we initially tried the following approaches:

labels	input	model
multilabel	raw abstract	Google's BERT
multilabel	DTM from raw abstract	algorithm adaptation
multilabel	DTM from raw abstract	binary relevance (and variations)

The models performance was very limited. However, in contrast to spatial scale, we tried to extend the size of the training set by using the author-derived keywords presents in the Web of Science dataset. Using such author-derived keywords, we extended the number of observations from 1,500 to 3,800. Despite this significant increase in the training set, performance remained poor. For few examples, we could determined that it is likely that some of the abstract do not contain information about the location, especially when author precised the location in keywords.

Further, exploratory data analysis (EDA) revealed that observations with multilabel only represent 2% of the complete dataset. In consequence, the problem was recasted into a multi-class problem using the raw corpus rather than the abstracts.

labels	input	model
multiclass	DTM from raw corpus	algorithm adaptation
multiclass	DTM from raw corpus	binary relevance (and variations)

Nonetheless, these approach did not yield satisfactory results. Additional EDA underlined that part of the necessary information was absent from the training data. For example, the term mexico was absent from the TDM. The TDM was recomputed from the raw corpus with costumized cleaning to ensure that the correct data was left in the training data. This was performed by using a dictionnary of country names. In addition, we included additional term derived from a geographical database from Natural Earth Data. These data includes local administrative unit, glaciated area, rivers, lakes, dams, populated places and geographical regions (e.g. the Andes).

labels	input	model
multiclass	country count from DTM from raw corpus	random forest, knn, svm, ann, multinomial

Random Forest outperformed the other models. Furthermore, adding more geographical keywords led to significant over-fitting in the final trained model. A RF model was fine-tuned with grid search across 20 values of the mtry parameter. The best value was found to be 6.

Funding Source