Document Metadata

For each document in the corpus, we tried to obtain the following metadata:

• spatial scale
• temporal scale
• country
• funding source

Spatial Scale

After careful investigation of the topics resulting from the LDA, we found that the spatial scale as well as the components of the water budget can both be obtained by using the LDA topics.

Temporal Scale

Problem Definition

The problem at hand is a classification task. Classification can have multiple labels or one label with different values. In a multilabel classification, documents have the spatial scales labels: "Day" and "Week" and "Year" at the same time. A multi-class example for the current problem is one label "Timescale" with values "Day", "Week", "Year" and so on. Multi-class problems are simpler than multi-labels problem.

Two main approaches exist to solve such a multi-label classification problem. The first one is to use algorithm adapted for such a task. The second one is to transform the problem to make it tractable. Such a transformation can be achieved through two schemes: Binary Relevance (BR) and Label Powerset (LP). In the LP approach, the multilabel problem is transformed into a multi-class classification of the labelsets. In the BR approach, the problem is decomposed in a set of binary problems. In addition, we tested more refined BR approaches: classifier chains were the binary classifiers are trained in a given order to predict the labels; nested stacking were the labels are approximated successively by learners and dependent binary relevance where every other labels are used as the input for each label-specific binary learners and stacking which combine dependent binary relevance and nested stacking.

The main performance metric used is the multilabel Hamming loss corresponding to the proportion of incorrectly predicted labels. We also use the subset 0 1 metric corresponding to the proportion of observations for which the complete set of label is correctly predicted.

Results

For deriving the spatial scale metadata we initially tried the following approaches:

labels	input	model
multilabel	DTM from cleaned corpus	algorithm adaptation
multilabel	LDA topics membership	algorithm adaptation
multilabel	DTM from cleaned corpus	binary relevance (and variations)
multilabel	LDA topics membership	binary relevance (and variations)
multilabel	DTM from cleaned corpus	label powerset
multilabel	LDA topics membership	label powerset

As these approaches yielded poor performance, we simplified the labels into three labels corresponding to short term, long term and very long term.

labels	input	model
multilabel (simple)	DTM from cleaned corpus	algorithm adaptation
multilabel (simple)	LDA topics membership	algorithm adaptation
multilabel (simple)	DTM from cleaned corpus	binary relevance (and variations)
multilabel (simple)	LDA topics membership	binary relevance (and variations)
multilabel (simple)	DTM from cleaned corpus	label powerset
multilabel (simple)	LDA topics membership	label powerset
multiclass	DTM from cleaned corpus	algorithm adaptation

As these approaches led to poor performance, we tried to use the latest deep-learning model for natural language processing released by Google: BERT.

labels	input	model
multilabel	raw corpus	Google's BERT
multilabel (simple)	raw corpus	Google's BERT

The model performance remained poor. We hypothetized that this was due to the noise present in the raw corpus: email addresses, formatting from journals, etc. The next iteration of models was thus run on abstracts both for previous traditional approaches and BERT.

labels	input	model
multilabel (simple)	DTM from abstracts	algorithm adaptation
multilabel (simple)	DTM from abstracts	binary relevance (and variations)
multilabel (simple)	raw abstract	Google's BERT
multiclass	raw abstract	Google's BERT

Summary

The poor results of the different approaches applied point toward a rather difficult issue to assign the temporal scales of a study solely based on text-mining techniques or on deep NLP models. This is likely due because the spatial scale information is often tenuous in most studies and because the training set is at least two orders of magnitude lower than what is usually used in the deep NLP literature.

Country Location

Funding Source