-
Notifications
You must be signed in to change notification settings - Fork 0
Document Metadata
For each document in the corpus, we tried to obtain the following metadata:
- spatial scale
- temporal scale
- country
- funding source
After careful investigation of the topics resulting from the LDA, we found that the spatial scale as well as the components of the water budget can both be obtained by using the LDA topics.
The problem at hand is a classification task. Classification can have multiple labels or one label with different values. In a multilabel classification, documents have the spatial scales labels: "Day" and "Week" and "Year" at the same time. A multi-class example for the current problem is one label "Timescale" with values "Day", "Week", "Year" and so on. Multi-class problems are simpler than multi-labels problem.
Two main approaches exist to solve such a multi-label classification problem. The first one is to use algorithm adapted for such a task. The second one is to transform the problem to make it tractable. Such a transformation can be achieved through two schemes: Binary Relevance (BR) and Label Powerset (LP). In the LP approach, the multilabel problem is transformed into a multi-class classification of the labelsets. In the BR approach, the problem is decomposed in a set of binary problems. In addition, we tested more refined BR approaches: classifier chains were the binary classifiers are trained in a given order to predict the labels; nested stacking were the labels are approximated successively by learners and dependent binary relevance where every other labels are used as the input for each label-specific binary learners and stacking which combine dependent binary relevance and nested stacking.
The main performance metric used is the multilabel Hamming loss corresponding to the proportion of incorrectly predicted labels. We also use the subset 0 1 metric corresponding to the proportion of observations for which the complete set of label is correctly predicted.
For deriving the spatial scale metadata we initially tried the following approaches:
| labels | input | model |
|---|---|---|
| multilabel | DTM from cleaned corpus | algorithm adaptation |
| multilabel | LDA topics membership | algorithm adaptation |
| multilabel | DTM from cleaned corpus | binary relevance (and variations) |
| multilabel | LDA topics membership | binary relevance (and variations) |
| multilabel | DTM from cleaned corpus | label powerset |
| multilabel | LDA topics membership | label powerset |
As these approaches yielded poor performance, we simplified the labels into three labels corresponding to short term, long term and very long term.
| labels | input | model |
|---|---|---|
| multilabel (simple) | DTM from cleaned corpus | algorithm adaptation |
| multilabel (simple) | LDA topics membership | algorithm adaptation |
| multilabel (simple) | DTM from cleaned corpus | binary relevance (and variations) |
| multilabel (simple) | LDA topics membership | binary relevance (and variations) |
| multilabel (simple) | DTM from cleaned corpus | label powerset |
| multilabel (simple) | LDA topics membership | label powerset |
| multiclass | DTM from cleaned corpus | algorithm adaptation |
As these approaches led to poor performance, we tried to use the latest deep-learning model for natural language processing released by Google: BERT.
| labels | input | model | eval.loss | global_step | seq_length |
|---|---|---|---|---|---|
| multilabel | raw corpus | Google's BERT | 0.44 | 235 | 512 |
| multilabel | raw corpus | Google's BERT | 0.45 | 1178 | 512 |
| multilabel | raw corpus | Google's BERT | 0.87 | 2209 | 128 |
| multilabel (simple) | raw corpus | Google's BERT | 0.56 | 235 | 512 |
The model performance remained poor with a mean AUC of 0.61 and a loss of 0.43. We hypothetized that this was due to the noise present in the raw corpus: email addresses, formatting from journals, etc. The next iteration of models was thus run on abstracts both for previous traditional approaches and BERT.
| labels | input | model |
|---|---|---|
| multilabel (simple) | DTM from abstracts | algorithm adaptation |
| multilabel (simple) | DTM from abstracts | binary relevance (and variations) |
| multilabel (simple) | raw abstract | Google's BERT |
| multiclass | raw abstract | Google's BERT |
The poor results of the different approaches applied point toward a rather difficult issue to assign the temporal scales of a study solely based on text-mining techniques or on deep NLP models. This is likely due because the spatial scale information is often tenuous in most studies and because the training set is at least two orders of magnitude lower than what is usually used in the deep NLP literature.
As for the spatial scale metadata, the problem at hand is a classification task. Building on the experience from the spatial scale metadata, we initially tried the following approaches:
| labels | input | model |
|---|---|---|
| multilabel | raw abstract | Google's BERT |
| multilabel | DTM from raw abstract | algorithm adaptation |
| multilabel | DTM from raw abstract | binary relevance (and variations) |
Aggregated Result:
multilabel.randomForestSRC
multilabel.hamloss.test.mean=0.0498778
multilabel.subset01.test.mean=0.9455497
multilabel.acc.test.mean=0.0544503
multilabel.tpr.test.mean=0.0484706
multilabel.ppv.test.mean=0.6266298
multilabel.f1.test.mean=0.0544503
The models performance was very limited. However, in contrast to spatial scale, we tried to extend the size of the training set by using the author-derived keywords presents in the Web of Science dataset. Using such author-derived keywords, we extended the number of observations from 1,500 to 3,800. Despite this significant increase in the training set, performance remained poor. For few examples, we could determined that it is likely that some of the abstract do not contain information about the location, especially when author precised the location in keywords.
Further, exploratory data analysis (EDA) revealed that observations with multilabel only represent 2% of the complete dataset. In consequence, the problem was recasted into a multi-class problem using the raw corpus rather than the abstracts.
| labels | input | model |
|---|---|---|
| multiclass | DTM from raw corpus | algorithm adaptation |
| multiclass | DTM from raw corpus | binary relevance (and variations) |
Nonetheless, these approach did not yield satisfactory results. Additional EDA underlined that part of the necessary information was absent from the training data. For example, the term mexico was absent from the TDM. The TDM was recomputed from the raw corpus with costumized cleaning to ensure that the correct data was left in the training data. This was performed by using a dictionnary of country names. In addition, we included additional term derived from a geographical database from Natural Earth Data. These data includes local administrative unit, glaciated area, rivers, lakes, dams, populated places and geographical regions (e.g. the Andes).
| labels | input | model |
|---|---|---|
| multiclass | country count from DTM from raw corpus | random forest, knn, svm, ann, multinomial |
Random Forest outperformed the other models.
Furthermore, adding more geographical keywords led to significant over-fitting in the final trained model.
A RF model was fine-tuned with grid search across 20 values of the mtry parameter. The best value was found to be 6.
Argentina Bahamas Barbados Belize Bolivia Brazil Chile Colombia Costa.Rica Cuba Ecuador El.Salvador Guatemala Haiti Honduras Irrelevant Jamaica Mexico Nicaragua Panama Paraguay Peru Uruguay Venezuela -err.- -n-
Argentina 255 0 0 0 2 28 11 3 2 1 3 1 1 0 0 18 1 15 1 1 2 5 1 2 98 353
Bahamas 0 24 0 0 0 2 0 1 0 0 0 0 0 0 0 4 0 2 0 0 0 0 0 1 10 34
Barbados 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9
Belize 0 0 0 14 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 15
Bolivia 0 0 0 0 77 3 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 6 83
Brazil 32 3 3 5 6 868 49 13 3 1 14 1 4 2 2 73 3 64 5 12 2 16 6 4 323 1191
Chile 9 1 1 0 4 22 279 3 4 2 1 1 0 0 0 20 0 12 3 1 0 7 0 3 94 373
Colombia 1 0 0 0 0 4 2 68 0 0 0 0 0 0 0 3 0 2 1 0 0 0 1 0 14 82
Costa.Rica 1 0 0 0 1 2 0 0 53 0 2 0 0 0 0 2 0 0 0 0 0 1 0 0 9 62
Cuba 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 1 0 1 0 0 0 0 0 0 2 18
Ecuador 1 0 0 0 1 4 2 0 0 0 63 0 0 0 0 3 0 1 0 1 0 0 0 0 13 76
El.Salvador 1 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 1 8
Guatemala 0 0 0 0 0 0 1 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 1 18
Haiti 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 1 0 0 0 0 0 0 1 8
Honduras 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 13
Irrelevant 25 1 1 2 5 63 30 10 11 4 7 0 1 2 2 356 1 47 2 5 1 6 3 3 232 588
Jamaica 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 2 20 1 0 0 0 0 0 0 5 25
Mexico 24 1 1 1 8 50 13 4 11 1 4 2 1 1 2 77 4 404 0 0 3 12 2 3 225 629
Nicaragua 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 1 0 1 2 29
Panama 0 0 0 0 0 3 0 1 0 0 0 0 0 0 0 3 0 1 0 47 0 1 0 0 9 56
Paraguay 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0 1 12
Peru 2 0 0 0 2 1 2 2 0 1 1 0 1 0 0 0 0 0 0 1 0 90 0 0 13 103
Uruguay 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 2 30 0 5 35
Venezuela 2 0 1 0 0 1 1 0 0 0 0 1 1 0 0 1 0 2 0 0 0 0 0 48 10 58
-err.- 98 6 7 9 30 187 111 38 32 10 32 6 9 5 6 207 9 149 12 23 8 51 13 17 1075 NA
-n- 353 30 16 23 107 1055 390 106 85 26 95 13 26 12 19 563 29 553 39 70 19 141 43 65 NA 3878
precision recall false.discovery.rate
Argentina 0.7223796 0.7223796 0.27762040
Bahamas 0.7058824 0.8000000 0.29411765
Barbados 1.0000000 0.5625000 0.00000000
Belize 0.9333333 0.6086957 0.06666667
Bolivia 0.9277108 0.7196262 0.07228916
Brazil 0.7287993 0.8227488 0.27120067
Chile 0.7479893 0.7153846 0.25201072
Colombia 0.8292683 0.6415094 0.17073171
Costa.Rica 0.8548387 0.6235294 0.14516129
Cuba 0.8888889 0.6153846 0.11111111
Ecuador 0.8289474 0.6631579 0.17105263
El.Salvador 0.8750000 0.5384615 0.12500000
Guatemala 0.9444444 0.6538462 0.05555556
Haiti 0.8750000 0.5833333 0.12500000
Honduras 1.0000000 0.6842105 0.00000000
Irrelevant 0.6054422 0.6323268 0.39455782
Jamaica 0.8000000 0.6896552 0.20000000
Mexico 0.6422893 0.7305606 0.35771065
Nicaragua 0.9310345 0.6923077 0.06896552
Panama 0.8392857 0.6714286 0.16071429
Paraguay 0.9166667 0.5789474 0.08333333
Peru 0.8737864 0.6382979 0.12621359
Uruguay 0.8571429 0.6976744 0.14285714
Venezuela 0.8275862 0.7384615 0.17241379
precision recall false.discovery.rate
Min. :0.6054 Min. :0.5385 Min. :0.00000
1st Qu.:0.7870 1st Qu.:0.6215 1st Qu.:0.08057
Median :0.8560 Median :0.6673 Median :0.14401
Mean :0.8398 Mean :0.6677 Mean :0.16018
3rd Qu.:0.9194 3rd Qu.:0.7164 3rd Qu.:0.21300
Max. :1.0000 Max. :0.8227 Max. :0.39456
mtry=6 :
acc.test.mean=0.5997827
mmce.test.mean=0.4002173
multiclass.au1u.test.mean=0.7654618
multiclass.aunu.test.mean=0.7830815