Merge branch '#49-set-up-a-binary-classifier' of https://github.com/w…

…ri-dssg/policy-data-analyzer into #49-set-up-a-binary-classifier

README.md

@@ -7,18 +7,18 @@ Current Roadmap
 
 ### Phase 1: Augmenting training data
 
-1. [ ] Fine-tune S-BERT on existing labeled data from 5 countries (`WRI_Policy_Tags.xlsx` file)
-2. [ ] Find methods to improve performance of S-BERT for data augmentation purposes
+1. [X] Fine-tune S-BERT on existing labeled data from 5 countries (`WRI_Policy_Tags.xlsx` file)
+2. [X] Find methods to improve performance of S-BERT for data augmentation purposes
 3. [ ] Build pipeline for further fine tuning as we get more data
-4. [ ] Classify the policy instrument of new sentences from El Salvador and Chile policy documents 
-5. [ ] Manually review the model tags and tag more examples (2 reviewers)
-6. [ ] Build pipeline to create excel documents for manual reviewing/tagging
+4. [X] Classify the policy instrument of new sentences from El Salvador and Chile policy documents 
+5. [X] Manually review the model tags and tag more examples (2 reviewers)
+6. [X] Build pipeline to create excel documents for manual reviewing/tagging
 7. [ ] Explore other models if needed
 
 ### Phase 2: Modeling
 
-1. [ ] Develop a model to first identify whether a sentence contains an incentive instrument, or is an icentive at all
-2. [ ] Develop a model that classifies incentive instruments (direct payment, tax deduction, etc.)
+1. [X] Develop a model to first identify whether a sentence contains an incentive instrument, or is an icentive at all
+2. [X] Develop a model that classifies incentive instruments (direct payment, tax deduction, etc.)
 
 -------------------------------------
 

tasks/evaluate_model/notebooks/avg-f1-results-compilation.csv

@@ -0,0 +1,7 @@
+,Experiment number,model,f1-score,epochs,test_perc
+0,13,paraphrase-xlm-r-multilingual-v1,0.5,6,0.15
+1,14,paraphrase-xlm-r-multilingual-v1,0.52,8,0.3
+2,15,paraphrase-xlm-r-multilingual-v1,0.58,10,0.2
+3,12,paraphrase-xlm-r-multilingual-v1,0.72,6,0.2
+4,10,paraphrase-xlm-r-multilingual-v1,0.75,10,0.3
+5,11,paraphrase-xlm-r-multilingual-v1,0.67,8,0.25

tasks/evaluate_model/notebooks/weighted-f1-results-compilation.csv

@@ -0,0 +1,7 @@
+,Experiment number,model,f1-score,epochs,test_perc
+0,13,paraphrase-xlm-r-multilingual-v1,0.4,6,0.15
+1,14,paraphrase-xlm-r-multilingual-v1,0.48,8,0.3
+2,15,paraphrase-xlm-r-multilingual-v1,0.65,10,0.2
+3,12,paraphrase-xlm-r-multilingual-v1,0.71,8,0.2
+4,10,paraphrase-xlm-r-multilingual-v1,0.82,10,0.3
+5,11,paraphrase-xlm-r-multilingual-v1,0.66,8,0.25

tasks/evaluate_model/output/avg-f1-results-compilation.csv

@@ -0,0 +1,7 @@
+,Experiment number,model,f1-score,epochs,test_perc
+0,0,paraphrase-xlm-r-multilingual-v1,0.59,8,0.15
+1,1,paraphrase-xlm-r-multilingual-v1,0.57,4,0.15
+2,3,paraphrase-xlm-r-multilingual-v1,0.55,6,0.3
+3,4,stsb-xlm-r-multilingual,0.72,4,0.3
+4,5,paraphrase-xlm-r-multilingual-v1,0.51,8,0.15
+5,2,paraphrase-xlm-r-multilingual-v1,0.79,10,0.25

tasks/evaluate_model/output/weighted-f1-results-compilation.csv

@@ -0,0 +1,7 @@
+,Experiment number,model,f1-score,epochs,test_perc
+0,13,paraphrase-xlm-r-multilingual-v1,0.4,6,0.15
+1,14,paraphrase-xlm-r-multilingual-v1,0.48,8,0.3
+2,15,paraphrase-xlm-r-multilingual-v1,0.65,10,0.2
+3,12,paraphrase-xlm-r-multilingual-v1,0.71,8,0.2
+4,10,paraphrase-xlm-r-multilingual-v1,0.82,10,0.3
+5,11,paraphrase-xlm-r-multilingual-v1,0.66,8,0.25