This repository is associated with the paper A Rationale-Centric Framework for Human-in-the-loop Machine Learning (Accepted to the main conference of ACL2022)
Tested Python 3.6, and requiring the following packages, which are available via PIP:
- Required: numpy >= 1.19.5
- Required: scikit-learn >= 0.21.1
- Required: pandas >= 1.1.5
- Required: torch >= 1.9.0
- Required: transformers >= 4.8.2
- Required: datasets>=1.14.0
- Required: nltk>=3.6.5
or pip install -r requirements.txt
.
├── datasets # IMDb datasets, human labelled rationales, counterfactuals examples (Hovy et al.)
├── AL_results # Experimental outputs of baseline active learning
├── DP_results # Experimental outputs of baseline duplication
├── RR_results # Experimental outputs of baseline random replacement
├── MR_results # Experimental outputs of baseline missing rationales
├── FR_results # Experimental outputs of baseline false rationales
├── SF_results # Experimental outputs of our approach static semi-factuals
├── full_results # Experimental outputs of baseline training with the full training set
├── Hybrid_results # Experimental outputs of our approach dynamic human-intervened correction
└── README.md
- For running the code, you should add some code in trainer.py (as shown below) under the transformers directory (in my device on ~/Anaconda3/Lib/site-packages/transformers/trainer.py):
-
Random sampling a certain number (25 in our experiments) of positives and negatives storing in AL_results/AL_step0_IMDb_trainer_{seed}_ {num_instances_each_class}/keys.txt
-
Please run scripts with step0 first, then go to IMDb_AL_example_selection_step1.ipynb to extract another 50 examples from the unlabelled pool according to uncertainty sampling. Then can run scripts with step1.
See static_semi_factual_generation.ipynb
Run IMDb_step1_generate_false_rationales_position.py Then see IMDb_generate_false_rationales_examples.ipynb
Run IMDb_step1_generate_missing_rationales_examples.py
For Yelp and Amazon OOD test please go OOD_Testing_Amazon and OOD_Testing_Yelp, for in-domain and other OOD testing please go In-domain_OOD_all.py. Since the size limited, we won't put the OOD data on github, you can search and download those datasets online freely or contact jinghui.lu@ucdconnect.ie or lujinghui1@sensetime.com