- Python 2.7
- PyTorch 0.1.12.post2 (GPU)
Install Python dependency
pip install -r requirements.txt
Download data and pretrained model
Download the zip file from Google Drive, and copy it to the folder of code.
Precompute Confidence Metrics
Train a language model
./train_lm.sh ifttt ./train_lm.sh django
Precompute all the confidence metrics, and perform uncertainty backpropagation
(This step can be skipped. The precomputed data have been cached in the zip file.)
# arg2: gpu id ./compute_metric.sh ifttt 0 ./compute_metric.sh django 0
The intermediate results are saved to "data_model/[ifttt|django]/*.eval".
- Train a confidence scoring model.
- Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).
./train_scoring.sh ifttt ./train_scoring.sh django
Evaluate confidence backpropagation and attention-based method against inferred ground truth.
./eval_conf_bp_att.sh ifttt ./eval_conf_bp_att.sh django