No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Latest commit 7d70587 May 21, 2018
Permalink
Failed to load latest commit information.
onmt init May 9, 2018
utils init May 9, 2018
.gitignore init May 9, 2018
PTB_Tree_eval.py init May 9, 2018
README.md Update README.md May 11, 2018
clean.sh init May 9, 2018
compute_metric.sh init May 9, 2018
eval_conf_bp.py init May 9, 2018
eval_conf_bp_att.sh init May 9, 2018
evaluate.py init May 9, 2018
install_kenlm.sh init May 9, 2018
preprocess.py init May 9, 2018
requirements.txt Update requirements.txt May 21, 2018
train.py init May 9, 2018
train_lm.sh init May 9, 2018
train_scoring.sh init May 9, 2018

README.md

Confidence Modeling for Neural Semantic Parsing

Setup

Requirements

Install Python dependency

pip install -r requirements.txt

Install KenLM

./install_kenlm.sh

Download data and pretrained model

Download the zip file from Google Drive, and copy it to the folder of code.

unzip acl18confidence_data_model.zip

Precompute Confidence Metrics

Train a language model

./train_lm.sh ifttt
./train_lm.sh django

Precompute all the confidence metrics, and perform uncertainty backpropagation

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./compute_metric.sh ifttt 0
./compute_metric.sh django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".

Usage

Confidence Estimation

  • Train a confidence scoring model.
  • Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).
./train_scoring.sh ifttt
./train_scoring.sh django

Uncertainty Interpretation

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./eval_conf_bp_att.sh ifttt
./eval_conf_bp_att.sh django

Acknowledgments