Confidence Modeling for Neural Semantic Parsing

Setup

Requirements

• Python 2.7
• PyTorch 0.1.12.post2 (GPU)

Install Python dependency

pip install -r requirements.txt

Install KenLM

./install_kenlm.sh

Download data and pretrained model

Download the zip file from Google Drive, and copy it to the folder of code.

unzip acl18confidence_data_model.zip

Precompute Confidence Metrics

Train a language model

./train_lm.sh ifttt
./train_lm.sh django

Precompute all the confidence metrics, and perform uncertainty backpropagation

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./compute_metric.sh ifttt 0
./compute_metric.sh django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".

Usage

Confidence Estimation

• Train a confidence scoring model.
• Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).

./train_scoring.sh ifttt
./train_scoring.sh django

Uncertainty Interpretation

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./eval_conf_bp_att.sh ifttt
./eval_conf_bp_att.sh django

Acknowledgments

• The implementation is based on OpenNMT-py.
• XGBoost: gradient tree boosting model.
• KenLM: language model.