Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Confidence Modeling for Neural Semantic Parsing

Setup

Requirements

Install Python dependency

pip install -r requirements.txt

Install KenLM

./install_kenlm.sh

Download data and pretrained model

Download the zip file from Google Drive, and copy it to the folder of code.

unzip acl18confidence_data_model.zip

Precompute Confidence Metrics

Train a language model

./train_lm.sh ifttt
./train_lm.sh django

Precompute all the confidence metrics, and perform uncertainty backpropagation

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./compute_metric.sh ifttt 0
./compute_metric.sh django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".

Usage

Confidence Estimation

  • Train a confidence scoring model.
  • Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).
./train_scoring.sh ifttt
./train_scoring.sh django

Uncertainty Interpretation

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./eval_conf_bp_att.sh ifttt
./eval_conf_bp_att.sh django

Acknowledgments

About

No description, website, or topics provided.

Resources

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.