Skip to content

peterbhase/LAS-NL-Explanations

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 

Leakage-Adjusted Simulatability

This is the codebase for the paper:
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?
Peter Hase, Shiyue Zhang, Harry Xie, Mohit Bansal. Findings of EMNLP 2020

Repository Structure

|__ sim_experiments/ --> Directory with models and experiment scripts
    |__ data/ --> includes eSNLI and CoS-E data used in experiments
    |__ models/ --> includes wrappers for T5 model for multiple choice tasks
    |__ training_reports/ --> includes wrappers for T5 model for multiple choice tasks
    |__ main.py --> train task model or simulator model on SNLI or CQA
    |__ T5-2-agent_main.py --> train models for multi-agent experiments [SGD] in paper
    |__ compute_sim.py --> script for computing LAS score for model explanations
    |__ run_tasks.py --> script for running experimental conditions across seeds
    |__ *.py --> utilities for data loading, explanation sampling, etc.
    |__ causal_estimation.Rmd --> R markdown script used to calibrate simulator models and compute LAS with *k* bins
|__ alternative_objectives/ --> Directory with additional experimental scripts
    |__ code coming soon
|__ rl_experiments/ --> Directory with code for multi-agent [RL] experimental scripts
    |__ see internal README
|__ human_experiments/ --> Directory with R code for analyzing human experiment data        
    |__ ratings_analysis.Rmd --> R markdown script used to analyze human quality ratings
    |__ expert_analysis.Rmd --> R markdown script used to analyze expert simulation data
    |__ more human experiment code coming soon
|__ requirements.txt

Requirements

  • Python 3.6
  • torch 1.4
  • transformers 2.5.1
  • sacrebleu
  • pandas
  • numpy

Reproducing Experiments

See READMEs in each directory for instructions on reproducing each set of experiments.

About

Code for paper "Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published