Skip to content

dongpng/eval_explanations_naacl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 

Repository files navigation

Comparing automatic and human evaluation of local explanations for text classification

Nguyen, NAACL 2018

Data

Sources

The paper uses two datasets:

  1. Movie reviews: Zaidan et al. (NAACL 2007), using the data with rationales (but rationales were not used in this paper).

  2. Twenty news groups: The 20news-bydate version is used with the following two categories: alt.atheism and soc.religion.christian

Processed data

The processed data, including vocabulary files and trained machine learning models, can be found in the output_data folder.

Output of experiments

These files can be download here (75.1MB). The zip file contains two directories: experiments & output_data

Explanations

The explanations can be generated as follows for two classifiers (LR and MLP):

Movie

experiments.explanations_rationales() generates:

  • ../experiments/rationales/rationales_MLP_explanations.csv
  • ../experiments/rationales/rationales_lr_explanations.csv

20News

experiments.explanations_news() generates:

  • ../experiments/news/news_MLP_explanations.csv
  • ../experiments/news/news_lr_explanations.csv

Crowdsourcing

Data

  • ../experiments/rationales/rationales_cf_responses.csv (movie reviews)
  • ../experiments/news/news_cf_responses.csv (20news)
  • ../experiments/rationales/rationales_cf_responses_with_noise.csv (movie reviews, with noise)

Analysis

  • analysis.rmd

Code

  • src/analysis.rmd R analysis code
  • src/classifierwrapper.py wrapper around keras and scikit-learn models
  • src/dataset.py reading, processing and saving the datasets
  • src/experiments.py main file to generate the explanations
  • src/explanation_evaluation.py computes the automatic evaluation metrics
  • src/explanation_test.py some tests
  • src/explanation_util.py some utility methods
  • src/keras_networks.py code to train the models using keras
  • src/process_crowdflower_annotations.py computes correlations between automatic and human evaluations and prints out the results
  • src/rationales_dataset.py to process the movie data
  • src/twentynewsgroups_dataset.py to process the 20news data

Libraries

About

Code for Nguyen, NAACL 2018

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Languages