An Evaluation Method for Feature Attribution Explainers

This is the code repository for the AAAI 2022 paper Do Feature Attribution Methods Correctly Attribute Features? by Yilun Zhou, Serena Booth, Marco Tulio Ribeiro and Julie Shah. A video presentation is available here.

TLDR: We "unit test" several popular feature attribution algorithms for CV and NLP models to see if they can identify features known to be highly important to model predictions; (un)surprisingly, they mostly can't.

What's Next: Don't stop at local explanations. Ensuring that high-quality model understanding is derived from these local explanations is equally important. Check out our ExSum framework for more details on this aspect.

Requirements

This repository has minimal dependencies. Specifically, it requires tqdm for the progress bar, numpy and torch for numerical computing, matplotlib for plotting, Pillow for image manipulation, and lime and shap for the implementation of LIME and SHAP methods. All dependencies can be installed with pip install -r requirements.txt.

Code Structure

This repository contains two folders.

• image_saliency contains code related to training and explaining the ResNet-34 model on our bird dataset.
• text_rationale_attention contains code related to training and explaining rationale and attention models on the modified BeerAdvocate dataset.

Please refer to the README.md in the respective folder for detailed information.

Contact and Citation

For any questions, please contact Yilun Zhou at yilun@mit.edu. The paper can be cited as

@inproceedings{zhou2022feature,
    title = {Do Feature Attribution Methods Correctly Attribute Features?},
    author = {Zhou, Yilun and Booth, Serena and Ribeiro, Marco Tulio and Shah, Julie},
    booktitle = {Proceedings of the 36th AAAI Conference on Artificial Intelligence},
    year = {2022},
    month = {Feb},
    publisher = {AAAI}
}