This is a Keras implementation of the the paper Brian Kenji Iwana, Ryouhei Kuroki, and Seiichi Uchida, Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation, ICCV Workshops, 2019. arXiv
Convolutional Neural Networks (CNN) have become state-of-the-art in the field of image classification. However, not everything is understood about their inner representations. This paper tackles the interpretability and explainability of the predictions of CNNs for multi-class classification problems. Specifically, we propose a novel visualization method of pixel-wise input attribution called Softmax-Gradient Layer-wise Relevance Propagation (SGLRP). The proposed model is a class discriminate extension to Deep Taylor Decomposition (DTD) using the gradient of softmax to back propagate the relevance of the output probability to the input image. Through qualitative and quantitative analysis, we demonstrate that SGLRP can successfully localize and attribute the regions on input images which contribute to a target object's classification. We show that the proposed method excels at discriminating the target objects class from the other possible objects in the images. We confirm that SGLRP performs better than existing Layer-wise Relevance Propagation (LRP) based methods and can help in the understanding of the decision process of CNNs.
- 2020/01/26: SGLRPSeqA, SGLRPSeqB - New implentation of SGLRP based on Sequential LRP. White paper incoming, for now if you use SGLRPSeqA or SGLRPSeqB, cite the paper below. Warning, experimental, sometimes it doesn't work as expected.
- 2019/12/20: GPU docker support
- 2019/11/02: ICCV Workshop on Explainable AI
- 2019/08/06: ArXiv paper posted
- 2019/07/23: Initial commit
SGLRP is a class contrastive extension of LRP. The general idea is that a relevance penalty is propagated through the network to create the relevance heatmaps.
Specifically, we use the gradient of softmax as the initial relevance signal for LRP. Or,
where t is the target class and n is the other classes.
This code was developed in Python 3.5.2. and requires Tensorflow 1.10.0
pip install keras==2.1.5 numpy==1.14.5 matplotlib==2.2.2 innvestigate==1.0.8.3 scikit-image==0.15.0
sudo docker build -t sglrp .
docker run --runtime nvidia -rm -it -p 127.0.0.1:8888:8888 -v `pwd`:/work -w /work sglrp jupyter notebook --allow-root
utils.visualizations.SGLRP(partial_model, target_id, relu=False, low=-1., high=1., **kwargs)
partial_model : Keras Model instance
A keras model with the output of softmax cut off, but the input to the output layer intact. The partial model can be found using the innvestigate library using the innvestigate.utils.keras.graph.pre_softmax_tensors() function.
target_id : int
The index of the target class.
relu : bool
Controls if ReLU is applied to the visualization. True means that only the positive relevance is shown; False shows both the positive and negative relevance.
low : float
The upper bounds of the inputs.
high : float
The lower bounds of the inputs.
SGLRP.analyze(input_imgs)
input_imgs : 4D array
Array of input images in the format of (image_id, height, width, channel).
4D array
Array of heatmaps in the format of (image_id, height, width, channel).
from utils.visualizations import SGLRP
import innvestigate.utils as iutils
model = [keras model]
partial_model = Model(inputs=model.inputs, outputs=iutils.keras.graph.pre_softmax_tensors(model.outputs))
target_id = [id of target class]
analysis = SGLRP(partial_model, target_id).analyze(input_imgs)
B. K. Iwana, R. Kuroki, and S. Uchida, "Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation," in International Conference on Computer Vision Workshops, 2019.
@inproceedings{iwana2019explaining,
title={Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation},
author={Iwana, Brian Kenji and Kuroki, Ryohei and Uchida, Seiichi},
booktitle={International Conference on Computer Vision Workshops},
year={2019}
}