Visual explanations are useful for understanding how black box machine learning models make decisions. Visualizing how a model makes decisions can help evaluate how reliable a model will be on unseen data and, when sensible, can increase user trust in the predictions of a model.
Visual explanations are often generated by quantifying the importance of input features for the model's decision. Popular tools for determining importance include gradients (Grad-CAM) and Shapley values (SHAP). This package computes visual explanations for text and image classifiers by analyzing how masking features impacts a model's decision. Intuitively, the more important a feature is, the greater the impact masking that feature should have on the model's output. FeatMask ranks features by the effect masking the feature has on model output and produces a visual explanation highlighting the most significant features for the model's decision.
For image classifiers, FeatMask computes the importance of image patches by determining how model output changes if a patch is masked with a solid color, by default white. The default visualization builds a heatmap over the image, with red identifying the features which have the greatest effect on the model's output. The image below (left) shows the explanation produced by FeatMask for the identification of a penguin using the ResNet-50 architecture trained on ImageNet, compared to the explanation produced by Grad-CAM++ (right). Both explanations suggest that the model "looks at" the penguin when identifying a penguin in the image.
Visual explanations can assist with more complex image classification problems as well. For example, classifiers can be used to identify cancer based on medical images, such as the breast histopathology images in the BreakHis dataset. Here, a distinguishing feature of cancer is nuclear pleomorphism, seen in the nuclei in the image below (left) as irregular dark purple stains. Applying FeatMask (right) to a CNN model trained to identify cancer on BreakHis suggests that the model can identify nuclear pleomorphism as a significant feature for predicting cancer.
FeatMask can also provide visual explanations for text classifiers, such as models used for sentiment analysis. Texts can be separated into features by individuating words or by using a predefined tokenizer. Features can be masked by replacing individual tokens with a space, which translates to an [UNK] token for BERT models. For binary classification problems, importance is visualized by highlighting a feature's positive contribution in red and negative contribution in blue. For example, below is the visualization for a fine-tuned BERT model's assignment of negative sentiment to a movie review from IMDB, with positive contributions highlighted in red and negative contributions in blue.
Here, we can see that the negative sentiment mostly comes from the features "poor", "sterile," and "not...endures", while positive aspects of the review include "good (not great) performance" and "beautiful". We can compare this with the visual explanation determined by the features' Shapley values, which agrees with FeatMask on the most significant features for the model.
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018, March). Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV) (pp. 839-847).
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626).