Class activation maps API explained

[kapsner]

Hi,

I do have some questions on how to correctly apply the monai class activation maps (CAM) API and would be very happy to get some help on this topic.

• a) What are the differences between monai.visualize.class_activation_maps.CAM and monai.visualize.class_activation_maps.GradCAM?
• b) How are the CAMs to be interpreted when setting the argument class_idx=None vs. class_idx=y_true?
• c) What does the argument layer_idx=- 1 exactly do? (I was unable to find further information on this argument in the API documentation)
• d) Which layers (across different model architectures) are typically to be chosen for the argument target_layers?

Many thanks in advance.

[rijobro]

a. for a difference between CAM and Grad-CAM, see this paper, specifically the section starting "Most relevant to our approach is the Class Activation Mapping (CAM) approach to localization...". Grad-CAM++ has also been wrapped in MONAI, see this paper for an explanation.

b. class_idx=None will use the label inferred by the model -- this may be correct or incorrect depending on how good your model is. class_idx=y_true will use the correct label. Both can be useful "which parts of the image were important for the model to reach a given decision?" vs. "which parts of the image were important for the model to arrive at the correct decision?"

c. if you click the "source" link in the help documentation, you get a bit more information (link): "index of the target layer if there are multiple target layers". -1 will use the network's last layer, which is the most frequently used layer for CAM/Grad-CAM (hopefully this answers your fourth question, too).

[kapsner]

@rijobro Thank you very much for your answers and the deeper insights.

Indeed, regarding a) I wasn't sure, if CAM really refers to this paper. Maybe, this could be added to the API description of monai.visualize.class_activation_maps.CAM?

[kapsner]

Regarding question b) from above:
I have now worked a bit with GradCAM++ and often observe, that even if images were correctly classified, GradCAMs created with class_idx=None look completely different compared to those generated with the provided correct labels (class_idx=y_true).

Example of a correctly classified image (binary classification):
class_idx=None:

(same image) class_idx=1:

How is this to be interpreted? I would expect no differences in both GradCAMs in case of a correct classification.

Or does it mean, that the classifier considered parts of an image as relevant that let by chance / with great uncertainty (e.g. probability near but > 0.5 in a binary classification) to the correctly class and providing the true label just helps to see, to which image part the classifier would have looked at if it has known the correct class? (Which would indicate, that the classifier is not really "good").

Or is there something else to be considered?

[rijobro]

Ok, thanks for bringing this to our attention, I'll create an issue for it. If you wanted to help with debugging, it would be really useful to create a minimum (not)working example. If it were using a pre-trained densenet, then we wouldn't even need to do any training to see the issue.

[rijobro]

Issue created: #1710

[kapsner]

Thanks a lot - I will see if I can provide a minimum working example soon!

[rijobro]

@kapsner - do you think you could try this PR and let me know if it solves your problem? #1734

[kapsner]

@rijobro I appologize for not yet providing a working example - didn't find enough time yet.

However, I can confirm, that this PR solves the problem for me! Thank you very much!!!