Using mobilevit attention layers to create heat map

[zas97]

Is it possible to use the attention layers of the mobilevit model to create a heatmap on the image input with the parts of the image used? I'm looking to something like this :

[sacmehta]

Hi @zas97 ,

This is definitely possible and you need to write the code for the same. We do not support it.

In our MobileViTv2 paper, we did visualize the context maps and you may use it as a reference for implementing for multi-headed attention in transformers.

[arunsubk]

Hi @zas97 , Could you please let me know if you were able to find or create such code as suggested by @sacmehta ? If yes some pointers could really help. I am unable to wrap my head around how I can get an attention heat map from a final attention map for MobileVit of shape 1,4,4,16,16 i.e. batch_size, nbr of heads, dimension, attention score, attention score for input image dimension of 256,256,3 can be remapped to attention heatmap.

Thanks,
Arun

[zas97]

Not really I left the project