Gradient Bug in backprop.visualize

[goldblum]

I tried generating a saliency map using the following code:

backprop = Backprop(net)
target_class = 4
backprop.visualize(image, target_class, guided=True)

My net is a standard PyTorch neural network, and the image is a (1,3,84,84) image. However, I get the following error:

File "visualizations.py", line 89, in test
backprop.visualize(image, target_class, guided=True)
File "/home/user/.local/lib/python3.6/site-packages/flashtorch/saliency/backprop.py", line 168, in visualize
guided=guided)
File "/home/user/.local/lib/python3.6/site-packages/flashtorch/saliency/backprop.py", line 120, in calculate_gradients
output.backward(gradient=target)
File "/home/user/.local/lib/python3.6/site-packages/torch/tensor.py", line 150, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/user/.local/lib/python3.6/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
File "/home/user/.local/lib/python3.6/site-packages/flashtorch/saliency/backprop.py", line 214, in _record_gradients
if self.gradients.shape == grad_in[0].shape:
AttributeError: 'NoneType' object has no attribute 'shape'

Do I need to turn the image into a parameter with requires_grad = True?
Is this usually done by the load_image function? I found that this function was re-sizing my image to 224x224. Thanks for your help!

[sumanyugupta]

Facing the same error. I tried using ResNet50 instead of AlexNet and got the same issue as you. However, using AlexNet works just fine.

[goldblum]

I have avoided this error by turning on the requires_grad=True, but the packages still does not work fully for me, so I am not sure if this solved the problem.

[MisaOgura]

Hi @goldblum and @sumanyugupta ,

Apologies for the late reply.

I'll be happy to assist if you are still having problems using FlashTorch.

Many thanks,
Misa

[Lecanyu]

@MisaOgura
Hi, I face the same issue. The shape of my input image is (1,2,3,84,84), and I have some .view and concatenation operations in my model inference.

Can this library handle the special input case?

BTW,
The second dimension indicates the input is a pair of images.
The first dimension indicates that these two images belong to a category.

[MisaOgura]

Hi @Lecanyu,

Thanks for reporting the issue!

I will try to look into this in the coming weeks.

Many thanks,
Misa

[ptempczyk]

@MisaOgura
Hi. I have the same problem. Other network than alexnet doesn't work even on the owl example (e.g mobilenet_v2). Retain grad helps, but saliency maps doesn't make any sense.

[MisaOgura]

Apologies folks for the lateness of this reply - I hope everyone is well and staying safe.

@goldblum The error message indicates that the gradients that's flowing into the first 2D convolutional layer is None. I'm closing this for now, but If you are still experiencing problems, please reopen this issue with the architecture of your network.

@Lecanyu I haven't got around to add support for custom input shapes, so at the moment the Backprop class only supports for input images with a shape (N, C, H, W) as detailed here. I will try keep this feature in mind though - if you can create a feature request using this template that will help me keep track.

@sumanyugupta @ptempczyk I can confirm that the example notebook works with pre-trained ResNet50 & MobileNetv2 loaded from torchvision.models (see below).

ResNet50:

MobileNetv2: