3D images not supported?

[FredrikM97]

Tried to use SmoothGradCAMpp, ScoreCAM and SSCAM on pytorch ResNet architecture.

SmoothGradCAMpp complains when taking the sum on the weights:
RuntimeError: The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension 1

Input: (1, 1, 79, 190, 158)

I printed the input and the received the following:
"Weight.View:" torch.Size([2048, 5, 1, 1]) "Hook Squeese:" torch.Size([2048, 3, 6, 5]) "Weights:" torch.Size([2048, 5]) "Hook:" torch.Size([1, 2048, 3, 6, 5])

Question: Is it reasonable that SSCAM allocates 18gb memory on the GPU?

[frgfm]

Hello again @FredrikM97 👋

Thanks for helping to catch some bugs! Could you produce a minimal snippet to produce this error please?
I don't have a 3DResNet available so I'm not able to reproduce the error!

Also, where is the RuntimeError raised please?

[FredrikM97]

No worries! Thank you for quick replies!

The error occurs at line 122 in cams.py:
batch_cams = (weights.view(*weights.shape, 1, 1) * self.hook_a.squeeze(0)).sum(0) # type: ignore[union-attr]
It looks like

Here is a sample of a 3D nifti image and the resnet: 3D_network.zip. Note that the resnet is configured for grayscale images. and three output classes.

To open the file:

import nibabel as nib
img = nib.load(example_filename).get_fdata()

Here is a sample of the code I use:

from resnet import resnet50
device = 'cuda'
model = resnet50().to(device)
model.eval()

input_tensor = torch.from_numpy(img).unsqueeze(0).unsqueeze(0).float()

cam = SmoothGradCAMpp(
    model, 
    'layer4',
    batch_size = 1,
    num_samples = 1,
    std = 2.0,
    input_shape = (1,1, 79, 224, 224),
)
with torch.no_grad(): 
    scores = model(input_tensor)
    activation_map = cam(scores.squeeze(0).argmax().item(), scores).cpu()

Update:
Before the avgpool layer in ResNet the shape of the data is torch.Size([1, 2048, 3, 6, 5]). So the hook seem to be working. However after: alpha = grad_2 / (2 * grad_2 + (grad_3 * init_fmap).sum(dim=(2, 3), keepdim=True)) the following shapes are observed:

alpha.shape: torch.Size([1, 2048, 3, 7, 7]) 
self.hook_g.squeeze(0).shape: torch.Size([2048, 3, 7, 7])
torch.relu(self.hook_g.squeeze(0)).shape:  torch.Size([2048, 3, 7, 7])
alpha.squeeze_(0).mul_(torch.relu(self.hook_g.squeeze(0))).shape:  torch.Size([2048, 3, 7, 7])

The last step is to perform .sum(dim=(1, 2)) which I assume is the cause of the new shape torch.Size([2048, 5, 1, 1]).

[zsddd]

不用担心！感谢您的快速回复！

错误发生在 cams.py 中的第 122 行：如下所示batch_cams = (weights.view(*weights.shape, 1, 1) * self.hook_a.squeeze(0)).sum(0) # type: ignore[union-attr]

以下是3D nifti图像和resnet的示例：3D_network.zip。请注意，resnet 是为灰度图像配置的。和三个输出类。

要打开文件：

import nibabel as nib
img = nib.load(example_filename).get_fdata()

以下是我使用的代码示例：

from resnet import resnet50
device = 'cuda'
model = resnet50().to(device)
model.eval()

input_tensor = torch.from_numpy(img).unsqueeze(0).unsqueeze(0).float()

cam = SmoothGradCAMpp(
    model, 
    'layer4',
    batch_size = 1,
    num_samples = 1,
    std = 2.0,
    input_shape = (1,1, 79, 224, 224),
)
with torch.no_grad(): 
    scores = model(input_tensor)
    activation_map = cam(scores.squeeze(0).argmax().item(), scores).cpu()

**更新：**在 ResNet 中的平均池层之前，数据的形状是火炬。大小（[1， 2048， 3， 6， 5]）。所以钩子似乎正在起作用。但是，在之后：观察到以下形状：alpha = grad_2 / (2 * grad_2 + (grad_3 * init_fmap).sum(dim=(2, 3), keepdim=True))

alpha.shape: torch.Size([1, 2048, 3, 7, 7]) 
self.hook_g.squeeze(0).shape: torch.Size([2048, 3, 7, 7])
torch.relu(self.hook_g.squeeze(0)).shape:  torch.Size([2048, 3, 7, 7])
alpha.squeeze_(0).mul_(torch.relu(self.hook_g.squeeze(0))).shape:  torch.Size([2048, 3, 7, 7])

最后一步是执行，我认为这是新形状的原因.sum(dim=(1, 2)) ``torch.Size([2048, 5, 1, 1]).

Hello, have you finished your 3D visualization? Can you share it? Thank you very much.

[frgfm]

Hi there 👋

The issue raised by @FredrikM97 was solved last year. I'm not sure what you're referring to regarding the 3D visualization?
CAMs are working on 3D data if that's your question :)

[zsddd]

Thank you for quick replies!

Hi there 👋

The issue raised by @FredrikM97 was solved last year. I'm not sure what you're referring to regarding the 3D visualization? CAMs are working on 3D data if that's your question :)
Thank you for quick replies!
My program has the following problems:cannot register a hook on a tensor that doesn't require gradient
This is my program:
file=r'E:\pythonProject\data\train\128\label404347_4 1.npy'
con_arr1 = np.load(file,allow_pickle=True) # 读取npy文件
con_arr=con_arr1[0][0]
device = 'cuda'
model = resnet18(num_classes=1).to(device)
model.eval()

input_tensor = np.array(con_arr)
input_tensor=torch.from_numpy(input_tensor)
input_tensor =input_tensor.unsqueeze(0).unsqueeze(0).float().to(device)

print(input_tensor.shape)

cam = SmoothGradCAMpp( model,'layer1',num_samples = 1,std = 2.0,input_shape = (1,128,128,128))
with torch.no_grad():
scores = model(input_tensor)
activation_map = cam(scores.squeeze(0).argmax().item(), scores).cpu()