UNet does not work for my own data

[Cpresident]

Hello,

I modified code from the [brats_segmentation_3d] tutorial with my own data (2 input MRI series and 11-class output segmentation)

the code worked fine in the original brat tutorial and brat data, but showed error Sizes of tensors must match except in dimension 4. Got 2 and 1 (The offending index is 0) in my case. I have worked on this a half a day, done lots of search, stiil don't know why (guess a torch.cat problem?). Thanks a lot!

I generated a train_loader_sample with next(iter(train_loader)) with batchsize=2, the image and label shape is

loadersample["image"].shape
print(loadersample["image"].shape, loadersample["label"].shape)

[Output]torch.Size([2, 2, 400, 400, 8]) torch.Size([2, 11, 400, 400, 8])

and created the model as the tutorial:

# standard PyTorch program style: create UNet, DiceLoss and Adam optimizer
device = torch.device("cuda:0")
model = UNet(
    dimensions=3,
    in_channels=2,
    out_channels=11,
    channels=(16, 32, 64, 128, 256),
    strides=(2, 2, 2, 2),
    num_res_units=2,
).to(device)
loss_function = DiceLoss(to_onehot_y=False, sigmoid=True, squared_pred=True)
optimizer = torch.optim.Adam(
    model.parameters(), 1e-4, weight_decay=1e-5, amsgrad=True
)

inputs, labels = (loadersample["image"].to(device), loadersample["label"].to(device))
print(inputs.shape)
optimizer.zero_grad()
outputs = model(inputs)
outputs

and showed

[Output]torch.Size([2, 2, 400, 400, 8])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-60-8bb8d76f38a1> in <module>
      2 print(inputs.shape)
      3 optimizer.zero_grad()
----> 4 outputs = model(inputs)
      5 outputs

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\monai\networks\nets\unet.py in forward(self, x)
    186 
    187     def forward(self, x: torch.Tensor) -> torch.Tensor:
--> 188         x = self.model(x)
    189         return x
    190 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\monai\networks\layers\simplelayers.py in forward(self, x)
    122 
    123     def forward(self, x: torch.Tensor) -> torch.Tensor:
--> 124         y = self.submodule(x)
    125 
    126         if self.mode == "cat":

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\monai\networks\layers\simplelayers.py in forward(self, x)
    122 
    123     def forward(self, x: torch.Tensor) -> torch.Tensor:
--> 124         y = self.submodule(x)
    125 
    126         if self.mode == "cat":

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\monai\networks\layers\simplelayers.py in forward(self, x)
    125 
    126         if self.mode == "cat":
--> 127             return torch.cat([x, y], dim=self.dim)
    128         if self.mode == "add":
    129             return torch.add(x, y)

RuntimeError: Sizes of tensors must match except in dimension 4. Got 2 and 1 (The offending index is 0)

[rijobro]

Hi, the problem is the depth of your image being just 8. This doesn't allow for the number of convolutions you have requested.

With a bigger image it works fine:

import torch
from monai.networks.nets import UNet
device = torch.device("cuda:0")
model = UNet(
    dimensions=3,
    in_channels=2,
    out_channels=11,
    channels=(16, 32, 64, 128, 256),
    strides=(2, 2, 2, 2),
    num_res_units=2,
).to(device)
model(torch.rand((2,2,400,400,16), device=device))

Or with a smaller network, your original image works fine, too:

import torch
from monai.networks.nets import UNet
device = torch.device("cuda:0")
model = UNet(
    dimensions=3,
    in_channels=2,
    out_channels=11,
    channels=(16, 32, 64, 128),
    strides=(2, 2, 2),
    num_res_units=2,
).to(device)
model(torch.rand((2,2,400,400,8), device=device))

[Cpresident]

oh, I see. Really thanks!

[Cpresident]

@rijobro Hi, rijobro, another question please help: does the depth of MRI is relaetd to the kernal size and stride?

When I want to send a 400x400x14 MRI cube this time, the model doesn't work again(but 15, 16 is ok). I am not a professional coder and did search a lot but still cannot understand why. Wonder if you can explain a little, or any paper recommanded for reading?

import torch
from monai.networks.nets import UNet
device = torch.device("cuda:0")
model = UNet(
    dimensions=3,
    in_channels=2,
    out_channels=13,
    channels=(16, 32, 64, 128),
    strides=(2, 2, 2),
    num_res_units=2,).to(device)
model(torch.rand((2,2,400,400,14), device=device))
-----
RuntimeError: Sizes of tensors must match except in dimension 4. Got 8 and 7 (The offending index is 0)

[Vivekrpg]

I am running into another issue on a related note. I am trying to use UNet model tutorial given by applying to my own data. However, I am running into this error

`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in
49 print(val_labels.shape)
50 print(torch.max(val_labels))
---> 51 value, _ = dice_metric(y_pred=val_outputs, y=val_labels)
52 metric_count += len(value)
53 metric_sum += value.item() * len(value)

~/monai_code/MONAI/monai/metrics/meandice.py in call(self, y_pred, y)
64 warnings.warn("y_pred is not a binarized tensor here!")
65 if not torch.all(y.byte() == y):
---> 66 raise ValueError("y should be a binarized tensor.")
67 dims = y_pred.ndimension()
68 if dims < 3:

ValueError: y should be a binarized tensor.`

On debugging, I realized that in the given tutorial, seg had all values between 0 and 1 and hence dice_metric converts the tensor to a byte. However, the seg mask that we have in our data has values of more than 1. What is the best method to normalize the label in Monai platform?

[rijobro]

Are you doing single- or multi-label segmentation? Do you use post_label and post_pred as per https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb?

[Vivekrpg]

Thanks Richard for highlighting the issue. I am running semantic segmentation and using num_classes=2
Here are the post_pred and post_label I am using

post_pred = Compose([Activations(softmax=True), AsDiscrete(argmax=True, to_onehot=True, n_classes=2)])
post_label = AsDiscrete(to_onehot=True, n_classes=2)
loss_function = DiceLoss(to_onehot_y=True, softmax=True)

Although, training goes fine, however, at some stage it fails with the following error:
`---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in
37 print(f"sum of outputs is {outputs.sum().item()}")
38 loss = loss_function(outputs, labels)
---> 39 loss.backward()
40 optimizer.step()
41 epoch_loss += loss.item()

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
219 retain_graph=retain_graph,
220 create_graph=create_graph)
--> 221 torch.autograd.backward(self, gradient, retain_graph, create_graph)
222
223 def register_hook(self, hook):

~/anaconda3/lib/python3.7/site-packages/torch/autograd/init.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
130 Variable.execution_engine.run_backward(
131 tensors, grad_tensors, retain_graph, create_graph,
--> 132 allow_unreachable=True) # allow_unreachable flag
133
134

RuntimeError: CUDA error: device-side assert triggered`

My understanding is that this error happens when loss_function is given values that could be out of the bounds. Do you have any suggestions to handle this? Would I need to use post_pred in training method as well?