Device mismatch issue in anomalydetection_tutorial_classifier_guidance.ipynb

[KumoLiu]

[2025-09-16T21:25:26.408Z] Running ./generation/anomaly_detection/anomalydetection_tutorial_classifier_guidance.ipynb
[2025-09-16T21:25:26.408Z] Checking PEP8 compliance...
[2025-09-16T21:25:27.035Z] Running notebook...
[2025-09-16T21:25:27.035Z] Before:
[2025-09-16T21:25:27.035Z]     "max_epochs = 2000\n",
[2025-09-16T21:25:27.035Z]     "max_epochs = 1000\n",
[2025-09-16T21:25:27.035Z] After:
[2025-09-16T21:25:27.035Z]     "max_epochs = 1\n",
[2025-09-16T21:25:27.035Z]     "max_epochs = 1\n",
[2025-09-16T21:25:27.035Z] Before:
[2025-09-16T21:25:27.035Z]     "val_interval = 20\n",
[2025-09-16T21:25:27.035Z]     "val_interval = 10\n",
[2025-09-16T21:25:27.319Z] After:
[2025-09-16T21:25:27.319Z]     "val_interval = 1\n",
[2025-09-16T21:25:27.319Z]     "val_interval = 1\n",
[2025-09-16T21:25:39.867Z] MONAI version: 1.5.0+20.g401ea4a0
[2025-09-16T21:25:39.867Z] Numpy version: 1.24.4
[2025-09-16T21:25:39.867Z] Pytorch version: 2.5.0a0+e000cf0ad9.nv24.10
[2025-09-16T21:25:39.867Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2025-09-16T21:25:39.867Z] MONAI rev id: 401ea4a0af2b9d475989a78e1bae387875fa2384
[2025-09-16T21:25:39.867Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2025-09-16T21:25:39.867Z] 
[2025-09-16T21:25:39.867Z] Optional dependencies:
[2025-09-16T21:25:39.867Z] Pytorch Ignite version: 0.4.11
[2025-09-16T21:25:39.867Z] ITK version: 5.4.4
[2025-09-16T21:25:39.867Z] Nibabel version: 5.3.2
[2025-09-16T21:25:39.867Z] scikit-image version: 0.25.2
[2025-09-16T21:25:39.867Z] scipy version: 1.14.0
[2025-09-16T21:25:39.867Z] Pillow version: 10.4.0
[2025-09-16T21:25:39.867Z] Tensorboard version: 2.16.2
[2025-09-16T21:25:39.867Z] gdown version: 5.2.0
[2025-09-16T21:25:39.867Z] TorchVision version: 0.20.0a0
[2025-09-16T21:25:39.867Z] tqdm version: 4.66.5
[2025-09-16T21:25:39.867Z] lmdb version: 1.7.3
[2025-09-16T21:25:39.867Z] psutil version: 6.0.0
[2025-09-16T21:25:39.867Z] pandas version: 2.2.2
[2025-09-16T21:25:39.867Z] einops version: 0.8.0
[2025-09-16T21:25:39.867Z] transformers version: 4.40.2
[2025-09-16T21:25:39.867Z] mlflow version: 3.3.2
[2025-09-16T21:25:39.867Z] pynrrd version: 1.1.3
[2025-09-16T21:25:39.867Z] clearml version: 2.0.3rc0
[2025-09-16T21:25:39.867Z] 
[2025-09-16T21:25:39.867Z] For details about installing the optional dependencies, please visit:
[2025-09-16T21:25:39.867Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2025-09-16T21:25:39.867Z] 
[2025-09-16T21:25:41.356Z] papermill  --progress-bar --log-output -k python3
[2025-09-16T21:25:41.970Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2025-09-16T21:25:41.970Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2025-09-16T21:25:41.970Z] 
[2025-09-16T21:25:42.980Z] Executing:   0%|          | 0/32 [00:00<?, ?cell/s]
[2025-09-16T21:26:01.477Z] Executing:   3%|▎         | 1/32 [00:00<00:27,  1.11cell/s]
[2025-09-16T21:26:14.010Z] Executing:   9%|▉         | 3/32 [00:19<03:21,  6.95s/cell]
[2025-09-16T21:39:22.209Z] Executing:  16%|█▌        | 5/32 [00:30<02:51,  6.37s/cell]
[2025-09-16T21:40:09.192Z] Executing:  38%|███▊      | 12/32 [13:30<27:19, 81.95s/cell]
[2025-09-16T21:40:09.464Z] Executing:  44%|████▍     | 14/32 [14:27<20:56, 69.83s/cell]
[2025-09-16T21:40:12.780Z] Executing:  50%|█████     | 16/32 [14:27<14:03, 52.74s/cell]
[2025-09-16T21:40:27.806Z] Executing:  56%|█████▋    | 18/32 [14:30<09:12, 39.47s/cell]
[2025-09-16T21:40:27.806Z] Executing:  62%|██████▎   | 20/32 [14:43<06:07, 30.60s/cell]
[2025-09-16T21:40:30.448Z] Executing:  75%|███████▌  | 24/32 [14:45<02:19, 17.45s/cell]
[2025-09-16T21:40:30.448Z] Executing:  75%|███████▌  | 24/32 [14:48<04:56, 37.01s/cell]
[2025-09-16T21:40:30.734Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2025-09-16T21:40:30.734Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2025-09-16T21:40:30.734Z] Traceback (most recent call last):
[2025-09-16T21:40:30.734Z]   File "/usr/local/bin/papermill", line 7, in <module>
[2025-09-16T21:40:30.734Z]     sys.exit(papermill())
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2025-09-16T21:40:30.734Z]     return self.main(*args, **kwargs)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2025-09-16T21:40:30.734Z]     rv = self.invoke(ctx)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2025-09-16T21:40:30.734Z]     return ctx.invoke(self.callback, **ctx.params)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2025-09-16T21:40:30.734Z]     return __callback(*args, **kwargs)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2025-09-16T21:40:30.734Z]     return f(get_current_context(), *args, **kwargs)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill
[2025-09-16T21:40:30.734Z]     execute_notebook(
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook
[2025-09-16T21:40:30.734Z]     raise_for_execution_errors(nb, output_path)
[2025-09-16T21:40:30.734Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors
[2025-09-16T21:40:30.734Z]     raise error
[2025-09-16T21:40:30.734Z] papermill.exceptions.PapermillExecutionError: 
[2025-09-16T21:40:30.734Z] ---------------------------------------------------------------------------
[2025-09-16T21:40:30.734Z] Exception encountered at "In [12]":
[2025-09-16T21:40:30.734Z] ---------------------------------------------------------------------------
[2025-09-16T21:40:30.734Z] RuntimeError                              Traceback (most recent call last)
[2025-09-16T21:40:30.734Z] Cell In[12], line 28
[2025-09-16T21:40:30.734Z]      26 # Get model prediction
[2025-09-16T21:40:30.734Z]      27 noisy_img = scheduler.add_noise(images, noise, timesteps)  # add t steps of noise to the input image
[2025-09-16T21:40:30.734Z] ---> 28 pred = classifier(noisy_img, timesteps)
[2025-09-16T21:40:30.734Z]      30 loss = F.cross_entropy(pred, classes.long())
[2025-09-16T21:40:30.734Z]      32 loss.backward()
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1734     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
[2025-09-16T21:40:30.734Z]    1735 else:
[2025-09-16T21:40:30.734Z] -> 1736     return self._call_impl(*args, **kwargs)
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1742 # If we don't have any hooks, we want to skip the rest of the logic in
[2025-09-16T21:40:30.734Z]    1743 # this function, and just call forward.
[2025-09-16T21:40:30.734Z]    1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
[2025-09-16T21:40:30.734Z]    1745         or _global_backward_pre_hooks or _global_backward_hooks
[2025-09-16T21:40:30.734Z]    1746         or _global_forward_hooks or _global_forward_pre_hooks):
[2025-09-16T21:40:30.734Z] -> 1747     return forward_call(*args, **kwargs)
[2025-09-16T21:40:30.734Z]    1749 result = None
[2025-09-16T21:40:30.734Z]    1750 called_always_called_hooks = set()
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/networks/nets/diffusion_model_unet.py:2059, in DiffusionModelEncoder.forward(self, x, timesteps, context, class_labels)
[2025-09-16T21:40:30.734Z]    2055 if self.out is None:
[2025-09-16T21:40:30.734Z]    2056     self.out = nn.Sequential(
[2025-09-16T21:40:30.734Z]    2057         nn.Linear(h.shape[1], 512), nn.ReLU(), nn.Dropout(0.1), nn.Linear(512, self.out_channels)
[2025-09-16T21:40:30.734Z]    2058     )
[2025-09-16T21:40:30.734Z] -> 2059 output: torch.Tensor = self.out(h)
[2025-09-16T21:40:30.734Z]    2061 return output
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1734     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
[2025-09-16T21:40:30.734Z]    1735 else:
[2025-09-16T21:40:30.734Z] -> 1736     return self._call_impl(*args, **kwargs)
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1742 # If we don't have any hooks, we want to skip the rest of the logic in
[2025-09-16T21:40:30.734Z]    1743 # this function, and just call forward.
[2025-09-16T21:40:30.734Z]    1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
[2025-09-16T21:40:30.734Z]    1745         or _global_backward_pre_hooks or _global_backward_hooks
[2025-09-16T21:40:30.734Z]    1746         or _global_forward_hooks or _global_forward_pre_hooks):
[2025-09-16T21:40:30.734Z] -> 1747     return forward_call(*args, **kwargs)
[2025-09-16T21:40:30.734Z]    1749 result = None
[2025-09-16T21:40:30.734Z]    1750 called_always_called_hooks = set()
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py:250, in Sequential.forward(self, input)
[2025-09-16T21:40:30.734Z]     248 def forward(self, input):
[2025-09-16T21:40:30.734Z]     249     for module in self:
[2025-09-16T21:40:30.734Z] --> 250         input = module(input)
[2025-09-16T21:40:30.734Z]     251     return input
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1734     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
[2025-09-16T21:40:30.734Z]    1735 else:
[2025-09-16T21:40:30.734Z] -> 1736     return self._call_impl(*args, **kwargs)
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
[2025-09-16T21:40:30.734Z]    1742 # If we don't have any hooks, we want to skip the rest of the logic in
[2025-09-16T21:40:30.734Z]    1743 # this function, and just call forward.
[2025-09-16T21:40:30.734Z]    1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
[2025-09-16T21:40:30.734Z]    1745         or _global_backward_pre_hooks or _global_backward_hooks
[2025-09-16T21:40:30.734Z]    1746         or _global_forward_hooks or _global_forward_pre_hooks):
[2025-09-16T21:40:30.734Z] -> 1747     return forward_call(*args, **kwargs)
[2025-09-16T21:40:30.734Z]    1749 result = None
[2025-09-16T21:40:30.734Z]    1750 called_always_called_hooks = set()
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py:125, in Linear.forward(self, input)
[2025-09-16T21:40:30.734Z]     124 def forward(self, input: Tensor) -> Tensor:
[2025-09-16T21:40:30.734Z] --> 125     return F.linear(input, self.weight, self.bias)
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/meta_tensor.py:283, in MetaTensor.__torch_function__(cls, func, types, args, kwargs)
[2025-09-16T21:40:30.734Z]     281 if kwargs is None:
[2025-09-16T21:40:30.734Z]     282     kwargs = {}
[2025-09-16T21:40:30.734Z] --> 283 ret = super().__torch_function__(func, types, args, kwargs)
[2025-09-16T21:40:30.734Z]     284 # if `out` has been used as argument, metadata is not copied, nothing to do.
[2025-09-16T21:40:30.734Z]     285 # if "out" in kwargs:
[2025-09-16T21:40:30.734Z]     286 #     return ret
[2025-09-16T21:40:30.734Z]     287 if _not_requiring_metadata(ret):
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] File /usr/local/lib/python3.10/dist-packages/torch/_tensor.py:1453, in Tensor.__torch_function__(cls, func, types, args, kwargs)
[2025-09-16T21:40:30.734Z]    1450     return NotImplemented
[2025-09-16T21:40:30.734Z]    1452 with _C.DisableTorchFunctionSubclass():
[2025-09-16T21:40:30.734Z] -> 1453     ret = func(*args, **kwargs)
[2025-09-16T21:40:30.734Z]    1454     if func in get_default_nowrap_functions():
[2025-09-16T21:40:30.734Z]    1455         return ret
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] 
[2025-09-16T21:40:30.734Z] real	14m49.487s
[2025-09-16T21:40:30.734Z] user	14m4.541s
[2025-09-16T21:40:30.734Z] sys	6m29.186s

[ericspod]

This is a result of the forward pass in DiffusionModelEncoder instantiating trainable layers. When this happens the new layers are created on CPU rather than the device of some parameter in the network. The created layers are not going to be optimised since these won't be pass as part of the parameter set to any optimiser. This change was introduced with Project-MONAI/MONAI#8514

@virginiafdez could you please have a look at this? There should be some way of figuring out what the h.shape[1] value will be in the constructor ahead of time and construct the linear layer there, otherwise the value will have to be constructor parameter and the user will have to figure it out.