Invertd in post_transforms composition RuntimeError

[dianemarquette]

Hi!

I've been trying to get Invertd working with my code. I used a similar post transforms composition to the one provided in unet_inference_dict.py. Unfortunately, I get the following errors and I'm a bit stuck.

Traceback (most recent call last):
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/transform.py", line 49, in apply_transform
    return transform(data)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/croppad/dictionary.py", line 777, in inverse
    box_start = np.asarray(extra_info["box_start"])
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/torch/tensor.py", line 621, in __array__
    return self.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/inverse_batch_transform.py", line 92, in __call__
    return first(inv_loader)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/utils/misc.py", line 71, in first
    for i in iterable:
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/inverse_batch_transform.py", line 48, in __getitem__
    return self.invertible_transform.inverse(data)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/compose.py", line 165, in inverse
    data = apply_transform(t.inverse, data, self.map_items)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/transform.py", line 73, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <bound method CropForegroundd.inverse of <monai.transforms.croppad.dictionary.CropForegroundd object at 0x7fd9e6e3be50>>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/transform.py", line 49, in apply_transform
    return transform(data)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 566, in __call__
    inverted = self.inverter(input_dict)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/inverse_batch_transform.py", line 97, in __call__
    raise RuntimeError(re_str)
RuntimeError: applying transform <bound method CropForegroundd.inverse of <monai.transforms.croppad.dictionary.CropForegroundd object at 0x7fd9e6e3be50>>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "infer_keri.py", line 37, in <module>
    infer()
  File "infer_keri.py", line 33, in infer
    trainer.predict(model=trained_model, datamodule=data)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 631, in predict
    results = self._run(model)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 795, in dispatch
    self.accelerator.start_predicting(self)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 102, in start_predicting
    self.training_type_plugin.start_predicting(trainer)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 152, in start_predicting
    self._results = trainer.run_stage()
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 806, in run_stage
    return self.run_predict()
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1083, in run_predict
    self.predict_loop.predict_step(batch, batch_idx, dataloader_idx)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/trainer/predict_loop.py", line 111, in predict_step
    predictions = self.trainer.accelerator.predict_step(args)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 265, in predict_step
    return self.training_type_plugin.predict_step(*args)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 167, in predict_step
    return self.lightning_module.predict_step(*args, **kwargs)
  File "/media/diane/Shared Data/KeriNet/src/models/keri_model.py", line 133, in predict_step
    self.trainer.datamodule.post_transforms(batch)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/compose.py", line 155, in __call__
    input_ = apply_transform(_transform, input_, self.map_items)
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/monai/transforms/transform.py", line 73, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.Invertd object at 0x7fd9e56f1520>
Exception ignored in: <function tqdm.__del__ at 0x7fda200480d0>
Traceback (most recent call last):
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/tqdm/std.py", line 1145, in __del__
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/tqdm/std.py", line 1299, in close
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/tqdm/std.py", line 1492, in display
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/tqdm/std.py", line 1148, in __str__
  File "/home/diane/miniconda3/envs/lightning-hydra/lib/python3.8/site-packages/tqdm/std.py", line 1450, in format_dict
TypeError: cannot unpack non-iterable NoneType object

Any tip or advice would be greatly appreciated :)

[Nic-Ma]

Hi @dianemarquette ,

Could you please help share your test program to reproduce the issue?
I feel maybe you moved unexpected data to GPU, but sure, we may need to enhance the logic to make Invertd robust.

Thanks.

[Nic-Ma]

submitted PR to enhance the doc: #2358.

Thanks.

[dianemarquette]

@Nic-Ma The code you suggested works great ! Thanks for your help. I'm now able to run the post transforms smoothly 🎉

[Nic-Ma]

Sounds great!

[rijobro]

@dianemarquette Just FYI, we're also working on have our transformations support torch.Tensor by default (and therefore be we'd be ambivalent to CPU/GPU data): #2231.

[dianemarquette]

@rijobro Thanks for the information! I'll be keeping an eye on this pull request :)