testing error modules/transform_visualization.ipynb

[wyli]

Describe the bug

[2023-01-10T15:57:42.182Z] Running ./modules/transform_visualization.ipynb
[2023-01-10T15:57:42.182Z] Checking PEP8 compliance...
[2023-01-10T15:57:42.182Z] .*federated_learning*
[2023-01-10T15:57:42.182Z] .*transchex_openi*
[2023-01-10T15:57:42.182Z] .*unetr_*
[2023-01-10T15:57:42.182Z] .*profiling_train_base_nvtx*
[2023-01-10T15:57:42.182Z] .*benchmark_global_mutual_information*
[2023-01-10T15:57:42.182Z] .*spleen_segmentation_3d_visualization_basic*
[2023-01-10T15:57:42.182Z] .*full_gpu_inference_pipeline*
[2023-01-10T15:57:42.182Z] .*generate_random_permutations*
[2023-01-10T15:57:42.182Z] .*transforms_update_meta_data*
[2023-01-10T15:57:42.182Z] .*video_seg*
[2023-01-10T15:57:42.182Z] .*tcia_dataset*
[2023-01-10T15:57:42.182Z] .*hovernet_torch*
[2023-01-10T15:57:42.182Z] .*preprocess_detect_scene_and_split_fold*
[2023-01-10T15:57:42.182Z] .*preprocess_to_build_detection_dataset*
[2023-01-10T15:57:42.182Z] .*preprocess_extract_images_from_video*
[2023-01-10T15:57:42.182Z] .*transfer_mmar*
[2023-01-10T15:57:42.182Z] .*MRI_reconstruction*
[2023-01-10T15:57:42.182Z] .*monailabel_HelloWorld_radiology_3dslicer*
[2023-01-10T15:57:42.182Z] .*monailabel_monaibundle_3dslicer_multiorgan_seg*
[2023-01-10T15:57:42.182Z] .*monailabel_pancreas_tumor_segmentation_3DSlicer*
[2023-01-10T15:57:42.182Z] .*monailabel_endoscopy_cvat_tooltracking*
[2023-01-10T15:57:42.182Z] .*monailabel_pathology_nuclei_segmentation_QuPath*
[2023-01-10T15:57:42.182Z] .*monailabel_radiology_spleen_segmentation_OHIF*
[2023-01-10T15:57:42.182Z] Running notebook...
[2023-01-10T15:57:47.426Z] MONAI version: 0.8.1+823.gea06e1ae
[2023-01-10T15:57:47.426Z] Numpy version: 1.22.2
[2023-01-10T15:57:47.426Z] Pytorch version: 1.10.2+cu102
[2023-01-10T15:57:47.426Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-01-10T15:57:47.426Z] MONAI rev id: ea06e1ae1551c0c6e22a804f1094385371c5f6ff
[2023-01-10T15:57:47.426Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-01-10T15:57:47.426Z] 
[2023-01-10T15:57:47.426Z] Optional dependencies:
[2023-01-10T15:57:47.426Z] Pytorch Ignite version: 0.4.10
[2023-01-10T15:57:47.426Z] Nibabel version: 5.0.0
[2023-01-10T15:57:47.426Z] scikit-image version: 0.19.3
[2023-01-10T15:57:47.426Z] Pillow version: 7.0.0
[2023-01-10T15:57:47.426Z] Tensorboard version: 2.9.0
[2023-01-10T15:57:47.426Z] gdown version: 4.6.0
[2023-01-10T15:57:47.426Z] TorchVision version: 0.11.3+cu102
[2023-01-10T15:57:47.426Z] tqdm version: 4.64.1
[2023-01-10T15:57:47.426Z] lmdb version: 1.4.0
[2023-01-10T15:57:47.426Z] psutil version: 5.9.4
[2023-01-10T15:57:47.426Z] pandas version: 1.1.5
[2023-01-10T15:57:47.426Z] einops version: 0.6.0
[2023-01-10T15:57:47.426Z] transformers version: 4.21.3
[2023-01-10T15:57:47.426Z] mlflow version: 2.1.1
[2023-01-10T15:57:47.426Z] pynrrd version: 1.0.0
[2023-01-10T15:57:47.426Z] 
[2023-01-10T15:57:47.426Z] For details about installing the optional dependencies, please visit:
[2023-01-10T15:57:47.426Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-01-10T15:57:47.426Z] 
[2023-01-10T15:57:48.788Z] /usr/local/lib/python3.8/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-01-10T15:57:48.788Z]   warnings.warn(
[2023-01-10T15:59:30.984Z] 
Executing:   0%|          | 0/38 [00:00<?, ?cell/s]
Executing:   3%|▎         | 1/38 [00:01<00:48,  1.31s/cell]
Executing:   8%|▊         | 3/38 [00:02<00:20,  1.67cell/s]
Executing:  11%|█         | 4/38 [00:09<01:45,  3.09s/cell]
Executing:  13%|█▎        | 5/38 [00:39<06:35, 11.98s/cell]
Executing:  18%|█▊        | 7/38 [00:43<03:39,  7.09s/cell]
Executing:  32%|███▏      | 12/38 [01:26<03:30,  8.08s/cell]
Executing:  53%|█████▎    | 20/38 [01:29<01:02,  3.48s/cell]
Executing:  58%|█████▊    | 22/38 [01:41<01:02,  3.93s/cell]2023-01-10 15:59:30.041346: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2023-01-10T15:59:30.984Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2023-01-10T15:59:30.984Z] 2023-01-10 15:59:30.203670: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2023-01-10T15:59:30.984Z] 2023-01-10 15:59:30.942909: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/torch/lib:/usr/local/lib/python3.8/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2023-01-10T15:59:30.984Z] 2023-01-10 15:59:30.943033: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/torch/lib:/usr/local/lib/python3.8/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2023-01-10T15:59:30.984Z] 2023-01-10 15:59:30.943041: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2023-01-10T15:59:41.147Z] 
Executing:  63%|██████▎   | 24/38 [01:43<00:47,  3.36s/cell]
Executing:  66%|██████▌   | 25/38 [01:47<00:43,  3.38s/cell]
Executing:  74%|███████▎  | 28/38 [01:49<00:24,  2.45s/cell]
Executing:  74%|███████▎  | 28/38 [01:52<00:40,  4.00s/cell]
[2023-01-10T15:59:41.147Z] /usr/local/lib/python3.8/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-01-10T15:59:41.147Z]   warnings.warn(
[2023-01-10T15:59:41.147Z] Traceback (most recent call last):
[2023-01-10T15:59:41.147Z]   File "/usr/local/bin/papermill", line 8, in <module>
[2023-01-10T15:59:41.147Z]     sys.exit(papermill())
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
[2023-01-10T15:59:41.147Z]     return self.main(*args, **kwargs)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
[2023-01-10T15:59:41.147Z]     rv = self.invoke(ctx)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
[2023-01-10T15:59:41.147Z]     return ctx.invoke(self.callback, **ctx.params)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
[2023-01-10T15:59:41.147Z]     return __callback(*args, **kwargs)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 26, in new_func
[2023-01-10T15:59:41.147Z]     return f(get_current_context(), *args, **kwargs)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/cli.py", line 250, in papermill
[2023-01-10T15:59:41.147Z]     execute_notebook(
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-01-10T15:59:41.147Z]     raise_for_execution_errors(nb, output_path)
[2023-01-10T15:59:41.147Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-01-10T15:59:41.147Z]     raise error
[2023-01-10T15:59:41.147Z] papermill.exceptions.PapermillExecutionError: 
[2023-01-10T15:59:41.147Z] ---------------------------------------------------------------------------
[2023-01-10T15:59:41.147Z] Exception encountered at "In [15]":
[2023-01-10T15:59:41.147Z] ---------------------------------------------------------------------------
[2023-01-10T15:59:41.147Z] TraitError                                Traceback (most recent call last)
[2023-01-10T15:59:41.147Z] Cell In[15], line 1
[2023-01-10T15:59:41.147Z] ----> 1 view(image=data["image"][0, 0, :, :, :] * 255, label_image=data["label"][0, 0, :, :, :] * 255, gradient_opacity=0.4)
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/itkwidgets/widget_viewer.py:1090, in view(image, label_image, label_image_names, label_image_weights, label_image_blend, cmap, lut, select_roi, interpolation, gradient_opacity, opacity_gaussians, channels, slicing_planes, shadow, blend_mode, point_sets, point_set_colors, point_set_opacities, point_set_representations, point_set_sizes, geometries, geometry_colors, geometry_opacities, ui_collapsed, rotate, annotations, axes, mode, **kwargs)
[2023-01-10T15:59:41.147Z]    1087     if image is None and len(images):  # only one image is rendered
[2023-01-10T15:59:41.147Z]    1088         image = images[0]
[2023-01-10T15:59:41.147Z] -> 1090 viewer = Viewer(image=image,
[2023-01-10T15:59:41.147Z]    1091                 label_image=label_image,
[2023-01-10T15:59:41.147Z]    1092                 label_image_names=label_image_names,
[2023-01-10T15:59:41.147Z]    1093                 label_image_blend=label_image_blend,
[2023-01-10T15:59:41.147Z]    1094                 label_image_weights=label_image_weights,
[2023-01-10T15:59:41.147Z]    1095                 cmap=cmap,
[2023-01-10T15:59:41.147Z]    1096                 lut=lut,
[2023-01-10T15:59:41.147Z]    1097                 select_roi=select_roi,
[2023-01-10T15:59:41.147Z]    1098                 interpolation=interpolation,
[2023-01-10T15:59:41.147Z]    1099                 gradient_opacity=gradient_opacity,
[2023-01-10T15:59:41.147Z]    1100                 opacity_gaussians=opacity_gaussians,
[2023-01-10T15:59:41.147Z]    1101                 slicing_planes=slicing_planes,
[2023-01-10T15:59:41.147Z]    1102                 shadow=shadow, blend_mode=blend_mode,
[2023-01-10T15:59:41.147Z]    1103                 point_sets=point_sets,
[2023-01-10T15:59:41.147Z]    1104                 point_set_colors=point_set_colors,
[2023-01-10T15:59:41.147Z]    1105                 point_set_opacities=point_set_opacities,
[2023-01-10T15:59:41.147Z]    1106                 point_set_representations=point_set_representations,
[2023-01-10T15:59:41.147Z]    1107                 point_set_sizes=point_set_sizes,
[2023-01-10T15:59:41.147Z]    1108                 geometries=geometries, geometry_colors=geometry_colors,
[2023-01-10T15:59:41.147Z]    1109                 geometry_opacities=geometry_opacities,
[2023-01-10T15:59:41.147Z]    1110                 rotate=rotate, ui_collapsed=ui_collapsed,
[2023-01-10T15:59:41.147Z]    1111                 annotations=annotations, axes=axes, mode=mode,
[2023-01-10T15:59:41.147Z]    1112                 **kwargs)
[2023-01-10T15:59:41.147Z]    1113 return viewer
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/itkwidgets/widget_viewer.py:369, in Viewer.__init__(self, **kwargs)
[2023-01-10T15:59:41.147Z]     365 if have_label_image:
[2023-01-10T15:59:41.147Z]     366     # Interpolation is not currently supported with label maps
[2023-01-10T15:59:41.147Z]     367     kwargs['interpolation'] = False
[2023-01-10T15:59:41.147Z] --> 369 super(Viewer, self).__init__(**kwargs)
[2023-01-10T15:59:41.147Z]     371 if not self.image and not self.label_image:
[2023-01-10T15:59:41.147Z]     372     return
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/ipywidgets/widgets/widget.py:477, in Widget.__init__(self, **kwargs)
[2023-01-10T15:59:41.147Z]     475 """Public constructor"""
[2023-01-10T15:59:41.147Z]     476 self._model_id = kwargs.pop('model_id', None)
[2023-01-10T15:59:41.147Z] --> 477 super(Widget, self).__init__(**kwargs)
[2023-01-10T15:59:41.147Z]     479 Widget._call_widget_constructed(self)
[2023-01-10T15:59:41.147Z]     480 self.open()
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py:1358, in __init__(self, *args, **kwargs)
[2023-01-10T15:59:41.147Z]    1356 if change.type == "change":
[2023-01-10T15:59:41.147Z]    1357     if change.old is not Undefined:
[2023-01-10T15:59:41.147Z] -> 1358         self.set_trait(name, change.old)
[2023-01-10T15:59:41.147Z]    1359     else:
[2023-01-10T15:59:41.147Z]    1360         self._trait_values.pop(name)
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py:1738, in set_trait(self, name, value)
[2023-01-10T15:59:41.147Z]    1736     if name in c.__dict__.get("_trait_default_generators", {}):
[2023-01-10T15:59:41.147Z]    1737         return c._trait_default_generators[name]  # type:ignore[attr-defined]
[2023-01-10T15:59:41.147Z] -> 1738 return trait.default
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traittypes/traittypes.py:108, in Array.set(self, obj, value)
[2023-01-10T15:59:41.147Z]     107 def set(self, obj, value):
[2023-01-10T15:59:41.147Z] --> 108     new_value = self._validate(obj, value)
[2023-01-10T15:59:41.147Z]     109     old_value = obj._trait_values.get(self.name, self.default_value)
[2023-01-10T15:59:41.147Z]     110     obj._trait_values[self.name] = new_value
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py:735, in _validate(self, obj, value)
[2023-01-10T15:59:41.147Z]     731     meth_name = "_%s_validate" % self.name
[2023-01-10T15:59:41.147Z]     732     cross_validate = getattr(obj, meth_name)
[2023-01-10T15:59:41.147Z]     733     _deprecated_method(
[2023-01-10T15:59:41.147Z]     734         cross_validate,
[2023-01-10T15:59:41.147Z] --> 735         obj.__class__,
[2023-01-10T15:59:41.147Z]     736         meth_name,
[2023-01-10T15:59:41.147Z]     737         "use @validate decorator instead.",
[2023-01-10T15:59:41.147Z]     738     )
[2023-01-10T15:59:41.147Z]     739     value = cross_validate(value, self)
[2023-01-10T15:59:41.147Z]     740 return value
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/ipydatawidgets/ndarray/traits.py:25, in NDArray.validate(self, obj, value)
[2023-01-10T15:59:41.147Z]      24 def validate(self, obj, value):
[2023-01-10T15:59:41.147Z] ---> 25     value = super(NDArray, self).validate(obj, value)
[2023-01-10T15:59:41.147Z]      26     if value is None or value is Undefined:
[2023-01-10T15:59:41.147Z]      27         return value
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traittypes/traittypes.py:105, in Array.validate(self, obj, value)
[2023-01-10T15:59:41.147Z]     103 except (ValueError, TypeError) as e:
[2023-01-10T15:59:41.147Z]     104     raise TraitError(e)
[2023-01-10T15:59:41.147Z] --> 105 return super(Array, self).validate(obj, value)
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/traittypes/traittypes.py:76, in SciType.validate(self, obj, value)
[2023-01-10T15:59:41.147Z]      74 try:
[2023-01-10T15:59:41.147Z]      75     for validator in self.validators:
[2023-01-10T15:59:41.147Z] ---> 76         value = validator(self, value)
[2023-01-10T15:59:41.147Z]      77     return value
[2023-01-10T15:59:41.147Z]      78 except (ValueError, TypeError) as e:
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] File /usr/local/lib/python3.8/dist-packages/ipydatawidgets/ndarray/traits.py:40, in shape_constraints.<locals>.validator(trait, value)
[2023-01-10T15:59:41.147Z]      38     return value
[2023-01-10T15:59:41.147Z]      39 if len(value.shape) != len(args):
[2023-01-10T15:59:41.147Z] ---> 40     raise TraitError('%s shape expected to have %s components, but got %s components' % (
[2023-01-10T15:59:41.147Z]      41         trait.name, len(args), value.shape))
[2023-01-10T15:59:41.147Z]      42 for i, constraint in enumerate(args):
[2023-01-10T15:59:41.147Z]      43     if constraint is not None:
[2023-01-10T15:59:41.147Z] 
[2023-01-10T15:59:41.147Z] TraitError: label_image_weights shape expected to have 1 components, but got () components

cc @Nic-Ma @KumoLiu

[KumoLiu]

Hi @wyli, I have checked this file in MONAI 1.1.0, and everything works well. I just noticed your MONAI version is 0.8.1 from the error message you posted. Do I need to verify this file in that version?
Thanks!

[wyli]

thanks for checking, looks like it's a versioning issue and it works fine today, I'm closing this for now.

[wyli]

this is still an issue, one possible reason is that https://github.com/Project-MONAI/tutorials/blob/main/modules/TorchIO_MONAI_PyTorch_Lightning.ipynb changes the pytorch version to 1.10.2

[wyli]

could you please help test with the latest docker image? it seems the issue happens after upgrading the base image to 22.12 Project-MONAI/MONAI#5820 @KumoLiu

[KumoLiu]

could you please help test with the latest docker image? it seems the issue happens after upgrading the base image to 22.12 Project-MONAI/MONAI#5820 @KumoLiu

Hi @wyli, I have tested it on the latest image(docker pull projectmonai/monai:latest), and it works well. I'm not sure what did you mean it may be affected by https://github.com/Project-MONAI/tutorials/blob/main/modules/TorchIO_MONAI_PyTorch_Lightning.ipynb. But I also tested it on the 1.10.2 version of pytorch and it also works well.

[wyli]

I'm able to reproduce it with the latest docker projectmonai/monai:latest

and run these commands in the tutorials/ source code folder:

pip install -r requirements.txt
./runner.sh -t modules/transform_visualization.ipynb

[KumoLiu]

Hi @wyli, based on your command I can reproduce this error, I investigated a bit and found that there is a compatibility issue due to the version of traitlets. Here is a known issue that explains the problem, we can add traitlets<5.7.0 in the requirements.txt to solve this error. (just a workaround) Do I need to create a PR to fix it?

[wyli]

Thanks, that's probably the issue and the ticket you linked has been closed by a workaround. Is the workaround not working? We can skip the test of this notebook.

[KumoLiu]

It works, but since the version of the traitlets in the latest image is 5.7.1 so the test will run failed. If you run this command ./runner.sh -t modules/transform_visualization.ipynb twice, it will update the version of the traitlets and the test will pass in the second time.

[wyli]

I think we skip the test and keep this ticket open until the compatibility improves #1164