test error auto3dseg_autorunner_ref_api

[wyli]

[2022-10-02T01:11:31.866Z] Running ./auto3dseg/notebooks/auto3dseg_autorunner_ref_api.ipynb
[2022-10-02T01:11:32.125Z] Checking PEP8 compliance...
[2022-10-02T01:11:32.690Z] Running notebook...
[2022-10-02T01:11:32.690Z] Before:
[2022-10-02T01:11:32.690Z]     "max_epochs = 2000\n",
[2022-10-02T01:11:32.690Z] After:
[2022-10-02T01:11:32.690Z]     "max_epochs = 1\n",
[2022-10-02T01:11:36.872Z] MONAI version: 1.0.0+22.gffaa791c
[2022-10-02T01:11:36.872Z] Numpy version: 1.22.4
[2022-10-02T01:11:36.872Z] Pytorch version: 1.10.2+cu102
[2022-10-02T01:11:36.872Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-10-02T01:11:36.872Z] MONAI rev id: ffaa791c9e766d05623c283e4abefc1ef186f7c9
[2022-10-02T01:11:36.872Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-10-02T01:11:36.872Z] 
[2022-10-02T01:11:36.872Z] Optional dependencies:
[2022-10-02T01:11:36.872Z] Pytorch Ignite version: 0.4.10
[2022-10-02T01:11:36.872Z] Nibabel version: 4.0.2
[2022-10-02T01:11:36.872Z] scikit-image version: 0.19.3
[2022-10-02T01:11:36.872Z] Pillow version: 7.0.0
[2022-10-02T01:11:36.872Z] Tensorboard version: 2.10.1
[2022-10-02T01:11:36.872Z] gdown version: 4.5.1
[2022-10-02T01:11:36.872Z] TorchVision version: 0.11.3+cu102
[2022-10-02T01:11:36.872Z] tqdm version: 4.64.0
[2022-10-02T01:11:36.872Z] lmdb version: 1.3.0
[2022-10-02T01:11:36.872Z] psutil version: 5.9.1
[2022-10-02T01:11:36.872Z] pandas version: 1.1.5
[2022-10-02T01:11:36.872Z] einops version: 0.4.1
[2022-10-02T01:11:36.872Z] transformers version: 4.21.3
[2022-10-02T01:11:36.872Z] mlflow version: 1.29.0
[2022-10-02T01:11:36.872Z] pynrrd version: 0.4.3
[2022-10-02T01:11:36.872Z] 
[2022-10-02T01:11:36.872Z] For details about installing the optional dependencies, please visit:
[2022-10-02T01:11:36.872Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-10-02T01:11:36.872Z] 
[2022-10-02T01:11:38.244Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-10-02T01:11:38.244Z]   warnings.warn(
[2022-10-02T06:08:28.269Z] 
Executing:   0%|          | 0/24 [00:00<?, ?cell/s]
Executing:   4%|▍         | 1/24 [00:01<00:34,  1.48s/cell]
Executing:  12%|█▎        | 3/24 [00:06<00:45,  2.15s/cell]
Executing:  21%|██        | 5/24 [00:09<00:38,  2.01s/cell]
Executing:  29%|██▉       | 7/24 [01:06<03:44, 13.22s/cell]
Executing:  46%|████▌     | 11/24 [04:06<06:37, 30.61s/cell]
Executing:  58%|█████▊    | 14/24 [04:16<03:24, 20.48s/cell]Sending interrupt signal to process
[2022-10-02T06:08:28.271Z] Killing processes
[2022-10-02T06:08:28.654Z] kill finished with exit code 0

[mingxin-zheng]

Is it from the CI/CD? It may be related to this issue: Project-MONAI/MONAI#5277

I'll start the investigation and seek for more info when needed. Thanks

[wyli]

yes, it just takes too much time to run the notebook (more than 5h). I don't see any other error messages.

[wyli]

this is still an issue

[2022-10-18T16:26:11.809Z] Running ./auto3dseg/notebooks/auto3dseg_autorunner_ref_api.ipynb
[2022-10-18T16:26:11.809Z] Checking PEP8 compliance...
[2022-10-18T16:26:12.739Z] Running notebook...
[2022-10-18T16:26:12.739Z] Before:
[2022-10-18T16:26:12.739Z]     "max_epochs = 2  # change epoch number to 2 to cut down the notebook running time\n",
[2022-10-18T16:26:12.739Z] After:
[2022-10-18T16:26:12.739Z]     "max_epochs = 1  # change epoch number to 2 to cut down the notebook running time\n",
[2022-10-18T16:26:16.911Z] MONAI version: 1.0.0+53.gcd2f4e15
[2022-10-18T16:26:16.911Z] Numpy version: 1.22.2
[2022-10-18T16:26:16.911Z] Pytorch version: 1.10.2+cu102
[2022-10-18T16:26:16.911Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-10-18T16:26:16.911Z] MONAI rev id: cd2f4e15a050f2ff1abc66bf19d499ce3cd58a31
[2022-10-18T16:26:16.911Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-10-18T16:26:16.911Z] 
[2022-10-18T16:26:16.911Z] Optional dependencies:
[2022-10-18T16:26:16.911Z] Pytorch Ignite version: 0.4.10
[2022-10-18T16:26:16.911Z] Nibabel version: 4.0.2
[2022-10-18T16:26:16.911Z] scikit-image version: 0.19.3
[2022-10-18T16:26:16.911Z] Pillow version: 7.0.0
[2022-10-18T16:26:16.911Z] Tensorboard version: 2.10.0
[2022-10-18T16:26:16.911Z] gdown version: 4.5.3
[2022-10-18T16:26:16.911Z] TorchVision version: 0.11.3+cu102
[2022-10-18T16:26:16.911Z] tqdm version: 4.64.1
[2022-10-18T16:26:16.911Z] lmdb version: 1.3.0
[2022-10-18T16:26:16.911Z] psutil version: 5.9.2
[2022-10-18T16:26:16.911Z] pandas version: 1.1.5
[2022-10-18T16:26:16.911Z] einops version: 0.5.0
[2022-10-18T16:26:16.911Z] transformers version: 4.21.3
[2022-10-18T16:26:16.911Z] mlflow version: 1.29.0
[2022-10-18T16:26:16.911Z] pynrrd version: 1.0.0
[2022-10-18T16:26:16.911Z] 
[2022-10-18T16:26:16.911Z] For details about installing the optional dependencies, please visit:
[2022-10-18T16:26:16.911Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-10-18T16:26:16.911Z] 
[2022-10-18T16:26:17.843Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-10-18T16:26:17.843Z]   warnings.warn(

let's skip this test

[wyli]

Hi @mingxin-zheng, the dataset downloading appears to be slow, would be great for those notebooks to share the same copy of the downloaded dataset...

[mingxin-zheng]

@wyli I am wondering why the download is slow since the Task04 is 27MB only.

Anyway I will check if all the notebooks share the same copy.

[wyli]

I see, it seems all tests passed twice and the overall notebook testing is within 3h. there's no issue with it, I'm closing this now...