CalledProcessError

[y070504]

CalledProcessError Traceback (most recent call last)
File ~/miniconda3/lib/python3.9/site-packages/monai/apps/auto3dseg/bundle_gen.py:183, in BundleAlgo._run_cmd(self, cmd, devices_info)
182 ps_environ["CUDA_VISIBLE_DEVICES"] = devices_info
--> 183 normal_out = subprocess.run(cmd.split(), env=ps_environ, check=True, capture_output=True)
184 logger.info(repr(normal_out).replace("\n", "\n").replace("\t", "\t"))

File ~/miniconda3/lib/python3.9/subprocess.py:528, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
527 if check and retcode:
--> 528 raise CalledProcessError(retcode, process.args,
529 output=stdout, stderr=stderr)
530 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['python', '/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/scripts/train.py', 'run', "--config_file='/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/configs/network.yaml','/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/configs/hyper_parameters.yaml','/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/configs/transforms_train.yaml','/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/configs/transforms_infer.yaml','/deeplearning/auto3dseg/notebooks/work_dir/segresnet_1/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
/deeplearning/auto3dseg/notebooks/auto_runner.ipynb Cell 10 in <cell line: 2>()
1 runner = AutoRunner(input=input)
----> 2 runner.run()

File ~/miniconda3/lib/python3.9/site-packages/monai/apps/auto3dseg/auto_runner.py:586, in AutoRunner.run(self)
584 history = import_bundle_algo_history(self.work_dir, only_trained=False)
585 if not self.hpo:
...
[info] amp enabled

epoch 1/1250
learning rate is set to 2e-05

[Nic-Ma]

Hi @mingxin-zheng ,

Could you please help take a look at this error log?

Thanks in advance.

[mingxin-zheng]

Hi @y070504 , the problem occurred during the training of SegResNet. Would you mind providing some info of your system by running this command in your console/terminal?

nvidia-smi

Thanks

[y070504]

(base) root@LAPTOP-C900652C:/deeplearning/auto3dseg# nvidia-smi
Mon Oct 17 14:31:07 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.63 Driver Version: 512.36 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| N/A 40C P8 10W / N/A | 0MiB / 16384MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 25719 G /Xwayland N/A |
+-----------------------------------------------------------------------------+
(base) root@LAPTOP-C900652C:/deeplearning/auto3dseg#

[mingxin-zheng]

As I remember, the trainings in Auto3DSeg in 1.0 release can eat up 20GB of GPU memory in our 1.0 release. This may be the same cause of the issue, in which the subprocess call ran into out-of-memory (OOM) state and terminates.
It is just an educated guess as I don't see any stderr from the subprocess in the post.

Our latest work mitigated the risk of OOM in some datasets, which you may try if that resolves your issue here (as well as the other issue in #993). If you install monai via PIP, you can

pip install monai-weekly --upgrade

I will also update the tutorial in the next couple of days to use a smaller datasets (Task04) to demo. Once I am done, we can move on to confirm if this is either a OOM or something else.

[mingxin-zheng]

Hi @y070504 , with the recent update of notebook, we're closing this issue. Feel free to reopen it if the issue persists. Thanks!