run preprocess.py script encountering a problem 'CUDA error: an illegal memory access was encountered'

[shawnyuen]

the following is output of terminal:
2021-06-05 18:06:10.058 | INFO | nndet.planning.estimator:_estimate_mem_available:153 - Estimating in memory.
2021-06-05 18:06:10.059 | INFO | nndet.planning.estimator:measure:192 - Estimating on cuda:0 with shape [1, 192, 192, 192] and batch size 4 and num_instances 1
2021-06-05 18:06:13.341 | INFO | nndet.planning.estimator:measure:242 - Caught error (If out of memory error do not worry): CUDA error: an illegal memory access was encountered
Traceback (most recent call last):
File "preprocess.py", line 484, in
main()
File "/home/shawnyuen/projects/nnDetection/nndet/utils/check.py", line 58, in wrapper
return func(*args, **kwargs)
File "preprocess.py", line 477, in main
run(OmegaConf.to_container(cfg, resolve=True),
File "preprocess.py", line 404, in run
run_planning_and_process(
File "preprocess.py", line 231, in run_planning_and_process
plan_identifiers = planner.plan_experiment(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/experiment/v001.py", line 43, in plan_experiment
plan_3d = self.plan_base_stage(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/experiment/base.py", line 234, in plan_base_stage
architecture_plan = architecture_planner.plan(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/architecture/boxes/c002.py", line 127, in plan
res = super().plan(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/architecture/boxes/base.py", line 343, in plan
patch_size = self._plan_architecture(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/architecture/boxes/c002.py", line 205, in _plan_architecture
_, fits_in_mem = self.estimator.estimate(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/estimator.py", line 127, in estimate
res = self._estimate_mem_available(
File "/home/shawnyuen/projects/nnDetection/nndet/planning/estimator.py", line 154, in _estimate_mem_available
fixed, dynamic = self.measure(shape=target_shape,
File "/home/shawnyuen/projects/nnDetection/nndet/planning/estimator.py", line 253, in measure
network.cpu()
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 471, in cpu
return self._apply(lambda t: t.cpu())
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
module._apply(fn)
[Previous line repeated 4 more times]
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 381, in _apply
param_applied = fn(param)
File "/home/shawnyuen/anaconda3/envs/nndet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 471, in
return self._apply(lambda t: t.cpu())
RuntimeError: CUDA error: an illegal memory access was encountered

the following is my commands:
`
python generate_example.py --full --num_processes 12

python preprocess.py 000 -np 2 -npp 2
`

when processing this size [1, 192, 192, 192], error is raised.

[mibaumgartner]

Hi @shawnyuen ,

I encountered that error myself once but it did not occur after restarting the script (same preprocessing command you posted). Could you run nndet_env to check if cuda is available on your machine and eventually try to run the code with CUDA_VISIBLE_DEVICES set to the GPU you want to use for planning?

Best,
Michael

[shawnyuen]

Hi, @mibaumgartner ,
nndet_env and CUDA_VISIBLE_DEVICES are tried, but they don't work. Thanks, I will find other methods.

[mibaumgartner]

I'll add a different VRAM estimator over the next days (similar to nnUNet-Net's estimation which does not require a GPU). That will definitely fix it.

Unfortunately, it is quite hard for me to fix the error in the current estimator, since it does not occur on my machine.

[shawnyuen]

[1, 192, 192, 192] is a magic size. I generate toy data with size [128, 128, 128], it works.