CUDA error: an illegal memory access was encontered

[kmaeii]

Thanks for your great work!
When i run train.py at my dataset, i got this error. I got my dataset by
python prepare_data.py --out '/data/kmaeii/dataset/stylegan2/bag_texture_mdb' --n_worker 16 --size 128,256 '/data/kmaeii/dataset/stylegan2'

[kmaeii]

By the way, when i run train.py again, this error happened at different places.

[rosinality]

Maybe custom op related error. Could let me know your GPUs, CUDA & pytorch version?

Also, if you set CUDA_LAUNCH_BLOCKING=1 before run the train.py, then it will be easier to spot the location of the error.

[kmaeii]

Thanks for your response.
I am runing this code with 1 TITAN Xp GPU and the CUDA version is 10.0.130, the pytroch version is 1.3.1。

I fixed this error by add the code 'torch.cuda.set_device(1)', because i only want to run this code on the GPU 1.

Before i just change " device = 'cuda' " to " device = 'cuda:1' ", and then found GPU 0 and GPU 1 both are used, so, i got the error above.

Once again, thanks for your great work!!

[rosinality]

Glad to hear that the problem is resolved. By the way, wouldn't it be simpler to use CUDA_VISIBLE_DEVICES?

[kmaeii]

I try "os.environ['CUDA_VISIBLE_DEVICES'] = '1' " first, but the error still appears.