CUDA error: device-side assert triggered

[Devicharith]

create web directory ./checkpoints/city/web...
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)
Traceback (most recent call last):
File "train.py", line 40, in
trainer.run_generator_one_step(data_i)
File "/content/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step
g_losses, generated = self.pix2pix_model(data, mode='generator')
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
return self.module(*inputs[0], **kwargs[0])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/content/SPADE/models/pix2pix_model.py", line 46, in forward
input_semantics, real_image)
File "/content/SPADE/models/pix2pix_model.py", line 136, in compute_generator_loss
input_semantics, real_image, compute_kld_loss=self.opt.use_vae)
File "/content/SPADE/models/pix2pix_model.py", line 191, in generate_fake
z, mu, logvar = self.encode_z(real_image)
File "/content/SPADE/models/pix2pix_model.py", line 183, in encode_z
mu, logvar = self.netE(real_image)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, *kwargs)
File "/content/SPADE/models/networks/encoder.py", line 40, in forward
x = F.interpolate(x, size=(256, 256), mode='bilinear')
File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 3709, in interpolate
return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:1055 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f9e94813a22 in /usr/local/lib/python3.7/dist-packages/torch/lib/libc10.so)
frame #1: + 0x10983 (0x7f9e94a74983 in /usr/local/lib/python3.7/dist-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0x1a7 (0x7f9e94a76027 in /usr/local/lib/python3.7/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f9e947fd5a4 in /usr/local/lib/python3.7/dist-packages/torch/lib/libc10.so)
frame #4: + 0xa2ef72 (0x7f9eeb372f72 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_python.so)
frame #5: + 0xa2f011 (0x7f9eeb373011 in /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_python.so)

frame #21: __libc_start_main + 0xe7 (0x7f9eeee3fbf7 in /lib/x86_64-linux-gnu/libc.so.6)

[mingyuliutw]

New implementation available at imaginaire repository

We have a reimplementation of the SPADE method that is more performant. It is avaiable at Imaginaire