Cudnn error when using SPADE

[emanueledelsozzo]

Hello,
I am trying to use SPADE on a custom dataset. Here the command I run:
python train.py --name myTest --dataset_mode custom --label_dir datasets/label_dir --image_dir datasets/image_dir --no_instance --gpu_ids 0,1 --batchSize 2

However, after the creation of the web directory, I get this error:

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [21,0,0] Assertion indexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [23,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [24,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [25,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [26,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [27,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [28,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [29,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [30,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [31,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]` failed.
Traceback (most recent call last):
File "train.py", line 40, in
trainer.run_generator_one_step(data_i)
File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step
g_losses, generated = self.pix2pix_model(data, mode='generator')
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(*input, **kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 46, in forward
input_semantics, real_image)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 137, in compute_generator_loss
input_semantics, real_image, compute_kld_loss=self.opt.use_vae)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 196, in generate_fake
fake_image = self.netG(input_semantics, z=z)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/repos/testSPADE/SPADE/models/networks/generator.py", line 89, in forward
x = self.fc(x)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:564)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f25a3999441 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f25a3998d7a in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: + 0x13652 (0x7f25a2f45652 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x50 (0x7f25a3989ce0 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: + 0x30facb (0x7f2573acdacb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #5: + 0x1420bb (0x7f25a3f270bb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3c30f4 (0x7f25a41a80f4 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x3c3141 (0x7f25a41a8141 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #24: __libc_start_main + 0xe7 (0x7f25a8297b97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
`
I am using python 3.6, CUDA 10.0, Torch 1.1.0, CuDNN 7.5.0
I have also tried on another machine with CUDA 9.0, Torch 1.0.0, CuDNN 7.4.1, but I got the same error.

Could you help me?

[alexsannikoff]

@emanueledelsozzo Check that in your custom dataset doesn't exist class with label > num_labels in your config. Add CUDA_LAUNCH_BLOCKING=1 before python ... and see normal traceback.

[emanueledelsozzo]

Hello @alexsannikoff and thank you for your reply!
Here the log I get adding CUDA_LAUNCH_BLOCKING=1

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [450,0,0] Assertion indexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [454,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [457,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [463,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [464,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [465,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [466,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [467,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [468,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [469,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [470,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [471,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [472,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [473,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [474,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [475,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [476,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [477,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [478,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [479,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [416,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [417,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [418,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [419,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [420,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [421,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [422,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [423,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [424,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [425,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [426,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [427,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [428,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [429,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [430,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [431,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [432,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [434,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [436,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [447,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [416,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [417,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [418,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [419,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [420,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [421,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [422,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [423,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [424,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [425,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [426,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [427,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [428,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [429,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [430,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [431,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [432,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [433,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [434,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [435,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [436,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [437,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [438,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [441,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [442,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [443,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [444,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [445,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [446,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [447,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [256,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [257,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [258,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [259,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [260,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [261,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [262,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [263,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [264,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [265,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [266,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [267,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [268,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [269,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [270,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [271,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [273,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [274,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [275,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [276,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [277,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [263,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [265,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [266,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [267,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [268,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [269,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [270,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [271,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [272,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [273,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [274,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [275,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [276,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [277,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [387,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [388,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [389,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [399,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [400,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [401,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [402,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [403,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [404,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [405,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [406,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [407,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [408,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [409,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [410,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [411,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [412,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [413,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [414,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [415,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [450,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [451,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [452,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [453,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [454,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [455,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [456,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [457,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [458,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [459,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [460,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [461,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [462,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [463,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [464,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [465,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [466,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [467,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [468,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [469,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [470,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [471,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [472,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [473,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [474,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [475,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [476,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [477,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [478,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [479,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorScatterGather.cu line=380 error=59 : device-side assert triggered
Traceback (most recent call last):
File "train.py", line 40, in
trainer.run_generator_one_step(data_i)
File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step
g_losses, generated = self.pix2pix_model(data, mode='generator')
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in worker
output = module(*input, **kwargs)
File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 42, in forward
input_semantics, real_image = self.preprocess_input(data)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 123, in preprocess_input
input_semantics = input_label.scatter(1, label_map, 1.0)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:380
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:564)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fc95a205441 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fc95a204d7a in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: + 0x13652 (0x7fc9597b1652 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x50 (0x7fc95a1f5ce0 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: + 0x30facb (0x7fc94fb48acb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #5: + 0x1420bb (0x7fc95a7930bb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3c30f4 (0x7fc95aa140f4 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x3c3141 (0x7fc95aa14141 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #24: __libc_start_main + 0xe7 (0x7fc95eb03b97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)`

Probably the problem is related to your observation about the labels. What I am trying to do is to train a pix2pix able to learn the transformation from A to B. I am not really using any semantic information. I trained a similar network with pix2pixhd and it worked. Now I would like to do the same with SPADE, but I am not sure whether it is possible or not. I also tried to set netG to pix2pixhd, but I got the same error.
By the way, when I trained the pix2pixhd, I used to set label_nc to 0, as suggested in the pix2pixhd README. Here, if I try to do the same, I get this error:

Traceback (most recent call last): File "train.py", line 24, in <module> trainer = Pix2PixTrainer(opt) File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 19, in __init__ self.pix2pix_model = Pix2PixModel(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 25, in __init__ self.netG, self.netD, self.netE = self.initialize_networks(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 92, in initialize_networks netG = networks.define_G(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/__init__.py", line 52, in define_G return create_network(netG_cls, opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/__init__.py", line 41, in create_network net = cls(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/generator.py", line 38, in __init__ self.fc = nn.Conv2d(self.opt.semantic_nc, 16 * nf, 3, padding=1) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 327, in __init__ False, _pair(0), groups, bias, padding_mode) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 45, in __init__ self.reset_parameters() File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 48, in reset_parameters init.kaiming_uniform_(self.weight, a=math.sqrt(5)) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 323, in kaiming_uniform_ fan = _calculate_correct_fan(tensor, mode) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 292, in _calculate_correct_fan fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 222, in _calculate_fan_in_and_fan_out receptive_field_size = tensor[0][0].numel() IndexError: index 0 is out of bounds for dimension 0 with size 0

[aviel08]

Hi, I'm trying to train a custom dataset as well and I'm facing the same error. In my case I'm using labels and instances but that doesn't make any difference.

[alexsannikoff]

@emanueledelsozzo
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 42, in forward
input_semantics, real_image = self.preprocess_input(data)
File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 123, in preprocess_input
input_semantics = input_label.scatter(1, label_map, 1.0)

Here programm try to create one-hot encoding your semantic mask. And if num_classes on your mask > label_nc - this error occured.

What about learning image translation from A to B. In this code I think in pix2pixhd model using SPADE block, which use semantic mask for input. I think, that in this version you can't learn image translation without change code. The main idea this repo in SPADE block.

[alexsannikoff]

@aviel08 Check your semantic mask that max(np.unique(mask)) < label_nc, and don't forget that instance mask has more classes than semantic and using to separate objects with same class each other. label_nc - number of classes in semantic mask.

[emanueledelsozzo]

@alexsannikoff
ok, thank you for your clarification!

[vicmancr]

@emanueledelsozzo Did you get it to work? I am having the same problem. My labels have 3 semantic labels and the background.

I don't get how the labels are treated, since they are loaded in the range [0, 255] and then converted to one-hot encoding. But what happens if some label is missing in a specific image?

[vdhyani96]

Hi guys, just in case it helps anyone. I was also facing the same issues about CUDA error and was able to resolve it by using --no_instance and --contain_dontcare_label arguments. Also, want to mention that I am not using instance maps with my custom dataset.

[zhangdan8962]

@vdhyani96 I am wondering if you are doing Image->Image or Semantic->Image task?

[vdhyani96]

@zhangdan8962 I am doing semantic->image task

[Reheiznekrok]

Hello, I had the same error when attempting to train with a custom data set. I made some changes to my data set and now it works. I don't know if it was just one or all of my changes that made the difference, but for the ones who are as desperate as me, here are all my changes:

• Make sure the data set has the exact same structure as in datasets/coco_stuff/, i.e. save images as .jpg, labels as .png, both saved in separate folders. The images should have 3 channels (last), the labels are only of shape width x height with labels as gray scale values. Names should consist of 12 digits, counting from bottom up, e.g. 000000000184.jpg and must be identical for images and labels
• use the option --contains_dontcare_label and potentially a very high --labels_nc (and --no_instance if you don't have this in your data set)

General approach: first make it work on datasets/coco_stuff/ with --dataset_mode coco, then try coco_stuff as custom data set and then compare to your own.
Hope this helps someone,
cheers

[liping1997]

can be solved by
"python train.py --name work --dataset_mode custom --label_dir ./datasets/label --image_dir ./datasets/image --no_instance --contain_dontcare_label --label_nc 255"
the --contain_dontcare_label and --label_nc 255 must both be added

[Rajeswari-gn]

I have tried all the above solutions still I am getting the same error. Has anyone tried some other solutions which helped to sort out this issue?

I have no issues using the coco_stuff dataset, but I get this error only when using the custom dataset.

[liping1997]

谢谢，已收到。