/opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/cuda/Loss.cu:247: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed.

[fancy-chenyao]

您好，在调试您的代码的过程中遇到了这个错误，我用的自己的数据集，请问问题是出在哪里呢？具体报错如下所示，非常期待您的回复！
/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/cuda/Loss.cu:247: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed.
/opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/cuda/Loss.cu:247: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [3,0,0] Assertion t >= 0 && t < n_classes failed.
/opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/cuda/Loss.cu:247: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [5,0,0] Assertion t >= 0 && t < n_classes failed.
/opt/conda/conda-bld/pytorch_1634272126608/work/aten/src/ATen/native/cuda/Loss.cu:247: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [6,0,0] Assertion t >= 0 && t < n_classes failed.
0%| | 0/1125 [00:02<?, ?it/s]
Traceback (most recent call last):
File "main.py", line 7, in
processor.run()
File "/root/cascade-rcnn/solver/ddp_mix_solver.py", line 213, in run
self.train(epoch)
File "/root/cascade-rcnn/solver/ddp_mix_solver.py", line 113, in train
targets={"target": targets_tensor, "batch_len": batch_len})
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/cascade-rcnn/nets/cascade_rcnn.py", line 702, in forward
box_predicts, cls_predicts, roi_losses = self.cascade_head(feature_dict, boxes, valid_size, targets)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/cascade-rcnn/nets/cascade_rcnn.py", line 621, in forward
boxes, cls, loss = self.roi_heads[i](feature_dict, boxes, valid_size, targets)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/cascade-rcnn/nets/cascade_rcnn.py", line 590, in forward
cls_loss, box_loss = self.compute_loss(proposals, cls_predicts, box_predicts, targets)
File "/root/cascade-rcnn/nets/cascade_rcnn.py", line 564, in compute_loss
cls_loss = self.ce(loss_cls_predicts, loss_cls_targets)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 1152, in forward
label_smoothing=self.label_smoothing)
File "/root/miniconda3/envs/my-env/lib/python3.7/site-packages/torch/nn/functional.py", line 2846, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 2085) of binary: /root/miniconda3/envs/my-env/bin/python

[yuanfangshang888]

我感觉是博主是使用的分布式训练，但是你使用的是单GPU训练，使用才出现了这个问题，因为我看到了你bug中的RROR:torch.distributed.elastic.multiprocessing.api:failed，我的猜测是这样的，你有时间了可以尝试一下这个思路