RuntimeError: Expected cudaMemcpy(&mask_host[0], mask_dev, sizeof(unsigned long long) * boxes_num * col_blocks, cudaMemcpyDeviceToHost) == cudaSuccess to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

[fuma8]

Describe the bug
I run the code below, but get the error. Can anyone help me?

Reproduction

1. What command or script did you run?

python tools/demo.py configs/elephant/cityperson/cascade_hrnet.py ./models_pretrained/epoch_5.pth.stu demo/ result_demo/ 

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

Environment

• OS: Ubuntu20.04
• GCC 11.1.0
• PyTorch version 1.13.1+cu117
• How you installed PyTorch pip
• GPU model 1080Ti
• CUDA and CUDNN version 11.7

Error traceback
If applicable, paste the error trackback here.

['demo/2.png', 'demo/3.png', 'demo/1.png']
unexpected key in source state_dict: mask_head.0.conv_res.conv.weight, mask_head.0.conv_res.conv.bias, mask_head.1.conv_res.conv.weight, mask_head.1.conv_res.conv.bias, mask_head.2.conv_res.conv.weight, mask_head.2.conv_res.conv.bias

/home/kimishima/Downloads/Pedestron/mmdet/apis/inference.py:39: UserWarning: Class names are not saved in the checkpoint's meta data, use COCO classes by default.
  warnings.warn('Class names are not saved in the checkpoint\'s '
[                      ] 0/3, elapsed: 0s, ETA:/home/kimishima/Downloads/Pedestron/mmdet/core/bbox/transforms.py:56: UserWarning: This overload of addcmul is deprecated:
        addcmul(Tensor input, Number value, Tensor tensor1, Tensor tensor2, *, Tensor out)
Consider using one of the following signatures instead:
        addcmul(Tensor input, Tensor tensor1, Tensor tensor2, *, Number value, Tensor out) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:1420.)
  gx = torch.addcmul(px, 1, pw, dx)  # gx = px + pw * dx
Traceback (most recent call last):
  File "tools/demo.py", line 67, in <module>
    run_detector_on_dataset()
  File "tools/demo.py", line 63, in run_detector_on_dataset
    detections = mock_detector(model, im, output_dir)
  File "tools/demo.py", line 37, in mock_detector
    results = inference_detector(model, image)
  File "/home/kimishima/Downloads/Pedestron/mmdet/apis/inference.py", line 66, in inference_detector
    return _inference_single(model, imgs, img_transform, device)
  File "/home/kimishima/Downloads/Pedestron/mmdet/apis/inference.py", line 93, in _inference_single
    result = model(return_loss=False, rescale=True, **data)
  File "/home/kimishima/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/detectors/base.py", line 88, in forward
    return self.forward_test(img, img_meta, **kwargs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/detectors/base.py", line 79, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/detectors/cascade_rcnn.py", line 242, in simple_test
    proposal_list = self.simple_test_rpn(
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/detectors/test_mixins.py", line 10, in simple_test_rpn
    proposal_list = self.rpn_head.get_bboxes(*proposal_inputs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/core/fp16/decorators.py", line 127, in new_func
    return old_func(*args, **kwargs)
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/anchor_heads/anchor_head.py", line 219, in get_bboxes
    proposals = self.get_bboxes_single(cls_score_list, bbox_pred_list,
  File "/home/kimishima/Downloads/Pedestron/mmdet/models/anchor_heads/rpn_head.py", line 92, in get_bboxes_single
    proposals, _ = nms(proposals, cfg.nms_thr)
  File "/home/kimishima/Downloads/Pedestron/mmdet/ops/nms/nms_wrapper.py", line 43, in nms
    inds = nms_cuda.nms(dets_th, iou_thr)# inds = nms_cuda.nms(dets_th, iou_thr)
RuntimeError: Expected cudaMemcpy(&mask_host[0], mask_dev, sizeof(unsigned long long) * boxes_num * col_blocks, cudaMemcpyDeviceToHost) == cudaSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

[ashuta03]

I am getting the same error. Did you find a solution?