CUDA error in ntu_pose_extraction

[TalBarami]

Thanks for your error report and we appreciate it a lot.
If you feel we have help you, give us a STAR! 😆

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug

Hey, I have encountered a CUDA error upon attempting to execute the ntu_pose_extraction script.

Reproduction

1. What command or script did you run?

python ntu_pose_extraction.py S001C001P001R001A001_rgb.avi S001C001P001R001A001.pkl

2. Did you make any modifications on the code or config? Did you understand what you have modified? No
3. What dataset did you use? Provided example video

Environment

1. Please run PYTHONPATH=${PWD}:$PYTHONPATH python mmaction/utils/collect_env.py to collect necessary environment information and paste it here.

Environment:
'tail' is not recognized as an internal or external command,
operable program or batch file.
'gcc' is not recognized as an internal or external command,
operable program or batch file.
sys.platform: win32
Python: 3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3080
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3
NVCC: Not Available
GCC: n/a
PyTorch: 1.10.0
PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192829337
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX512
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.2
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,

TorchVision: 0.11.1
OpenCV: 4.5.4
MMCV: 1.4.0
MMCV Compiler: MSVC 192930137
MMCV CUDA Compiler: 11.3
MMAction2: 0.20.0+61d7eb8

I have followed the installation guidelines and managed to execute the recognition demo with current environment. I have both mmpose and mmdetection installed.

Error traceback

Traceback (most recent call last):
  File "ntu_pose_extraction.py", line 340, in <module>
    anno = ntu_pose_extraction(args.video)
  File "ntu_pose_extraction.py", line 307, in ntu_pose_extraction
    det_results = detection_inference(args, frame_paths)
  File "ntu_pose_extraction.py", line 67, in detection_inference
    result = inference_detector(model, frame_path)
  File "e:\mmdetection\mmdet\apis\inference.py", line 147, in inference_detector
    results = model(return_loss=False, rescale=True, **data)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\runner\fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "e:\mmdetection\mmdet\models\detectors\base.py", line 174, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "e:\mmdetection\mmdet\models\detectors\base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "e:\mmdetection\mmdet\models\detectors\two_stage.py", line 179, in simple_test
    proposal_list = self.rpn_head.simple_test_rpn(x, img_metas)
  File "e:\mmdetection\mmdet\models\dense_heads\dense_test_mixins.py", line 130, in simple_test_rpn
    proposal_list = self.get_bboxes(*rpn_outs, img_metas=img_metas)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\runner\fp16_utils.py", line 186, in new_func
    return old_func(*args, **kwargs)
  File "e:\mmdetection\mmdet\models\dense_heads\base_dense_head.py", line 96, in get_bboxes
    **kwargs)
  File "e:\mmdetection\mmdet\models\dense_heads\rpn_head.py", line 187, in _get_bboxes_single
    img_shape)
  File "e:\mmdetection\mmdet\models\dense_heads\rpn_head.py", line 232, in _bbox_post_process
    dets, _ = batched_nms(proposals, scores, ids, cfg.nms)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\ops\nms.py", line 307, in batched_nms
    dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\utils\misc.py", line 340, in new_func
    output = old_func(*args, **kwargs)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\ops\nms.py", line 172, in nms
    score_threshold, max_num)
  File "C:\Users\owner\anaconda3\envs\mmlab\lib\site-packages\mmcv\ops\nms.py", line 27, in forward
    bboxes, scores, iou_threshold=float(iou_threshold), offset=offset)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Bug fix

If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

Thanks,
Tal

[Powercoder64]

Experts on this website would provide better answers. For me. I commonly encounter this issue when there is a mismatch between the hyperparameter and actual data. For example, it can be the wrong image size, wrong image format (sometimes images are 4D and need to be converted to RGB only). or it can be the wrong config file with wrong values.

[zhouzaida]

Hi @TalBarami , thank you very much for your feedback! We checked our logs and found we missed the compute capability 8_6. We will re-compile and upload the new packages in the next few days.

[TalBarami]

Thanks a lot!

[zhouzaida]

Hi, we have updated the pre-built packages for mmcv-full 1.4.1 and 1.4.2. Please have a try.