windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) #1623

fcakyon · 2021-12-27T11:13:31Z

steps to reproduce:

setup env:

conda create -n mmlab python=3.9 -y 
conda activate mmlab 

conda install pytorch=1.10.0 torchvision=0.11.1 cudatoolkit=11.3 -c pytorch -y
pip install mmcv-full==1.4.2 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html

pip install git+https://github.com/open-mmlab/mmdetection.git@master

confirm cuda and pytorch installation:

# Check Pytorch installation
import torch, torchvision
print(torch.__version__, torch.cuda.is_available())

# Check MMDetection installation
import mmdet
print(mmdet.__version__)

# Check mmcv installation
from mmcv.ops import get_compiling_cuda_version, get_compiler_version
print(get_compiling_cuda_version())
print(get_compiler_version())

1.10.0 True
2.19.1
11.3
MSVC 192930137

confirm mmcv installation:

python -c "from mmcv.ops import RoIPool"

no error

clone mmdet repo for config files:

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection

download mmdet weight:

import urllib.request
from pathlib import Path

def download_from_url(from_url: str, to_path: str):
    Path(to_path).parent.mkdir(parents=True, exist_ok=True)
    urllib.request.urlretrieve(
        from_url,
        to_path,
    )

from_url='https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_tiny_8x8_300e_coco/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth'
to_path='weight.pth'
download_from_url(from_url, to_path)

attempt to perform inference:

from mmdet.apis import inference_detector, init_detector

# Choose to use a config and initialize the detector
config = 'configs/yolox/yolox_tiny_8x8_300e_coco.py'
# Setup a checkpoint file to load
checkpoint = 'weight.pth'
# initialize the detector
model = init_detector(config, checkpoint, device='cuda:0')
# Use the detector to do inference
img = 'demo/demo.jpg'
result = inference_detector(model, img)

error:

load checkpoint from local path: weight.pth
C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\torch\functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ..\aten\src\ATen\native\TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py:284: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ..\torch\csrc\utils\tensor_new.cpp:201.)
  flatten_bboxes[..., :4] /= flatten_bboxes.new_tensor(
Traceback (most recent call last):
  File "C:\Users\xxx\dev\mmdetection\tools\perform_inference.py", line 11, in <module>
    result = inference_detector(model, img)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\apis\inference.py", line 147, in inference_detector
    results = model(return_loss=False, rescale=True, **data)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\runner\fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "C:\Users\FxxxA\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\single_stage.py", line 102, in simple_test
    results_list = self.bbox_head.simple_test(
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\base_dense_head.py", line 360, in simple_test
    return self.simple_test_bboxes(feats, img_metas, rescale=rescale)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\dense_test_mixins.py", line 37, in simple_test_bboxes
    results_list = self.get_bboxes(
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py", line 294, in get_bboxes
    self._bboxes_nms(cls_scores, bboxes, score_factor, cfg))
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py", line 321, in _bboxes_nms
    dets, keep = batched_nms(bboxes, scores, labels, cfg.nms)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 307, in batched_nms
    dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\utils\misc.py", line 340, in new_func
    output = old_func(*args, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 171, in nms
    inds = NMSop.apply(boxes, scores, iou_threshold, offset,
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 26, in forward
    inds = ext_module.nms(
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

zhouzaida · 2021-12-27T11:28:06Z

Hi, what is your graphics card?

fcakyon · 2021-12-27T12:03:38Z

@zhouzaida its rtx3060 (compute capability 8.6)

zhouzaida · 2021-12-28T15:11:06Z

Could you try to install mmcv-full==1.4.1?

fcakyon · 2021-12-28T16:25:29Z

Could you try to install mmcv-full==1.4.1?

@zhouzaida

setup env with mmcv-full==1.4.1:

conda create -n mmlab python=3.9 -y 
conda activate mmlab 

conda install pytorch=1.10.0 torchvision=0.11.1 cudatoolkit=11.3 -c pytorch -y
pip install mmcv-full==1.4.1 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html

pip install git+https://github.com/open-mmlab/mmdetection.git@master

confirm cuda and pytorch installation:

# Check Pytorch installation
import torch, torchvision
print(torch.__version__, torch.cuda.is_available())

# Check MMDetection installation
import mmdet
print(mmdet.__version__)

# Check mmcv installation
from mmcv.ops import get_compiling_cuda_version, get_compiler_version
print(get_compiling_cuda_version())
print(get_compiler_version())

1.10.0 True
2.19.1
11.3
MSVC 192930137

confirm mmcv installation:

python -c "from mmcv.ops import RoIPool"

no error

clone mmdet repo for config files:

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection

download mmdet weight:

import urllib.request
from pathlib import Path

def download_from_url(from_url: str, to_path: str):
    Path(to_path).parent.mkdir(parents=True, exist_ok=True)
    urllib.request.urlretrieve(
        from_url,
        to_path,
    )

from_url='https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_tiny_8x8_300e_coco/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth'
to_path='weight.pth'
download_from_url(from_url, to_path)

attempt to perform inference:

from mmdet.apis import inference_detector, init_detector

# Choose to use a config and initialize the detector
config = 'configs/yolox/yolox_tiny_8x8_300e_coco.py'
# Setup a checkpoint file to load
checkpoint = 'weight.pth'
# initialize the detector
model = init_detector(config, checkpoint, device='cuda:0')
# Use the detector to do inference
img = 'demo/demo.jpg'
result = inference_detector(model, img)

same error:

load checkpoint from local path: weight.pth
C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\torch\functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ..\aten\src\ATen\native\TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py:284: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ..\torch\csrc\utils\tensor_new.cpp:201.)
  flatten_bboxes[..., :4] /= flatten_bboxes.new_tensor(
Traceback (most recent call last):
  File "C:\Users\xxx\dev\mmdetection\tools\perform_inference.py", line 11, in <module>
    result = inference_detector(model, img)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\apis\inference.py", line 147, in inference_detector
    results = model(return_loss=False, rescale=True, **data)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\runner\fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "C:\Users\FxxxA\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\detectors\single_stage.py", line 102, in simple_test
    results_list = self.bbox_head.simple_test(
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\base_dense_head.py", line 360, in simple_test
    return self.simple_test_bboxes(feats, img_metas, rescale=rescale)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\dense_test_mixins.py", line 37, in simple_test_bboxes
    results_list = self.get_bboxes(
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py", line 294, in get_bboxes
    self._bboxes_nms(cls_scores, bboxes, score_factor, cfg))
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmdet\models\dense_heads\yolox_head.py", line 321, in _bboxes_nms
    dets, keep = batched_nms(bboxes, scores, labels, cfg.nms)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 307, in batched_nms
    dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\utils\misc.py", line 340, in new_func
    output = old_func(*args, **kwargs)
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 171, in nms
    inds = NMSop.apply(boxes, scores, iou_threshold, offset,
  File "C:\Users\xxx\Miniconda3\envs\xxx\lib\site-packages\mmcv\ops\nms.py", line 26, in forward
    inds = ext_module.nms(
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

fcakyon · 2021-12-28T22:59:36Z

@zhouzaida maybe your precompiled mmcv windows binaries are not compiled for compute capability 8_6, might it be the reason? Have you tried running the pre-compiled mmcv binaries on RTX3XXX card on windows?

Refer to for the explanation: limbo018/DREAMPlace#56 (comment)

fcakyon · 2021-12-28T23:12:36Z

Based on this comment I am now updating my Nvidia Driver from 461.72 to 472.84: google/jax#5723 (comment) but still having the same error.

At this moment the most probable reason is that you are not compiling the mmcv kernels for compute capability 8_6 for windows wheel. As a result, mmcv does not work on RTX3XXX devices on Windows.

fcakyon · 2021-12-28T23:22:41Z

I found similar errors on mmcv+windows+compute capability 8_6 gpus:
MMCV(1.4.1) + Windows + RTX3070: open-mmlab/mmtracking#378
MMCV(1.4.0) + Windows + RTX3080: #1559
MMCV(1.4.0) + Windows + RTX3080: open-mmlab/mmaction2#1308

fcakyon · 2021-12-29T00:04:02Z

Btw this error is not present with mmcv 1.4.2, torch 1.9.0 and cuda 11.1:

conda create -n mmlab python=3.9 -y 
conda activate mmlab 

conda install pytorch=1.9.0 torchvision=0.10.0 cudatoolkit=11.1 -c pytorch -c conda-forge -y
pip install mmcv-full==1.4.2 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html

pip install git+https://github.com/open-mmlab/mmdetection.git@master

zhouzaida · 2021-12-29T06:51:54Z

Yet, thank you very much for your feedback! We checked our logs and found we missed the compute capability 8_6. We will re-compile and upload the new packages in the next few days.

zhouzaida · 2021-12-30T15:51:16Z

Btw this error is not present with mmcv 1.4.2, torch 1.9.0 and cuda 11.1:

conda create -n mmlab python=3.9 -y 
conda activate mmlab 

conda install pytorch=1.9.0 torchvision=0.10.0 cudatoolkit=11.1 -c pytorch -c conda-forge -y
pip install mmcv-full==1.4.2 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html

pip install git+https://github.com/open-mmlab/mmdetection.git@master

Hi, we have updated the pre-built packages for mmcv-full 1.4.1 and 1.4.2. Please have a try.

fcakyon · 2021-12-30T17:33:30Z

@zhouzaida thanks a lot for your fast responses, i confirmed this working:

setup env:

conda create -n mmlab python=3.9 -y 
conda activate mmlab 

conda install pytorch=1.10.0 torchvision=0.11.1 cudatoolkit=11.3 -c pytorch -y
pip install mmcv-full==1.4.2 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html

pip install mmdet==2.20.0

confirm cuda and pytorch installation:

# Check Pytorch installation
import torch, torchvision
print(torch.__version__, torch.cuda.is_available())

# Check MMDetection installation
import mmdet
print(mmdet.__version__)

# Check mmcv installation
from mmcv.ops import get_compiling_cuda_version, get_compiler_version
print(get_compiling_cuda_version())
print(get_compiler_version())

1.10.0 True
2.20.0
11.3
MSVC 192930137

confirm mmcv installation:

python -c "from mmcv.ops import RoIPool"

no error

clone mmdet repo for config files:

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection

download mmdet weight:

import urllib.request
from pathlib import Path

def download_from_url(from_url: str, to_path: str):
    Path(to_path).parent.mkdir(parents=True, exist_ok=True)
    urllib.request.urlretrieve(
        from_url,
        to_path,
    )

from_url='https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_tiny_8x8_300e_coco/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth'
to_path='weight.pth'
download_from_url(from_url, to_path)

attempt to perform inference:

from mmdet.apis import inference_detector, init_detector

# Choose to use a config and initialize the detector
config = 'configs/yolox/yolox_tiny_8x8_300e_coco.py'
# Setup a checkpoint file to load
checkpoint = 'weight.pth'
# initialize the detector
model = init_detector(config, checkpoint, device='cuda:0')
# Use the detector to do inference
img = 'demo/demo.jpg'
result = inference_detector(model, img)

no error 🚀

zhouzaida added the Installation label Dec 27, 2021

fcakyon changed the title ~~windows mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master)~~ windows 10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) Dec 27, 2021

fcakyon changed the title ~~windows 10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master)~~ windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) Dec 27, 2021

hhaAndroid mentioned this issue Dec 28, 2021

windows mmdet@master inference gives error on mmcv-full v1.4.2 (py3.9, cu113) open-mmlab/mmdetection#6903

Closed

fcakyon mentioned this issue Dec 28, 2021

RuntimeError: CUDA error: no kernel image is available for execution on the device open-mmlab/mmdetection#4335

Closed

fcakyon closed this as completed Dec 30, 2021

This was referenced Feb 27, 2022

ImportError: cannot import name 'setup_multi_processes' from 'mmdet.utils' open-mmlab/mmdetection#7272

Closed

Training with customized dataset - Balloon example open-mmlab/mmdetection#7275

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) #1623

windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) #1623

fcakyon commented Dec 27, 2021 •

edited

Loading

zhouzaida commented Dec 27, 2021

fcakyon commented Dec 27, 2021

zhouzaida commented Dec 28, 2021

fcakyon commented Dec 28, 2021

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 29, 2021 •

edited

Loading

zhouzaida commented Dec 29, 2021 •

edited

Loading

zhouzaida commented Dec 30, 2021

fcakyon commented Dec 30, 2021

windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) #1623

windows10 mmdet inference gives error on mmcv-full v1.4.2 (py3.9, cu113, mmdet@master) #1623

Comments

fcakyon commented Dec 27, 2021 • edited Loading

steps to reproduce:

error:

zhouzaida commented Dec 27, 2021

fcakyon commented Dec 27, 2021

zhouzaida commented Dec 28, 2021

fcakyon commented Dec 28, 2021

same error:

fcakyon commented Dec 28, 2021 • edited Loading

fcakyon commented Dec 28, 2021 • edited Loading

fcakyon commented Dec 28, 2021 • edited Loading

fcakyon commented Dec 29, 2021 • edited Loading

zhouzaida commented Dec 29, 2021 • edited Loading

zhouzaida commented Dec 30, 2021

fcakyon commented Dec 30, 2021

fcakyon commented Dec 27, 2021 •

edited

Loading

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 28, 2021 •

edited

Loading

fcakyon commented Dec 29, 2021 •

edited

Loading

zhouzaida commented Dec 29, 2021 •

edited

Loading