RuntimeError: Not compiled with GPU support

[FrancescoSaverioZuppichini]

Hi guys,

Thanks for the amazing work! I am trying to run the model but I got the following error:

Traceback (most recent call last):
  File "tools/test_grounding_net.py", line 222, in <module>
    main()
  File "tools/test_grounding_net.py", line 205, in main
    inference(
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/engine/inference.py", line 495, in inference
    output = model(images, captions=captions, positive_map=positive_map_label_to_token)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/modeling/detector/generalized_vl_rcnn.py", line 284, in forward
    proposals, proposal_losses, fused_visual_features = self.rpn(images, visual_features, targets, language_dict_features, positive_map,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/modeling/rpn/vldyhead.py", line 920, in forward
    proj_tokens, contrastive_logits, dot_product_logits, mlm_logits, shallow_img_emb_feats, fused_visual_features = self.head(features,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/modeling/rpn/vldyhead.py", line 739, in forward
    dyhead_tower = self.dyhead_tower(feat_inputs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/modeling/rpn/vldyhead.py", line 205, in forward
    temp_fea = [self.DyConv[1](feature, **conv_args)]
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/modeling/rpn/vldyhead.py", line 135, in forward
    x = self.conv(input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 235, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/layers/deform_conv.py", line 380, in forward
    return modulated_deform_conv(
  File "/workspace/GLIP-benchmark/GLIP/maskrcnn_benchmark/layers/deform_conv.py", line 184, in forward
    _C.modulated_deform_conv_forward(
RuntimeError: Not compiled with GPU support

My nvidia-smi output, I've run python setup.py build develop --user beforehand.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.57       Driver Version: 515.57       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:1D:00.0 Off |                  N/A |
| 30%   49C    P8    26W / 350W |    129MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Thanks :)

Cheers,

Francesco

[Haotian-Zhang]

Hi @FrancescoSaverioZuppichini , which PyTorch version did you install? This repo requires Pytorch>=1.9 and torchvision. After installing the Pytorch, please try to run the setup.py again to see if this error still exists.

[FrancescoSaverioZuppichini]

Hi @Haotian-Zhang , I am running that bad boy inside a nvcr.io/nvidia/pytorch:21.08-py3 docker container so pytorch 1.10

[Haotian-Zhang]

@FrancescoSaverioZuppichini It seems like the environment is not yet compatible with the deformable convolution? Could you please try using a different container that has Pytorch 1.9 and give it a try again? Thanks!

[Con6924]

Hi @Haotian-Zhang , I met the same error and tried with Pytorch 1.9 and CUDA 10.2 , but the error remains.

[Haotian-Zhang]

The problem remains for DCNv2 installation with your local env. In order for code to be compiled with DCN modules, please make sure torch.cuda.is_available() is True, and CUDA_HOME is not None (L31-40 in setup.py). If the problem still remains, please check CharlesShang/DCNv2#82.

[FrancescoSaverioZuppichini]

@Haotian-Zhang CUDA_HOME is set, and cuda is obviously available. I'll try with a more recent container, thanks a lot!

[FrancescoSaverioZuppichini]

@Haotian-Zhang can you share your installation process/ a docker file that works? Thanks a lot

[bofei5675]

For those who are still suffering from this problem...

I was using a slurm cluster and met the same issue RuntimeError: Not compiled with GPU support after running python setup.py build develop --user from README. I have torch==1.9.0 and cuda 11.1.

First, I noticed that I was on head node so no GPU with my machine. Then I entered a computing node with CUDA and GPU. Re-install it did not help.

Then, I noticed that ./build folder is in the directory, and I delete all compiled material I can find in my repo and then reinstall the repo by python setup.py build develop --user. Then, the issue has been resolved.

[Infinite-set]

I have the same problem, and my solution is to type export CUDA_HOME=/path/to/your/cuda/ and then delete the GLIP/build folder, then rerun the setup.py. This solution could also be found in DCNv2's issues.

[Mukil07]

Im still facing the same problem, even after deleting GLIP/build and running python setup.py build develop --user again. My virtual env has Pytorch 1.9 and CUDA 10.2

Could you please mention the correct list of installations you have performed which works. Thanks !

[Mukil07]

I could solve it finally using this virtual env

Pytorch 1.10.1 Torchvision 0.11.2 Torchaudio 0.10.1 CUDA 11.3

Also set path to CUDA_HOME and rerun setup.py