Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can not build setup.py with cuda 11 #4204

Closed
GiangHLe opened this issue Nov 30, 2020 · 9 comments
Closed

Can not build setup.py with cuda 11 #4204

GiangHLe opened this issue Nov 30, 2020 · 9 comments
Assignees
@ZwwWayne

Comments

@GiangHLe
Copy link

GiangHLe commented Nov 30, 2020 •
edited
Loading

Checklist

  1. I have searched related issues RuntimeError: nms is not compiled with GPU support #2882 and some issues in Detectron2 but still can not fix the problem

My problem is: The setup.py file can not recognize my GPU

What did I try?

Follow this issue, after install pytorch, I check with this code:

python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'

and it return True, then I install mmcv by pip install mmcv-full==latest+torch1.7.0+cu110 -f https://download.openmmlab.com/mmcv/dist/index.html and install requirement then run setup.py. It's false.

I delele the repo, remove enviroment and try again with comment of @sanjmohan in this issue, this time I test with

CUDA_VISIBLE_DEVICES=0 python setup.py develop

It still false, after remove all, I do that again, follow #2882 , I put these lines inside setup.py

print('\n\n\n\n\n=======')
print(torch.cuda.is_available())
print('=======\n\n\n\n')

And it return True

In mmdetection, I create my custom dataset and custom configs

Environment
sys.platform: linux
Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 3090
CUDA_HOME: /usr
NVCC: Build cuda_11.0_bu.TC445_37.28540450_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:

  • GCC 7.3
  • C++ Version: 201402
  • Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  • Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • NNPACK is enabled
  • CPU capability usage: AVX2
  • CUDA Runtime 11.0
  • NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_37,code=compute_37
  • CuDNN 8.0.3
  • Magma 2.5.2
  • Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.1
OpenCV: 4.4.0
MMCV: 1.2.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: not available
MMDetection: 2.6.0+8179440

Error traceback

  File "tools/test.py", line 208, in <module>
    main()
  File "tools/test.py", line 178, in main
    args.show_score_thr)
  File "/home/giang/Desktop/test/mmdetection/mmdet/apis/test.py", line 27, in single_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 42, in forward
    return super().forward(*inputs, **kwargs)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/detectors/base.py", line 182, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/detectors/base.py", line 159, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/detectors/two_stage.py", line 191, in simple_test
    x = self.extract_feat(img)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/detectors/two_stage.py", line 82, in extract_feat
    x = self.backbone(img)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/backbones/detectors_resnet.py", line 283, in forward
    outs = list(super(DetectoRS_ResNet, self).forward(x))
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/backbones/resnet.py", line 635, in forward
    x = res_layer(x)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/backbones/resnet.py", line 296, in forward
    out = _inner_forward(x)
  File "/home/giang/Desktop/test/mmdetection/mmdet/models/backbones/resnet.py", line 273, in _inner_forward
    out = self.conv2(out)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/mmcv/ops/saconv.py", line 104, in forward
    self.dilation, self.groups, 1)
  File "/home/giang/anaconda3/envs/four/lib/python3.7/site-packages/mmcv/ops/deform_conv.py", line 98, in forward
    im2col_step=cur_im2col_step)
RuntimeError: DeformConv is not compiled with GPU support
@ZwwWayne
Copy link
Collaborator

ZwwWayne commented Dec 5, 2020

Your MMCV is not successfully compiled in your environment. You need to re-install your MMCV in the environment with a GPU, with the command MMCV_WITH_OPS=1 pip install -e ..

@GiangHLe
Copy link
Author

GiangHLe commented Dec 7, 2020 •
edited
Loading 

/Desktop/mmdetection$ MMCV_WITH_OPS=1 pip install -e .
Obtaining file:///home/giang/Desktop/mmdetection
Requirement already satisfied: matplotlib in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmdet==2.6.0) (3.3.3)
Requirement already satisfied: mmpycocotools in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmdet==2.6.0) (12.0.3)
Requirement already satisfied: numpy in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmdet==2.6.0) (1.19.2)
Requirement already satisfied: six in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmdet==2.6.0) (1.15.0)
Requirement already satisfied: terminaltables in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmdet==2.6.0) (3.1.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from matplotlib->mmdet==2.6.0) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from matplotlib->mmdet==2.6.0) (8.0.1)
Requirement already satisfied: cycler>=0.10 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from matplotlib->mmdet==2.6.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from matplotlib->mmdet==2.6.0) (2.4.7)
Requirement already satisfied: python-dateutil>=2.1 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from matplotlib->mmdet==2.6.0) (2.8.1)
Requirement already satisfied: setuptools>=18.0 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmpycocotools->mmdet==2.6.0) (50.3.1.post20201107)
Requirement already satisfied: cython>=0.27.3 in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmpycocotools->mmdet==2.6.0) (0.29.21)
Installing collected packages: mmdet
  Attempting uninstall: mmdet
    Found existing installation: mmdet 2.6.0
    Uninstalling mmdet-2.6.0:
      Successfully uninstalled mmdet-2.6.0
  Running setup.py develop for mmdet
Successfully installed mmdet

It succeeded but the problem still there: RuntimeError: DeformConv is not compiled with GPU support

@ZwwWayne
Copy link
Collaborator

You should go to the directory of MMCV, and install MMCV by MMCV_WITH_OPS=1 pip install -e ..

@GiangHLe
Copy link
Author

I try and it return new problem, can you take a look

Obtaining file:///home/giang/Desktop/mmcv
Requirement already satisfied: addict in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmcv-full==1.2.2) (2.4.0)
Requirement already satisfied: numpy in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmcv-full==1.2.2) (1.19.2)
Requirement already satisfied: Pillow in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmcv-full==1.2.2) (8.0.1)
Requirement already satisfied: pyyaml in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmcv-full==1.2.2) (5.3.1)
Requirement already satisfied: yapf in /home/giang/anaconda3/envs/working/lib/python3.7/site-packages (from mmcv-full==1.2.2) (0.30.0)
Installing collected packages: mmcv-full
  Attempting uninstall: mmcv-full
    Found existing installation: mmcv-full 1.2.0
    Uninstalling mmcv-full-1.2.0:
      Successfully uninstalled mmcv-full-1.2.0
  Running setup.py develop for mmcv-full
    ERROR: Command errored out with exit status 1:
     command: /home/giang/anaconda3/envs/working/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/giang/Desktop/mmcv/setup.py'"'"'; __file__='"'"'/home/giang/Desktop/mmcv/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/giang/Desktop/mmcv/
    Complete output (111 lines):
    running develop
    running egg_info
    creating mmcv_full.egg-info
    writing mmcv_full.egg-info/PKG-INFO
    writing dependency_links to mmcv_full.egg-info/dependency_links.txt
    writing requirements to mmcv_full.egg-info/requires.txt
    writing top-level names to mmcv_full.egg-info/top_level.txt
    writing manifest file 'mmcv_full.egg-info/SOURCES.txt'
    reading manifest file 'mmcv_full.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'mmcv_full.egg-info/SOURCES.txt'
    running build_ext
    building 'mmcv._ext' extension
    creating /home/giang/Desktop/mmcv/build
    creating /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7
    creating /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv
    creating /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops
    creating /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc
    creating /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch
    Emitting ninja build file /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/build.ninja...
    Compiling objects...
    Using envvar MAX_JOBS (4) as the number of workers...
    [1/40] /usr/bin/nvcc -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/psamask_cuda.cu -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/psamask_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
    FAILED: /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/psamask_cuda.o
    /usr/bin/nvcc -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/psamask_cuda.cu -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/psamask_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
    nvcc fatal   : Unsupported gpu architecture 'compute_86'
    [2/40] /usr/bin/nvcc -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/carafe_cuda.cu -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/carafe_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
    FAILED: /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/carafe_cuda.o
    /usr/bin/nvcc -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/carafe_cuda.cu -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/carafe_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
    nvcc fatal   : Unsupported gpu architecture 'compute_86'
    [3/40] c++ -MMD -MF /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/nms_rotated.o.d -pthread -B /home/giang/anaconda3/envs/working/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/nms_rotated.cpp -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/nms_rotated.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:149:0,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
                     from /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch_cpp_helper.hpp:3,
                     from /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/nms_rotated.cpp:4:
    /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:84:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
     #pragma omp parallel for if ((end - begin) >= grain_size)
    
    [4/40] c++ -MMD -MF /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/tin_shift.o.d -pthread -B /home/giang/anaconda3/envs/working/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMMCV_WITH_CUDA -I/home/giang/Desktop/mmcv/mmcv/ops/csrc -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/TH -I/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/THC -I/home/giang/anaconda3/envs/working/include/python3.7m -c -c /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/tin_shift.cpp -o /home/giang/Desktop/mmcv/build/temp.linux-x86_64-3.7/mmcv/ops/csrc/pytorch/tin_shift.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:149:0,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                     from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
                     from /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch_cpp_helper.hpp:3,
                     from /home/giang/Desktop/mmcv/mmcv/ops/csrc/pytorch/tin_shift.cpp:1:
    /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:84:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
     #pragma omp parallel for if ((end - begin) >= grain_size)
    
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1522, in _run_ninja_build
        env=env)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/subprocess.py", line 512, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '4']' returned non-zero exit status 1.
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/giang/Desktop/mmcv/setup.py", line 213, in <module>
        zip_safe=False)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/setuptools/command/develop.py", line 136, in install_for_development
        self.run_command('build_ext')
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 653, in build_extensions
        build_ext.build_extensions(self)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
        _build_ext.build_extension(self, ext)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
        depends=ext.depends)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 482, in unix_wrap_ninja_compile
        with_cuda=with_cuda)
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1238, in _write_ninja_file_and_compile_objects
        error_prefix='Error compiling objects for extension')
      File "/home/giang/anaconda3/envs/working/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1538, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    ----------------------------------------
  Rolling back uninstall of mmcv-full
  Moving to /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/mmcv/
   from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/~mcv
  Moving to /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/mmcv_full-1.2.0.dist-info/
   from /home/giang/anaconda3/envs/working/lib/python3.7/site-packages/~mcv_full-1.2.0.dist-info
ERROR: Command errored out with exit status 1: /home/giang/anaconda3/envs/working/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/giang/Desktop/mmcv/setup.py'"'"'; __file__='"'"'/home/giang/Desktop/mmcv/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

@sundaykingg
Copy link

I had the same problem
GPU 0: GeForce RTX 3080
CUDA:11.1

@GiangHLe
Copy link
Author

@sundaykingg yes, my process raises a lot of bugs since I change to the 30XX series, even some bug with windows and Pytorch, I guess we have to wait until all the hardware match together, maybe this problem is out of MMDetection team's scope. If I was you I will downgrade to CUDA 11, it seems like there is not so much software support CUDA 11.1 now

@sundaykingg
Copy link

hi,can you try this way? #4165

@GiangHLe
Copy link
Author

GiangHLe commented Dec 17, 2020 •
edited
Loading

@sundaykingg I meet the bug above while trying to build mmcv from the source. Did you succeed?

@ZwwWayne
Copy link
Collaborator

It is caused by the 30xx GPUs. Please see the issue #4165 and our FAQ about this problem.

@LiWentomng LiWentomng mentioned this issue Dec 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ZwwWayne ZwwWayne

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@GiangHLe @sundaykingg @ZwwWayne