Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: roi_align_cuda #22

Closed
ZengyuanYu opened this issue Oct 13, 2018 · 32 comments
Closed

ImportError: roi_align_cuda #22

ZengyuanYu opened this issue Oct 13, 2018 · 32 comments

Comments

@ZengyuanYu
Copy link

I compiled and test images use this example, but when I import, I go this error?
Traceback (most recent call last):
File "tools/test.py", line 9, in
from mmdet.core import results2json, coco_eval
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/core/init.py", line 6, in
from .post_processing import * # noqa: F401, F403
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .bbox_nms import multiclass_nms
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from mmdet.ops import nms
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/ops/init.py", line 2, in
from .roi_align import RoIAlign, roi_align
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from .functions.roi_align import roi_align
File "/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .. import roi_align_cuda
ImportError: cannot import name 'roi_align_cuda'

I check floder, no roi_align_cuda, How I solve it? Messges in my compile:
Building roi align op...
running build_ext
building 'roi_align_cuda' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/TH -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c src/roi_align_cuda.cpp -o build/temp.linux-x86_64-3.5/src/roi_align_cuda.o -DTORCH_EXTENSION_NAME=roi_align_cuda -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
src/roi_align_cuda.cpp: In function ‘int roi_align_forward_cuda(at::Tensor, at::Tensor, int, int, float, int, at::Tensor)’:
src/roi_align_cuda.cpp:20:80: error: ‘AT_CHECK’ was not declared in this scope
#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
^
src/roi_align_cuda.cpp:24:3: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(x);
^
src/roi_align_cuda.cpp:31:3: note: in expansion of macro ‘CHECK_INPUT’
CHECK_INPUT(features);
^
src/roi_align_cuda.cpp: In function ‘int roi_align_backward_cuda(at::Tensor, at::Tensor, int, int, float, int, at::Tensor)’:
src/roi_align_cuda.cpp:20:80: error: ‘AT_CHECK’ was not declared in this scope
#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
^
src/roi_align_cuda.cpp:24:3: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(x);
^
src/roi_align_cuda.cpp:59:3: note: in expansion of macro ‘CHECK_INPUT’
CHECK_INPUT(top_grad);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Building roi pool op...
running build_ext
building 'roi_pool_cuda' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/TH -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c src/roi_pool_cuda.cpp -o build/temp.linux-x86_64-3.5/src/roi_pool_cuda.o -DTORCH_EXTENSION_NAME=roi_pool_cuda -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
src/roi_pool_cuda.cpp: In function ‘int roi_pooling_forward_cuda(at::Tensor, at::Tensor, int, int, float, at::Tensor, at::Tensor)’:
src/roi_pool_cuda.cpp:19:80: error: ‘AT_CHECK’ was not declared in this scope
#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
^
src/roi_pool_cuda.cpp:23:3: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(x);
^
src/roi_pool_cuda.cpp:30:3: note: in expansion of macro ‘CHECK_INPUT’
CHECK_INPUT(features);
^
src/roi_pool_cuda.cpp: In function ‘int roi_pooling_backward_cuda(at::Tensor, at::Tensor, at::Tensor, float, at::Tensor)’:
src/roi_pool_cuda.cpp:19:80: error: ‘AT_CHECK’ was not declared in this scope
#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
^
src/roi_pool_cuda.cpp:23:3: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(x);
^
src/roi_pool_cuda.cpp:57:3: note: in expansion of macro ‘CHECK_INPUT’
CHECK_INPUT(top_grad);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Building nms op...
rm .so
echo "Compiling nms kernels..."
Compiling nms kernels...
python setup.py build_ext --inplace
running build_ext
building 'cpu_nms' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/cuda/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c cpu_nms.cpp -o build/temp.linux-x86_64-3.5/cpu_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821:0,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from cpu_nms.cpp:621:
/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/cpu_nms.o -L/usr/local/cuda/lib64 -lcudart -o /home/yu/mmdetection/mmdet/ops/nms/cpu_nms.cpython-35m-x86_64-linux-gnu.so
building 'gpu_nms' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/cuda/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c gpu_nms.cpp -o build/temp.linux-x86_64-3.5/gpu_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821:0,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from gpu_nms.cpp:623:
/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
/usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c nms_kernel.cu -o build/temp.linux-x86_64-3.5/nms_kernel.o -arch=sm_52 --ptxas-options=-v -c --compiler-options -fPIC
ptxas info : 0 bytes gmem
ptxas info : Compiling entry function '_Z10nms_kernelifPKfPy' for 'sm_52'
ptxas info : Function properties for _Z10nms_kernelifPKfPy
128 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 38 registers, 20480 bytes smem, 344 bytes cmem[0], 12 bytes cmem[2]
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/gpu_nms.o build/temp.linux-x86_64-3.5/nms_kernel.o -L/usr/local/cuda/lib64 -lcudart -o /home/yu/mmdetection/mmdet/ops/nms/gpu_nms.cpython-35m-x86_64-linux-gnu.so
building 'cpu_soft_nms' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/cuda/include -I/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/home/yu/.virtualenvs/Pytorch/include/python3.5m -c cpu_soft_nms.cpp -o build/temp.linux-x86_64-3.5/cpu_soft_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821:0,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from cpu_soft_nms.cpp:621:
/home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
cpu_soft_nms.cpp: In function ‘PyObject
__pyx_pf_12cpu_soft_nms_cpu_soft_nms(PyObject*, PyArrayObject*, float, float, float, unsigned int)’:
cpu_soft_nms.cpp:2450:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
__pyx_t_11 = ((__pyx_v_pos < __pyx_v_N) != 0);
^
cpu_soft_nms.cpp:2961:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
__pyx_t_11 = ((__pyx_v_pos < __pyx_v_N) != 0);
^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/cpu_soft_nms.o -L/usr/local/cuda/lib64 -lcudart -o /home/yu/mmdetection/mmdet/ops/nms/cpu_soft_nms.cpython-35m-x86_64-linux-gnu.so

@hellock
Copy link
Member

hellock commented Oct 13, 2018

mmdetection only supports the latest stable version of PyTorch (0.4.1), which version are you using?

@ZengyuanYu
Copy link
Author

mmdetection only supports the latest stable version of PyTorch (0.4.1), which version are you using?

Thank for your replay. My version is 0.4.1. I try reinstall but also don't work. My cuda version 8.0, I can used it to train model about PyTorch.

torch 0.4.1
torchfile 0.1.0
torchnet 0.0.2
torchvision 0.2.1
tornado 5.1

@hellock
Copy link
Member

hellock commented Oct 13, 2018

The CUDA version doesn't matter. The error is AT_CHECK was not declared in this scope, but AT_CHECK should have be defined in pytorch 0.4.1. I guess there exists multiple pytorch versions in your environment? Could you try creating a new environment and compiling again? If you use conda, here is a script for your reference.

@ZengyuanYu
Copy link
Author

The CUDA version doesn't matter. The error is AT_CHECK was not declared in this scope, but AT_CHECK should have be defined in pytorch 0.4.1. I guess there exists multiple pytorch versions in your environment? Could you try creating a new environment and compiling again? If you use conda, here is a script for your reference.

Okay. I use mkvirtualenv creat a new env, delete all files and replay install PyTorch 0.4.1, download mmdetection and complie. But I got the same error. Next I try to do it on another PC.

@XipengLin
Copy link

@ZengyuanYu I encounter the same error when i run compile.sh. Have you solve this problem.

@ZengyuanYu
Copy link
Author

@ZengyuanYu I encounter the same error when i run compile.sh. Have you solve this problem.

No. I can't solve it. I got the same error when I complie it on another PC.

@ZengyuanYu
Copy link
Author

The CUDA version doesn't matter. The error is AT_CHECK was not declared in this scope, but AT_CHECK should have be defined in pytorch 0.4.1. I guess there exists multiple pytorch versions in your environment? Could you try creating a new environment and compiling again? If you use conda, here is a script for your reference.

I reinstall it on another PC but don't work!

@hellock
Copy link
Member

hellock commented Oct 13, 2018

Hi @ZengyuanYu We tried different environments but cannot reproduce your reported compiling error. Could you provide some environment information?

  • OS version
  • PATH
  • PYTHONPATH
  • Python version python -V
  • PyTorch path python -c "import torch; print(torch.get_file_path())"
  • PyTorch version python -c "import torch; print(torch.__version__)"

@ZengyuanYu
Copy link
Author

Hi @ZengyuanYu We tried different environments but cannot reproduce your reported compiling error. Could you provide some environment information?

  • OS version
  • PATH
  • PYTHONPATH
  • Python version python -V
  • PyTorch path python -c "import torch; print(torch.get_file_path())"
  • PyTorch version python -c "import torch; print(torch.__version__)"

Okay!

  • 1 OS Version
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description: Ubuntu 16.04.4 LTS
    Release: 16.04
    Codename: xenial

  • 2 PATH
    /home/yu/mmdetection

  • 3 Python PATH
    original: /usr/bin/python3.5
    myenv: /home/yu/.virtualenvs/Pytorch/bin/python3.5

  • 4 Python version
    Python 3.5.2

  • 5 PyTorch Path
    print(torch.get_file_path())
    /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages

  • 6 PyTorch version
    print(torch.version)
    0.4.1

@hellock
Copy link
Member

hellock commented Oct 13, 2018

What's the output of

tail /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/ATen/Error.h

@ZengyuanYu
Copy link
Author

What's the output of

tail /home/yu/.virtualenvs/Pytorch/lib/python3.5/site-packages/torch/lib/include/ATen/Error.h

#define AT_ASSERTM(cond, ...)
if (!(cond)) {
AT_ERROR(at::str(#cond, " ASSERT FAILED at ", FILE, ":", LINE, ", please report a bug to PyTorch. ", VA_ARGS));
}

#define AT_CHECK(cond, ...)
if (!(cond)) {
AT_ERROR(at::str(VA_ARGS));
}

@hellock
Copy link
Member

hellock commented Oct 13, 2018

It seems that the PyTorch in your virtual environment is ok. I am afraid that some other PyTorch are found during the compling, but not very sure.

Here is a walkaround you can try.

Modify

#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
#define CHECK_CONTIGUOUS(x) \
  AT_CHECK(x.is_contiguous(), #x, " must be contiguous ")

to

#define CHECK_CUDA(x) AT_ASSERT(x.type().is_cuda())
#define CHECK_CONTIGUOUS(x) AT_ASSERT(x.is_contiguous())

in mmdet/ops/roi_align/src/roi_align_kernel.cu and mmdet/ops/roi_pool/src/roi_pool_kernel.cu

@ZengyuanYu
Copy link
Author

It seems that the PyTorch in your virtual environment is ok. I am afraid that some other PyTorch are found during the compling, but not very sure.

Here is a walkaround you can try.

Modify

#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x, " must be a CUDAtensor ")
#define CHECK_CONTIGUOUS(x) \
  AT_CHECK(x.is_contiguous(), #x, " must be contiguous ")

to

#define CHECK_CUDA(x) AT_ASSERT(x.type().is_cuda())
#define CHECK_CONTIGUOUS(x) AT_ASSERT(x.is_contiguous())

in mmdet/ops/roi_align/src/roi_align_kernel.cu and mmdet/ops/roi_pool/src/roi_pool_kernel.cu

I do it but it's not work! If some other PyTorch found during the compling, I will delete all env in my PC and reinstall .

@hongdayu
Copy link

I also encountered such a problem, upgrading 4.0 to 4.1 is fine.

@fabienbaradel
Copy link

Hi @hellock , thanks for your script that's awesome.
I can build the roi align op and the roi pool op but I get a failure when Building nms op....
Below is the full log:

Building roi align op...
running build_ext
building 'roi_align_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/TH -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/THC -I/home/fbaradel/cudnn/cuda/include -I/home/fbaradel/anaconda3/envs/open-mmlab/include/python3.6m -c src/roi_align_cuda.cpp -o build/temp.linux-x86_64-3.6/src/roi_align_cuda.o -DTORCH_EXTENSION_NAME=roi_align_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/fbaradel/cudnn/cuda/bin/nvcc -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/TH -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/THC -I/home/fbaradel/cudnn/cuda/include -I/home/fbaradel/anaconda3/envs/open-mmlab/include/python3.6m -c src/roi_align_kernel.cu -o build/temp.linux-x86_64-3.6/src/roi_align_kernel.o -DTORCH_EXTENSION_NAME=roi_align_cuda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options '-fPIC' -std=c++11
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
creating build/lib.linux-x86_64-3.6
g++ -pthread -shared -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -L/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,-rpath=/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/src/roi_align_cuda.o build/temp.linux-x86_64-3.6/src/roi_align_kernel.o -L/home/fbaradel/cudnn/cuda/lib64 -lcudart -o build/lib.linux-x86_64-3.6/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.6/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so ->
Building roi pool op...
running build_ext
building 'roi_pool_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/TH -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/THC -I/home/fbaradel/cudnn/cuda/include -I/home/fbaradel/anaconda3/envs/open-mmlab/include/python3.6m -c src/roi_pool_cuda.cpp -o build/temp.linux-x86_64-3.6/src/roi_pool_cuda.o -DTORCH_EXTENSION_NAME=roi_pool_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/fbaradel/cudnn/cuda/bin/nvcc -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/TH -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/torch/lib/include/THC -I/home/fbaradel/cudnn/cuda/include -I/home/fbaradel/anaconda3/envs/open-mmlab/include/python3.6m -c src/roi_pool_kernel.cu -o build/temp.linux-x86_64-3.6/src/roi_pool_kernel.o -DTORCH_EXTENSION_NAME=roi_pool_cuda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options '-fPIC' -std=c++11
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
creating build/lib.linux-x86_64-3.6
g++ -pthread -shared -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -L/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,-rpath=/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/src/roi_pool_cuda.o build/temp.linux-x86_64-3.6/src/roi_pool_kernel.o -L/home/fbaradel/cudnn/cuda/lib64 -lcudart -o build/lib.linux-x86_64-3.6/roi_pool_cuda.cpython-36m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.6/roi_pool_cuda.cpython-36m-x86_64-linux-gnu.so ->
Building nms op...
rm *.so
rm: impossible de supprimer '*.so': Aucun fichier ou dossier de ce type
Makefile:8 : la recette pour la cible « clean » a échouée
make: *** [clean] Erreur 1
echo "Compiling nms kernels..."
Compiling nms kernels...
python setup.py build_ext --inplace
Compiling cpu_nms.pyx because it changed.
Compiling gpu_nms.pyx because it changed.
Compiling cpu_soft_nms.pyx because it changed.
[1/3] Cythonizing cpu_nms.pyx
/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/fbaradel/Github_Repos/mmdetection/mmdet/ops/nms/cpu_nms.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
[2/3] Cythonizing cpu_soft_nms.pyx
/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/fbaradel/Github_Repos/mmdetection/mmdet/ops/nms/cpu_soft_nms.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
[3/3] Cythonizing gpu_nms.pyx
/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/fbaradel/Github_Repos/mmdetection/mmdet/ops/nms/gpu_nms.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: gpu_nms.pyx:41:18: Use boundscheck(False) for faster access
warning: gpu_nms.pyx:41:45: Use boundscheck(False) for faster access
running build_ext
building 'cpu_nms' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/cuda/include -I/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/numpy/core/include -I/home/fbaradel/anaconda3/envs/open-mmlab/include/python3.6m -c cpu_nms.cpp -o build/temp.linux-x86_64-3.6/cpu_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
                 from /home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from cpu_nms.cpp:664:
/home/fbaradel/anaconda3/envs/open-mmlab/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
g++ -pthread -shared -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -L/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,-rpath=/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu_nms.o -L/usr/local/cuda/lib64 -lcudart -o /home/fbaradel/Github_Repos/mmdetection/mmdet/ops/nms/cpu_nms.cpython-36m-x86_64-linux-gnu.so
/home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat/ld: cannot find -lcudart
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
Makefile:4 : la recette pour la cible « all » a échouée
make: *** [all] Erreur 1

Do you have any idea where the problem comes from?
Thanks in advance

@hellock
Copy link
Member

hellock commented Oct 16, 2018

Hi @fabienbaradel, you may try this, supposing that /usr/local/cuda is your cuda root path.

export $LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"
./compile.sh

@fabienbaradel
Copy link

Thanks for your fast reply, it does not help I still have the same error. I think the problem comes from the last lines:

g++ -pthread -shared -B /home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat -L/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,-rpath=/home/fbaradel/anaconda3/envs/open-mmlab/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu_nms.o -L/usr/local/cuda/lib64 -lcudart -o /home/fbaradel/Github_Repos/mmdetection/mmdet/ops/nms/cpu_nms.cpython-36m-x86_64-linux-gnu.so
/home/fbaradel/anaconda3/envs/open-mmlab/compiler_compat/ld: cannot find -lcudart
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

The problem may come from the fact that it cannot find -lcudart and collect2.
Do I need to change some lines of a file?

@OceanPang
Copy link
Collaborator

OceanPang commented Oct 16, 2018

Hi @fabienbaradel , I found that during compiling roi align, the cuda lib path is /home/fbaradel/cudnn/cuda/lib64, but the later is /usr/local/cuda/lib64. Please check is there any mismatch or file lost.

Update: The CUDA_ROOT in nms is set default as /usr/local/cuda/lib64. We will make it more flexible quickly.

@fabienbaradel
Copy link

fabienbaradel commented Oct 16, 2018

Thanks a lot @OceanPang, it solves my problem (updating CUDA_ROOT in nms 👍 )

@hellock
Copy link
Member

hellock commented Oct 16, 2018

The nms op compiling problem should have been fixed in #35.

@ybpaopao
Copy link

@ZengyuanYu Hi, I also met the same problem. I wonder whether you solve it? Upgrading pytorch0.4.1 seems useless to me. Do you find any effective method?

@zl535320706
Copy link

zl535320706 commented Oct 17, 2018

Hi @ZengyuanYu We tried different environments but cannot reproduce your reported compiling error. Could you provide some environment information?

  • OS version
  • PATH
  • PYTHONPATH
  • Python version python -V
  • PyTorch path python -c "import torch; print(torch.get_file_path())"
  • PyTorch version python -c "import torch; print(torch.__version__)"

Hi, @fabienbaradel ,I also met the same problem, even can not pass "building 'gpu_nms' extension"

Building roi align op...
running build_ext
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py:118: UserWarning: 

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++ 4.8) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
building 'roi_align_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/TH -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c src/roi_align_cuda.cpp -o build/temp.linux-x86_64-3.6/src/roi_align_cuda.o -DTORCH_EXTENSION_NAME=roi_align_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
/usr/local/cuda/bin/nvcc -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/TH -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c src/roi_align_kernel.cu -o build/temp.linux-x86_64-3.6/src/roi_align_kernel.o -DTORCH_EXTENSION_NAME=roi_align_cuda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options '-fPIC' -std=c++11
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/ATen/Half-inl.h(17): error: identifier "__half_as_short" is undefined

1 error detected in the compilation of "/tmp/tmpxft_00007ea3_00000000-7_roi_align_kernel.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 2
Building roi pool op...
running build_ext
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py:118: UserWarning: 

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++ 4.8) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
building 'roi_pool_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/TH -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c src/roi_pool_cuda.cpp -o build/temp.linux-x86_64-3.6/src/roi_pool_cuda.o -DTORCH_EXTENSION_NAME=roi_pool_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
/usr/local/cuda/bin/nvcc -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/TH -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c src/roi_pool_kernel.cu -o build/temp.linux-x86_64-3.6/src/roi_pool_kernel.o -DTORCH_EXTENSION_NAME=roi_pool_cuda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options '-fPIC' -std=c++11
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/lib/include/ATen/Half-inl.h(17): error: identifier "__half_as_short" is undefined

1 error detected in the compilation of "/tmp/tmpxft_00007fa2_00000000-7_roi_pool_kernel.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 2
Building nms op...
rm -f *.so
echo "Compiling nms kernels..."
Compiling nms kernels...
python setup.py build_ext --inplace
running build_ext
building 'cpu_nms' extension
gcc -pthread -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c cpu_nms.cpp -o build/temp.linux-x86_64-3.6/cpu_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from cpu_nms.cpp:612:
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
g++ -pthread -shared -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -L/home/zhangli/anaconda3/envs/pytorch/lib -Wl,-rpath=/home/zhangli/anaconda3/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu_nms.o -o /home/zhangli/mmdetection/mmdet/ops/nms/cpu_nms.cpython-36m-x86_64-linux-gnu.so
building 'cpu_soft_nms' extension
gcc -pthread -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c cpu_soft_nms.cpp -o build/temp.linux-x86_64-3.6/cpu_soft_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from cpu_soft_nms.cpp:612:
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
cpu_soft_nms.cpp: In function ‘PyObject* __pyx_pf_12cpu_soft_nms_cpu_soft_nms(PyObject*, PyArrayObject*, float, float, float, unsigned int)’:
cpu_soft_nms.cpp:2441:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       __pyx_t_11 = ((__pyx_v_pos < __pyx_v_N) != 0);
                                    ^
cpu_soft_nms.cpp:2952:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       __pyx_t_11 = ((__pyx_v_pos < __pyx_v_N) != 0);
                                    ^
g++ -pthread -shared -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -L/home/zhangli/anaconda3/envs/pytorch/lib -Wl,-rpath=/home/zhangli/anaconda3/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu_soft_nms.o -o /home/zhangli/mmdetection/mmdet/ops/nms/cpu_soft_nms.cpython-36m-x86_64-linux-gnu.so
building 'gpu_nms' extension
gcc -pthread -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c gpu_nms.cpp -o build/temp.linux-x86_64-3.6/gpu_nms.o -Wno-unused-function -Wno-write-strings
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from gpu_nms.cpp:614:
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
nvcc -I/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/core/include -I/home/zhangli/anaconda3/envs/pytorch/include/python3.6m -c nms_kernel.cu -o build/temp.linux-x86_64-3.6/nms_kernel.o -c --compiler-options -fPIC
g++ -pthread -shared -B /home/zhangli/anaconda3/envs/pytorch/compiler_compat -L/home/zhangli/anaconda3/envs/pytorch/lib -Wl,-rpath=/home/zhangli/anaconda3/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/gpu_nms.o build/temp.linux-x86_64-3.6/nms_kernel.o -o /home/zhangli/mmdetection/mmdet/ops/nms/gpu_nms.cpython-36m-x86_64-linux-gnu.so

and my environment informations are as follow:

1.OS version:
Ubuntu 14.04.5 LTS
2.PATH:
/home/zhangli/anaconda3/envs/pytorch/bin:/usr/local/cuda/bin:/home/zhangli/cmake-3.12.2/bin:/home/zhangli/htop_1.0.2/bin:/home/zhangli/anaconda3/bin:/usr/local/cuda/bin:/usr/local/cuda-8.0/bin:/usr/local/cuda-7.5/bin:/usr/local/cuda-8.0/bin:/usr/local/cuda-7.0/bin:/usr/local/cuda/bin:usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
3.PYTHONPATH
/home/zhangli/anaconda3/envs/pytorch/bin/python
4.Python version
Python 3.6.5
5.PyTorch path
/home/zhangli/anaconda3/envs/pytorch/lib/python3.6/site-packages
6.PyTorch version
0.4.1

@OceanPang
Copy link
Collaborator

Hi @zl535320706 , there seems no error when building nms, but indeed errors when building roi align and roi pooling. The error is error: identifier "__half_as_short" is undefined. Please check is there a cuda_fp16.h in your CUDA_PATH/include, and you are using a correct nvcc.
Related pytorch_scatter/issues/8

@zl535320706
Copy link

zl535320706 commented Oct 17, 2018

Hi @zl535320706 , there seems no error when building nms, but indeed errors when building roi align and roi pooling. The error is error: identifier "__half_as_short" is undefined. Please check is there a cuda_fp16.h in your CUDA_PATH/include, and you are using a correct nvcc.
Related pytorch_scatter/issues/8

Thank you, follow your guide, I found the same error, and know that because my cuda version is older(V8.0.61), so the NVCC could not work well. So, compile mmdection need cuda 9.0 at least.

@hellock
Copy link
Member

hellock commented Oct 17, 2018

@zl535320706 Thanks for your feedback and we will take a look and test the cuda compatibility.

@chelixuan
Copy link

The CUDA version doesn't matter. The error is AT_CHECK was not declared in this scope, but AT_CHECK should have be defined in pytorch 0.4.1. I guess there exists multiple pytorch versions in your environment? Could you try creating a new environment and compiling again? If you use conda, here is a script for your reference.

Okay. I use mkvirtualenv creat a new env, delete all files and replay install PyTorch 0.4.1, download mmdetection and complie. But I got the same error. Next I try to do it on another PC.

hi~ I have the same problem with you. Have you solve it?
@ZengyuanYu

@ZengyuanYu
Copy link
Author

@ZengyuanYu Hi, I also met the same problem. I wonder whether you solve it? Upgrading pytorch0.4.1 seems useless to me. Do you find any effective method?

No, I can't solve it. I'm try to reinstall all envs!

@ybpaopao
Copy link

@ZengyuanYu

Hi, I have successfully performed the setup. My recommendation is that you can try cuda 9.0. There may be mistake in nvcc if you use cuda8 but i do not know how to fix it. Wish this works for you.

@ZengyuanYu
Copy link
Author

@ZengyuanYu

Hi, I have successfully performed the setup. My recommendation is that you can try cuda 9.0. There may be mistake in nvcc if you use cuda8 but i do not know how to fix it. Wish this works for you.

Yes! Thank you. I sucess use CUDA 9.0

@ZengyuanYu
Copy link
Author

@hellock Hi, This error is solved by CUDA 8.0->CUDA 9.0. When I compile, Both message is the same. So This is incredible.Thank you for your help, I wish you a happy weekend.I will continue to pay attention to mm-detection!

@Markovcom
Copy link

@zl535320706 hello,i meet the same problem, how to solve? thank you!

@zl535320706
Copy link

@zl535320706 hello,i meet the same problem, how to solve? thank you!

emmm, I don't solve the problem, because our server is used for public, and I have no root right to update the NVIDIA driver version.

@chaos1992 chaos1992 mentioned this issue May 8, 2021
liuhuiCNN pushed a commit to liuhuiCNN/mmdetection that referenced this issue May 21, 2021
FANGAreNotGnu pushed a commit to FANGAreNotGnu/mmdetection that referenced this issue Oct 23, 2023
* resource

* mv tasks into object method
FANGAreNotGnu pushed a commit to FANGAreNotGnu/mmdetection that referenced this issue Oct 23, 2023
* init

* Adding Hyperband (open-mmlab#4)

* refactor changes

* grammer

* asyc hyperband

* Initial commit (open-mmlab#5)

* Add dataset sanity check (open-mmlab#7)

* release resources (open-mmlab#6)

* Add dataset histogram viz and check (open-mmlab#8)

* Add dataset histogram viz and check

* Add matplotlib in setup

* Checkerpoint (open-mmlab#10)

* release resources

* rename example fils

* keep track of the best result

* serialization

* save load

* add util

* checkpoint and resume

* keeping task id

* terminator state

* rm comments

* Add autogluon backend fit and refine apis (open-mmlab#9)

* add autogluon backend fit and refine apis

* update

* update

* add some doc

* refine

* refine fit

* refine fit

* refine fit

* add guideline (open-mmlab#11)

* Add Plots for Visualization (open-mmlab#12)

* add plots

* current progress

* rm comment

* Refine fit (open-mmlab#13)

* refine fit

* minor update

* fix setup (open-mmlab#14)

* fix guide (open-mmlab#15)

* Add autogluon notebook (open-mmlab#16)

* add notebook

* update notebook

* Demo patch 1 (open-mmlab#17)

* mv dataset inside

* patch

* Demo (open-mmlab#20)

* fix

* fix 1

* add notebook

* setup (open-mmlab#19)

* Fix Checkerpoint (open-mmlab#22)

* resource

* mv tasks into object method

* Fixtypo (open-mmlab#23)

* Revert "Demo (open-mmlab#20)"

This reverts commit a8fa993b461b8cd424edbe772fe6b0264f6ee79a.

* fix

* Update AutoGluon Notebook (open-mmlab#24)

* Update notebook

* remove

* raise warning for resource (open-mmlab#25)

* [WIP] AutoGluon Distributed (open-mmlab#26)

* remote resource management

* add files

* remote resource management

* distributed scheduler

* add autogluon.distributed scheduler (open-mmlab#28)

* add cifar script and tensorboard (open-mmlab#27)

* patch for state-dict (open-mmlab#29)

* distributed with ssh helper (open-mmlab#31)

* ssh helper for distributed

* tutorial

* Refactor api and update image classification results (open-mmlab#30)

* refactor mxboard api and update img classification results

* Update notebook to work on mac

* update notebook and compact svg

* Multiprocess Queue Support MacOS (open-mmlab#33)

* Queue for Mac OS

* add queue

* Backend Tutorials (open-mmlab#32)

* init tutorial

* add figures

* add figures

* add comments

* merge and demo

* add plot

* img path

* Refine notebook and add dataset statistics (open-mmlab#34)

* refactor mxboard api and update img classification results

* Update notebook to work on mac

* update notebook and compact svg

* refine notebook and dataset

* add conda

* rm ipynb

* update notebook and dataset

* uncommnent dist

* notebook results update (open-mmlab#35)

* Add MINC experiments and Refine Data Loss Metric (open-mmlab#36)

* add minc exp

* fix bug

* add auto loss and metric

* update minc results

* fix kwargs (open-mmlab#37)

* Refine auto Dataset, Nets, Losses, Metrics, Optimizers and Pipeline (open-mmlab#38)

* add comments

* fix

* refine dataset

* Add Kaggle Shopee Classification script (open-mmlab#40)

* add kaggle shopee img classification example

* update results

* Update .gitignore

* Distributed FIFO and Bug Fix (open-mmlab#39)

* simple visualizer

* distributed scheduler progress

* local node message

* distributed fifo okay

* Add local helper (open-mmlab#42)

* add local helper

* Add Distributed ImageNet Example (open-mmlab#43)

* fix img dataset (open-mmlab#45)

* Add object detection (open-mmlab#41)

* add object detection voc

* fix

* update results and fix some issues

* fix search space

* update obj detection results

* Dist-hyperband, Doc and Tutorial (open-mmlab#48)


* dist hyperband

* add docs

* Refactor fit and dataset api (open-mmlab#50)

* advance api

* initial commit

* status

* advance api

initial commit

rm

* fix example issue (open-mmlab#51)

* current progress

* save model params (open-mmlab#53)

* add save model params

* add missing file

* resume at any point

* add missing import

* fix hyperband

* dist not implemented

* add tutorial doc (open-mmlab#55)

* mxutils

* add example and notebook

* add fit tutorial

* add notebook file

* Text Classification (open-mmlab#6)

* Initial commit for Text Classification classes

* Added results obejct in core.py

* Added Estimator package

* Rebase

* Added PyTest_Cache to git ignore

* Added FTML Optimizer

* Added impl for core.py

* Added method signatures for text classification model zoo

* Added typing hints to nets.py

* Wrapped up implementation of dataset to yield dataloaders

* Added TextData Transforms and Dataset Utils

* Added impl for pipeline

* Fixed errors + formatting commit

* Added beginner example for text_classification for sst dataset

* Added handler for data loader

* Refined DataLoaderHandler

* Printind the exception stack trace if any

* Replaced print with logs

* Fixed syntax error

* Changed default GPU counts

* Changed trial scheduler to default

* Changed Max_Training_Epochs to 5

* Fixed syntax error for string formatting

* Added metrics to the reporter

* Fixed reporter issues

* Uncommented plot training curves

* Fix import error

* Made reporter a handler

* Fixed args issue

* Added exponential search space

* Added batch_size as a hyperparam to dataset class

* Added more models to text_classification

* Removed big rnn for now

* Added rules for tokenization of text data

* Now printing network architecture as well

* Changed the rules for tokenization

* Added Dropouts and Dense Layers as a hyperparam

* Added todo to fine tune LM

* Changed upper bound for batch size

* Now printing task ID as well along with the result

* Now added task ID to the reporter as well

* Added num_gpus to the args

* Added unit tests (dummy for now)

* Added skeleton for autogluon initializers

* Added demo jupyter notebooks

* Updated IMDB notebook

* Updated Demo notebook for Stanford Sentiment Treebank dataset

* Added NER base structure

* adding pipeline + model zoo for NER

* adding LR warmup handler

* NER CoNLL2003 dataset

* NER dataset format conversion

* Added NER HPO codebase

* adding core + example for NER

* update pipeline, dataset, core

* fixes

* add eval helper code

* move data proc code to utils, fixes

* Added WNUT2017 dataset support

* fix num_classes

* fix num_classes

* add bertadam optimizer

* pre-defined parameters

* Increased the maximum sequence length

* move helper code to task utils

* Modified dataset preprocessing code

* fix class name, rebase

* fix

* Added comments for modifying and copying the NER data methods from GluonNLP toolkit

* The WNUT-2017 dataset now downloads automatically, user just needs to pass the dataset name

* pylint check(round 1)

* pylint check(round 2) and import seqeval library for fetching some NER data methods

* add multi-gpu support

* pylint check(round 3)

* Minor coding formats fix

* fix multi-gpu, working version

* Cleanup

* Minor code fix

* add default params for datasets

* Minor contructor fix

* update default seq len for wnut17

* adding demo notebook

* update demo notebook

* update notebook

* add early stopping

* update net construction config

* Initial commit for making MXBoard/TensorBoard as a handler to pass to the estimator

* Added TensorBoard requirements

* Added TensorBoard support to Text classification in the form of a handler

* Refactored the transforms, speeding up the data len functions

* Added dataset name for BERT

* Added BERTAdam optimizer for BERT

* Added BERT Networks

* Added BertClassifier block

* Added support for Bert Model to the pipeline

* Added DataLoader handler for BERT

* Now passing BertDataLoaderHandler to the Estimator, instead of using SentimentDataLoaderHandler()

* Added support for BERT Models and refactored pipeline.py in text-classification

* Bunch of pycharm formatting changes

* Added example classes for Glue SST2, MNLI and Yelp Datasets

* Fixed a typo for val set

* Fixed LR range issue

* Fixed missing argument to function call

* Fixed typo in model_zoo

* add support for ontonotes-v5, auto max seq len, cleanup

* [WIP] Unittest for Named Entity Recognition

* Added more unittest for Named Entity Recognition

* adding NER integration tests

* Added more integration test methods for NER

* Added nosetest module for NER

* fix nets, optims, batch_size api for NER & add advanced user example

* refactor Scheduler to pull out Terminator

* add missing files/fix Terminator

* set cpu affinity

* assign cpu affinity within the task

* integer casting

* terminator updates

* adding jnlpba, bc5cdr datasets + fixes

* Moved dataset/utils to text_classification_dataset/utils

* Added placeholder for buildspec.yml and pylintrc

* Refactored setup.py

* Added bdist info

* Fixed setup.py issue

* Added requirements.txt and reading it in setup.py

* Fixed wrong mapping of Sent : Label when reading tsv dataset

* Added Train Field indices and Val Field indices for TSV Datasets as kwargs

* Fixed issue of loading data lengths by using multi processing

* Added LR Warmup Handler use to Text Classification's pipeline

* Now plotting Train metrics as well at epoch end as well as fixed index issues while reading MNLI dataset

* Added support for GLUE - MRPC Dataset

* Updated the download dataset method

* Removed vocab getters and setters from dataset

* Now loading json files as SimpleDataset and removed methods to load dataset from gluonnlp

* Moved transforms from dataset to task

* Added num_workers parameter to the dataset

* Added losses/metrics and moved dataset class inside TC.task

* Removed core.py as it's not needed anymore

* Reduced code duplication by creating a lightweight dataset class

* Added MXBoard Handler to estimator

* Removed uncommented code for fifo scheduler

* Now printing the exception along with its stacktrace

* Now printing the exception along with its stacktrace

cr https://code.amazon.com/reviews/CR-11188375

* Added reading of datasets in .txt format

* Removed NER task from the CR

* Added support for multi-sentence in TextDataTransform

* Added support for multi-text datasets

* Added task specific optimizers for text classification

* Removed task-id from the reporter

* Removed task_id and EPOCH_END callback from reporter

* Removed big RNN and en_de_transformer

* Now making DataLoaderHandler a single class

* Removed ClassificationHead class

* Renamed init_env to init_hparams

* Removed MXBoard Handler

* Separated model, dataset, transforms from the method

* Added dataset.py to read GluonNLP Datasets

* Refactored the dataset class

* Fixing import issues

* Undoing formatting changes

* Undoing formatting changes

* Fixed issue with return of Batchify_Fn

* Now updating validation dataset labels as well

* Removed extra files from examples folder

* Undoing the CI changes

* Removed split and load and instead now calling nlp.split and load

* Removed initializer folder for now

* Removed Exponential

* Added _Dataset to read the different formats

* Undoing formatting changes

* Removed unused files

* Added deleted import for version

* Now passing results back to scheduler via reporter

* Addressed PR comments

* Refactor get_transform_fn into task.dataset

* Addressed PR Comments

* allow uploading files

* exception handl

* reporter

* add train val split

* split reko datasets

* handle pipeerror

* wip

* rm unused

* advanced API

* rm print

* try error

* import ok

* wip cifar training ok

* advanced api wip

* add missing file

* call method

* current progress

* controller sample okay

* rl progress, controller sample okay

* rl cifar example training okay

* rm comment

* Skopt searcher (open-mmlab#4)

* Added skopt_searcher.py for BayesOpt search routine + unit-test comparing this searcher against the RandomSampling searcher on a toy optimization problem. Remaining TODOs: 1) include script to benchmark skopt_searcher against Hyperband in real autogluon image-classification task. 2) There is an issue that get_config() may become stuck in infinite while loop (for all searchers). There is no termination condition to handle the case where all possible configs have already been tried (should be inherited from BaseSearcher or Scheduler should automatically terminate).

* edited BaseTask to allow for skopt Bayesian Optimization hyperparameters search via additional searcher argument value 'bayesopt'. Added train_cifar10.py example to compare random search with BayesOpt search (under default settings for all other flags in this script). Results are in new table added to scripts/image_classification/README.md

* Rebased master into skopt. Cleaned up documentation/comments to be more presentable

* rl training

* viz

* rl controller running okay

* rl controller state dict ready

* add dependencies

* update fit etc

* test pipeline

* merge hackthon docs

* add nas progress

* reorganize folders

* working progress

* major features

* Hackathon version freeze (open-mmlab#11)

* add image classification notebook and update api doc

* address comments

* update

* update result

* add functinoalities

* update tutorials

* update tutorial

* update

* update

* update mds

* update fit etc

* update fit hackathon

* update

* handle pipeerror

* update+

* add skopt

* add searcher and scheduler notebooks

* New version of image_classification_searcher.md	

Has a couple of remaining TODOs.  The biggest issue I see is that fit() in base_task.py cannot take any keyword arguments for constructing the Searcher.

* how to pass keyword args to searcher

removed all TODOs as well. This notebook execution still needs to be tested with the new base_task.fit code that uses 'searcher_options' as a kwarg.  

Something that is still missing from this tutorial is what is the hyperparameter search space that is actually being searched here?  A curious user probably wants to know this information.
I would add a short section right before "## Random hyperparameter search" to clarify this, for example:

By default, `image_classification.fit()` will search for hyperparameter values within the following search space: 
# TODO: explicitly list the default search space.

* added searcher_options

* add lr scheduler

* update example

* Update image_classification_scheduler.md

* minor typo correction

dict definitions for searcher_options corrected

* fix own dataset

* update large dataset test

* fixed skopt bug to handle ValueError exception

* Update image_classification_scheduler.md

* Update image_classification_scheduler.md

* try to fix large data

* address comments

* test large data

* fix pipeline

* test predict batch

* fix

* test pipeline

* img name

* Freeze autogluon version

* fit running

* add utils

* fit running, pending final fit

* evaluate

* docs

* fix merge error

* fix merge error

* fix

* fix

* init method for autogluon object attr

* docs compiled

* documentation

* rm sub docs

* address merging error

* address soem comments

* choice

* choice

* fix typo

* docs improvement

* address comments

* change list with choice

* rename

* fix typo

* address some comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants