maskrcnn benchmark error #547

paotato · 2019-03-09T08:10:51Z

win10+anaconda3+python3.7.2+cuda9.0

ROIAlign_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (_imp??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@xz)
ROIPool_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (_imp??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@xz)
SigmoidFocalLoss_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (_imp??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@xz)
ROIAlign_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (_imp?getCurrentCUDAStream@cuda@c10@@ya?AVCUDAStream@12@F@Z)
ROIPool_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (_imp?getCurrentCUDAStream@cuda@c10@@ya?AVCUDAStream@12@F@Z)
SigmoidFocalLoss_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (_imp?getCurrentCUDAStream@cuda@c10@@ya?AVCUDAStream@12@F@Z)
build\lib.win-amd64-3.7\maskrcnn_benchmark_C.cp37-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

have someone ever met? I would appreciate it if you could give me some advice.

fmassa · 2019-03-12T06:20:27Z

Sorry, I have no experience with Windows.

septentriones · 2019-03-18T19:55:00Z

I had the same issue with the only difference being that I'm using Cuda 10.0

After some digging I discovered that c10_cuda.lib is missing from the linking call and the symbols needed are present in c10_cuda.dll. I was able to add c10_cuda.lib as an extra linker argument in setup.py and build successfully.

bpfliegel · 2019-03-23T17:53:02Z

I had the same issue with the only difference being that I'm using Cuda 10.0

After some digging I discovered that c10_cuda.lib is missing from the linking call and the symbols needed are present in c10_cuda.dll. I was able to add c10_cuda.lib as an extra linker argument in setup.py and build successfully.

@septentriones Would you let us know exactly how did you do it? I tried for an hour, but had no luck so far. Any dirty hack is welcome :) Thanks.

septentriones · 2019-03-23T18:30:34Z

Sure. I was also working through some other compile issues (among other changes I had to cast to long in SigmoidFocalLuss_cuda.cu as in this previous commit) and tried a number of fixes. Currently my setup.py includes the changes suggested here and additionally has 'c10_cuda.lib' as an argument for extra_link_args. In summary I added:

extra_compile_args = {"cxx": ['/MD']}
extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']

and modified the ext_modules as:

ext_modules = [
    extension(
        "maskrcnn_benchmark._C",
        sources,
        include_dirs=include_dirs,
        define_macros=define_macros,
        extra_compile_args=extra_compile_args,
        extra_link_args = extra_link_args,
    )
]

I hope that works for you too!

Note: What led me to figuring this out was looking for the missing symbols in c10.lib and eventually c10_cuda.lib using dumpbin and undname from the VS console.

bpfliegel · 2019-03-23T18:47:25Z

Thanks really a lot for the extensive response. It doesn't work for me yet, but I think I messed up something, because I could get the linker working earlier with modifying the command manually - and that doesn't work now either :) Will be back with the results later :)

bpfliegel · 2019-03-23T18:53:30Z

Reinstalled conda environment with pytorch nightly and philferriere's coco and finally it compiles, thanks!!! Messed up my files somehow.

One thing to mention is that besides those links you mentioned (they guided me also), is that at the moment all three *.cu files needs to be changed, with long declarations:

long ceil_div(long a, long b) {
return (a + b - 1) / b;
}

I'll try to post a more extensive guide.

septentriones · 2019-03-23T19:13:00Z

I'm glad you finally got it working! As you mention, I did also use pytorch nightly and philferrier's coco. I only needed to change the one *.cu file I mentioned earlier though because the other two *.cu files already had the change (but is important to make sure that all three are modified similarly). Cheers!

bpfliegel · 2019-03-23T19:26:13Z

Hmm, I don't see that there are changed in the master repo. I need to change all three of them. Thanks a lot again!

bpfliegel · 2019-03-23T20:45:13Z

Full guide for install

Windows 10 laptop + GTX 1050 (2019 March 23)

CUDA

Install CUDA 10.1, Custom install, Development and Runtime, nothing else, it's likely your graphics driver will be updated even when you unclick it (congrats). When you select the components and the installer hangs and all checkboxes disappear, simply click back & next again.
Open up device manager and if the driver fails, install your video card's latest driver from NVIDIA website. (I installed that also from Cuda setup first and it messed up the driver).
Download cudnn-10.1-windows10-x64-v7.5.0.56 (requires registration). Extract to a temp folder. As putting it into a folder and adding to %PATH% won't work anymore for some reason, do the following:

Copy \cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin.
Copy \cuda\include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include.
Copy \cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64.

Visual Studio and friends

Install 2017 Community, C++ dev at least. Make certain you click 'VC++ 2017 version 15.4 v14.11 toolset' below the 'Individual components' tab.
Go to https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017 and download "Build Tools For Visual Studio" under "Tools for "Visual Studio", run it. Select: 'Workloads - Visual C++ build tools' and select from individual components also 'Windows 10 SDK' and 'VC++ 2015.3 v14.00 (v140) toolset for desktop'.
Add this to your PATH environment variables: C:\Program Files (x86)\Windows Kits\10\bin\x64
Copy rc.exe & rcdll.dll from C:\Program Files (x86)\Windows Kits\8.1\bin\x86 to C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin

Restart your computer at this point in any case.

Anaconda
Install Anaconda if you didn't do it yet. We will work with a new environment called 'seg' with python 3.6 (just that we can also install Tensorflow, but maybe it's not needed any more for TF and 3.7 is fine, I don't know, go ahead with default/3.7 if you don't care). We'll also work with pytorch-nightly as even 1.0.1 doesn't contain what we need. Execute the following commands:

conda create -n seg python=3.6
activate seg
conda install ipython
pip install ninja yacs cython matplotlib tqdm requests
conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0
conda install -c anaconda git
conda install nb_conda
pip install opencv-python opencv-contrib-python
pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"

PyTorch solution

Download/unzip or git this solution, open up Anaconda prompt and navigate into the working folder.
Navigate to maskrcnn_benchmark\csrc\cuda and modify ROIAlign_cuda.cu, ROIPool_cuda.cu and SigmoidFocalLoss_cuda.cu in the following manner: replace the call for THCCeilDiv to ceil_div1,ceil_div2,ceil_div3 respectively (replace all two calls in each of these files) and add the following function (in all of these files, so for the first file it is like below, just the name changes because of proper linking later):

long ceil_div1(long a, long b) { 
	return (a + b - 1) / b; 
}

Navigate back to the root folder of the solution, open up setup.py and modify the following two parts like this:

    extra_compile_args = {"cxx": ['/MD']}
    extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']

    ext_modules = [
        extension(
            "maskrcnn_benchmark._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
            extra_link_args = extra_link_args,
        )
    ]

Go back to Anaconda command prompt and execute:

set "VS150COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
set DISTUTILS_USE_SDK=1
call "%VS150COMNTOOLS%\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build develop

Time for a beer. It works on 2019.03.23, if you are from the future, you will have another battle, but hopefully this will make it easier.
Kudos to all authors who gave hints, was too busy stealing their ideas to record their names. One beer for septentriones above for certain!
If something is not 100%, PM me, so I can update this. Cheers!

Notes:
In case you run into an empty page when launching Jupyter, don't panic. Execute:

pip uninstall tornado
pip install tornado==5.1.1

pip uninstall notebook
pip install notebook==5.7.4

Re-launch and press CTRL+Shift+R in browser. Clear cache, and repeat relaunching with CTRL+Shift+R if still empty.

bpfliegel · 2019-03-23T20:52:33Z

@fmassa ^^^ Francisco, this might be interesting to you aswell. Thanks a lot to you guys for the great job!

bpfliegel · 2019-03-26T21:16:24Z

Updated the guide, CuDNN was not working as per the initial description.

peterhsu2018 · 2019-05-15T03:52:41Z

@bpfliegel Hi, I follow your steps and got the error as below:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\nvcc.exe -c d:\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc\cuda\deform_conv_cuda.cu -o build\temp.win-amd64-3.6\Release\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc\cuda\deform_conv_cuda.obj -Xcompiler /wd4819 -Xcompiler /MD -Id:\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\TH -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include" -IC:\Users\User\Anaconda3\envs\maskrcnn\include -IC:\Users\User\Anaconda3\envs\maskrcnn\include "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.14.26428\ATLMFC\include" "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.14.26428\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
deform_conv_cuda.cu
c:\program files\nvidia gpu computing toolkit\cuda\v9.0\include\crt/host_config.h(133): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions 2012, 2013, 2015 and 2017 are supported!
error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v9.0\\bin\\nvcc.exe' failed with exit status 2

How to fix it?
Thanks a lot!

nathalia-kim · 2021-03-16T22:30:11Z

In case anyone else is trying this in 2021: #547 (comment)

Full guide for install

Windows 10 laptop + GTX 1050 (2019 March 23)

CUDA

Install CUDA 10.1, Custom install, Development and Runtime, nothing else, it's likely your graphics driver will be updated even when you unclick it (congrats). When you select the components and the installer hangs and all checkboxes disappear, simply click back & next again.

Open up device manager and if the driver fails, install your video card's latest driver from NVIDIA website. (I installed that also from Cuda setup first and it messed up the driver).

Download cudnn-10.1-windows10-x64-v7.5.0.56 (requires registration). Extract to a temp folder. As putting it into a folder and adding to %PATH% won't work anymore for some reason, do the following:
Copy \cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin.
Copy \cuda\include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include.
Copy \cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64.
Visual Studio and friends

Install 2017 Community, C++ dev at least. Make certain you click 'VC++ 2017 version 15.4 v14.11 toolset' below the 'Individual components' tab.

Go to https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017 and download "Build Tools For Visual Studio" under "Tools for "Visual Studio", run it. Select: 'Workloads - Visual C++ build tools' and select from individual components also 'Windows 10 SDK' and 'VC++ 2015.3 v14.00 (v140) toolset for desktop'.

Add this to your PATH environment variables: C:\Program Files (x86)\Windows Kits\10\bin\x64

Copy rc.exe & rcdll.dll from C:\Program Files (x86)\Windows Kits\8.1\bin\x86 to C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin

Restart your computer at this point in any case.

Anaconda
Install Anaconda if you didn't do it yet. We will work with a new environment called 'seg' with python 3.6 (just that we can also install Tensorflow, but maybe it's not needed any more for TF and 3.7 is fine, I don't know, go ahead with default/3.7 if you don't care). We'll also work with pytorch-nightly as even 1.0.1 doesn't contain what we need. Execute the following commands:
conda create -n seg python=3.6
activate seg
conda install ipython
pip install ninja yacs cython matplotlib tqdm requests
conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0
conda install -c anaconda git
conda install nb_conda
pip install opencv-python opencv-contrib-python
pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"
PyTorch solution

Download/unzip or git this solution, open up Anaconda prompt and navigate into the working folder.

Navigate to maskrcnn_benchmark\csrc\cuda and modify ROIAlign_cuda.cu, ROIPool_cuda.cu and SigmoidFocalLoss_cuda.cu in the following manner: replace the call for THCCeilDiv to ceil_div1,ceil_div2,ceil_div3 respectively (replace all two calls in each of these files) and add the following function (in all of these files, so for the first file it is like below, just the name changes because of proper linking later):
long ceil_div1(long a, long b) { 
	return (a + b - 1) / b; 
}
Navigate back to the root folder of the solution, open up setup.py and modify the following two parts like this:
    extra_compile_args = {"cxx": ['/MD']}
    extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']
    ext_modules = [
        extension(
            "maskrcnn_benchmark._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
            extra_link_args = extra_link_args,
        )
    ]
Go back to Anaconda command prompt and execute:
set "VS150COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
set DISTUTILS_USE_SDK=1
call "%VS150COMNTOOLS%\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build develop
Time for a beer. It works on 2019.03.23, if you are from the future, you will have another battle, but hopefully this will make it easier.
Kudos to all authors who gave hints, was too busy stealing their ideas to record their names. One beer for septentriones above for certain!
If something is not 100%, PM me, so I can update this. Cheers!

Notes:
In case you run into an empty page when launching Jupyter, don't panic. Execute:
pip uninstall tornado
pip install tornado==5.1.1

pip uninstall notebook
pip install notebook==5.7.4
Re-launch and press CTRL+Shift+R in browser. Clear cache, and repeat relaunching with CTRL+Shift+R if still empty.

In case anyone else is trying this in 2021:

Change this line

conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0

With this:

conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch

Install apex:

git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install

Make sure you have the following versions:

Pytorch: 1.4.0
torchvision: 0.5.0
apex: 0.1.0

This worked for me on 2021.03.16 on my Windows 10.

fmassa added the windows label Mar 12, 2019

This was referenced Mar 23, 2019

D:\\VS2015\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1120 #593

Closed

build error #590

Closed

bpfliegel mentioned this issue Mar 23, 2019

build error #583

Closed

bpfliegel mentioned this issue Mar 26, 2019

win 10 demo error RuntimeError: Not compiled with GPU support #523

Open

msqiu mentioned this issue Aug 13, 2019

Install maskrcnn-benchmark on Windows(VS2017+CUDA10.1) #1042

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

maskrcnn benchmark error #547

maskrcnn benchmark error #547

paotato commented Mar 9, 2019

fmassa commented Mar 12, 2019

septentriones commented Mar 18, 2019

bpfliegel commented Mar 23, 2019

septentriones commented Mar 23, 2019

bpfliegel commented Mar 23, 2019 •

edited

bpfliegel commented Mar 23, 2019 •

edited

septentriones commented Mar 23, 2019

bpfliegel commented Mar 23, 2019

bpfliegel commented Mar 23, 2019 •

edited

bpfliegel commented Mar 23, 2019

bpfliegel commented Mar 26, 2019

peterhsu2018 commented May 15, 2019 •

edited

nathalia-kim commented Mar 16, 2021 •

edited

maskrcnn benchmark error #547

maskrcnn benchmark error #547

Comments

paotato commented Mar 9, 2019

fmassa commented Mar 12, 2019

septentriones commented Mar 18, 2019

bpfliegel commented Mar 23, 2019

septentriones commented Mar 23, 2019

bpfliegel commented Mar 23, 2019 • edited

bpfliegel commented Mar 23, 2019 • edited

septentriones commented Mar 23, 2019

bpfliegel commented Mar 23, 2019

bpfliegel commented Mar 23, 2019 • edited

bpfliegel commented Mar 23, 2019

bpfliegel commented Mar 26, 2019

peterhsu2018 commented May 15, 2019 • edited

nathalia-kim commented Mar 16, 2021 • edited

bpfliegel commented Mar 23, 2019 •

edited

bpfliegel commented Mar 23, 2019 •

edited

bpfliegel commented Mar 23, 2019 •

edited

peterhsu2018 commented May 15, 2019 •

edited

nathalia-kim commented Mar 16, 2021 •

edited