OSError: /usr/local/lib/libcaffe2_detectron_ops_gpu.so: undefined symbol:

[jma100]

I tried some solutions in issues, like exporting python path, but it didn't work for me.

/home/chopin/anaconda3/envs/caffe2/caffe2/build/caffe2/python to ~/.bashrc and then pip install utils .

I saw somewhere talking about the opencv version. I have 3.3.1 and the required version is 3.4.1. But caffe2 installed through conda (caffe2-cuda9.0-cudnn7) doesn't work with opencv 3.4.1.

Expected results

I was running this line:

python launch.py --cfg configs/video/2d_best/01_R101_best_hungarian-4GPU.yaml --mode test TEST.WEIGHTS pretrained_models/configs/video/2d_best/01_R101_best_hungarian.yaml/model_final.pkl

Actual results

And then I ran into this error:

Traceback (most recent call last):
File "tools/test_net.py", line 33, in
utils.c2.import_detectron_ops()
File "/home/jingweim/DetectAndTrack/lib/utils/c2.py", line 50, in import_detectron_ops
dyndep.InitOpsLibrary(detectron_ops_lib)
File "/home/jingweim/anaconda2/envs/detect_and_track/lib/python2.7/site-packages/caffe2/python/dyndep.py", line 35, in InitOpsLibrary
_init_impl(name)
File "/home/jingweim/anaconda2/envs/detect_and_track/lib/python2.7/site-packages/caffe2/python/dyndep.py", line 48, in _init_impl
ctypes.CDLL(path)
File "/home/jingweim/anaconda2/envs/detect_and_track/lib/python2.7/ctypes/init.py", line 366, in init
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/libcaffe2_detectron_ops_gpu.so: undefined symbol: _ZN6caffe28TypeMeta2IdINS_6TensorINS_11CUDAContextEEEEENS_11CaffeTypeIdEv

System information

• Operating system: Ubuntu 16.04
• Compiler version: gcc 5.4
• CUDA version: 9.0
• cuDNN version: 7.0
• NVIDIA driver version: 396
• GPU models (for all devices if they are not all the same): TITAN XP
• PYTHONPATH environment variable: /home/jingweim/anaconda2/envs/detect_and_track/include/caffe2/python:/usr/bin/python
• python --version output: Python 2.7.15 :: Anaconda custom (64-bit)
• Anything else that seems relevant: opencv 3.3.1

[zhangboshen]

Same problems here, I tried add PYTHONPATH, but it didn't help..

[lz199144]

same issue, have you solved?

[7crystle7]

When I use source code to make and install caffe2 following the official instruction, such error disappear.

[ir413]

Hi all, have you verified your Caffe2 installation by running one of Caffe2 tests? (e.g. this one). Also, are you using Caffe2 installed with Anaconda?

[zhangboshen]

@ir413 I used source code to make and install Caffe2 followed official instruction, and I also tested caffe2 installation as requested, but I still get this error.

[jma100]

I guess don't use the anaconda caffe2. Use this one https://github.com/pytorch/pytorch/tree/master/caffe2
I got it to work

[BanuSelinTosun]

I used the instructions in here, and did not get it to work: https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile
Later, I installed caffe2 with this code: conda install -c caffe2 caffe2-cuda9.0-cudnn7
and tested it with python -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'
got the correct answer as 1.
I finished the detectron instructions here: https://github.com/facebookresearch/Detectron/blob/b2d24147a9c614e04389b0b7dcbbf00fedc45e8d/INSTALL.md for caffe2, cocoapi, and until the last line of detectron.
And started getting this error when I run the test code: python2 $DETECTRON/detectron/tests/test_spatial_narrow_as_op.py
Which is the same exact error as above. Is this caffe2 installation error or incompatibility issue from detectron. And either way, what is the suggestion to resolve this error?
Thanks,
Selin

[jma100]

Hi Selin, just don't use conda to install caffe2. Get it from https://github.com/pytorch/pytorch/tree/master/caffe2. There are some opencv issues if you use conda

[BanuSelinTosun]

That makes sense. I started on the path of installing caffe2 in a different conda environment with these instructions: https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile

But now this is the error I am getting when the cmake is run at the very end:

[gadcam]

@BanuSelinTosun
The instructions read

Anaconda users: To build with Anaconda, follow the instructions on the Mac page.

So I think you can try again using these instructions https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=compile#anaconda-install-path

[BanuSelinTosun]

@gadcam Thanks, that was my second approach on the list. Though, I am using Ubuntu, not Mac. Or does Mac Page mean something else? 😃

I am running this:
./scripts/build_anaconda.sh --install-locally --cuda 8.0 --cudnn 6

The caffe2 installation looks good and passes the test. However, the detectron test is still failing:

[gadcam]

no kernel image is available for execution on the device

Looks like you found #260 so let us continue in #260.

[MonkeyNi]

I have met the same problem. And I solve the problem by rename this 'libcaffe2_detectron_ops_gpu.so' file in the following path 'OSError: /usr/local/lib/libcaffe2_detectron_ops_gpu.so'. Since it should reference 'pytorch/build/lib/libcaffe2_detectron_ops_gpu.so' which is under your caffe2 build path.
The reason for this problem is mostly due to the installation of more than one version of caffe2. Hope this be helpful.

[lilichu]

hi! @MonkeyNi how to rename 'libcaffe2_detectron_ops_gpu.so'? I can not understand it. could you please help me? thanks!

[lilichu]

well, when I install the new caffe2, and run python detectron/tests/test_spatial_narrow_as_op.py I encounter the same issue.

Found Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so
Traceback (most recent call last):
  File "detectron/tests/test_spatial_narrow_as_op.py", line 88, in <module>
    c2_utils.import_detectron_ops()
  File "/home/user/backup/detectron/detectron/utils/c2.py", line 42, in import_detectron_ops
    dyndep.InitOpsLibrary(detectron_ops_lib)
  File "/home/user/pytorch/build/caffe2/python/dyndep.py", line 35, in InitOpsLibrary
    _init_impl(name)
  File "/home/user/pytorch/build/caffe2/python/dyndep.py", line 48, in _init_impl
    ctypes.CDLL(path)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/libcaffe2_detectron_ops_gpu.so: undefined symbol: _ZN5fLI6438FLAGS_caffe2_max_keep_on_shrink_memoryE

I used to install caffe2 from source, now I reinstall it. In new caffe2, libcaffe2_detectron_ops_gpu.so is in pytorch/build/lib/libcaffe2_detectron_ops_gpu.so. In old caffe2, it is in /usr/local/lib/libcaffe2_detectron_ops_gpu.so.
I delete the old libcaffe2_detectron_ops_gpu.so in /usr/local/lib/libcaffe2_detectron_ops_gpu.so. everything is ok!

[agneevguin]

I am not sure if this an alternate way.

cd ~/.conda/envs/python2env/lib/python2.7/site-packages/torch/lib
sudo cp libcaffe2_detectron_ops_gpu.so /usr/local/lib/

I copied the library to the local.

[SpyderXu]

This problem was caused by different verision of caffe2, and the _CMAKE_INSTALL_PREFIX variable in env.py. Add LD_LIBRARY_PATH and edit the _CMAKE_INSTALL_PREFIX in env.py could solve this problem