ImportError: gpu_nms.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

[Sun-Ng]

when i run "python tools/test.py --cfg experiments/coco/hrnet/w32_256x192_adam_lr1e-3.yaml TEST.MODEL_FILE models/pytorch/pose_coco/pose_hrnet_w32_256x192.pth TEST.USE_GT_BBOX False", i get such an error:

Traceback (most recent call last):
  File "tools/test.py", line 31, in <module>
    import dataset
  File "/data1/home/github/deep-high-resolution-net.pytorch/tools/../lib/dataset/__init__.py", line 12, in <module>
    from .coco import COCODataset as coco
  File "/data1/home/github/deep-high-resolution-net.pytorch/tools/../lib/dataset/coco.py", line 22, in <module>
    from nms.nms import oks_nms
  File "/data1/home/github/deep-high-resolution-net.pytorch/tools/../lib/nms/nms.py", line 14, in <module>
    from .gpu_nms import gpu_nms
ImportError: /data1/home/github/deep-high-resolution-net.pytorch/tools/../lib/nms/gpu_nms.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

This is my environment package configuration：

Package                    Version    
---------------         -----------
ca-certificates           2019.3.9
certifi                   2019.3.9 
cffi                      1.12.2
cloudpickle               0.8.1
cudatoolkit               9.0 
cycler                    0.10.0 
cython                    0.29.6 
dask                      1.1.4 
decorator                 4.4.0
easydict                  1.7
intel-openmp              2019.1
json-tricks               3.13.1
kiwisolver                1.0.1
libblas                   3.8.0
libcblas                  3.8.0
libffi                    3.2.1
libgcc-ng                 7.3.0
libgfortran-ng            7.2.0
liblapack                 3.8.0 
libstdcxx-ng              7.3.0
matplotlib                3.0.3 
mkl                       2019.1
ncurses                   6.1
networkx                  2.2 
ninja                     1.9.0
numpy                     1.16.2
openblas                  0.3.5
opencv-python             3.4.1.15
openssl                   1.1.1b
pandas                    0.24.2
pillow                    5.4.1
pip                       19.0.3
protobuf                  3.7.1
pycocotools               2.0 
pycparser                 2.19
pyparsing                 2.3.1
python                    3.6.7
python-dateutil           2.8.0
pytorch-nightly           1.0.0.dev20190326
pytz                      2018.9
pywavelets                1.0.2
pyyaml                    5.1
readline                  7.0
scikit-image              0.14.2pi
scipy                     1.2.1
setuptools                40.8.0
shapely                   1.6.4
six                       1.12.0 
sqlite                    3.26.0
tensorboardx              1.6
tk                        8.6.9
toolz                     0.9.0
torchvision               0.2.2.post3
wheel                     0.33.1
xz                        5.2.4
yacs                      0.1.6 
zlib                      1.2.11

Does anybody have the same problem?

[Sun-Ng]

The cuda is 9.2 in my base environment, but i install the cudatoolkit=9.0 in conda env, when i reinstall cudatoolkit to 9.2 in conda env, it work.