OSError: warp.so: undefined symbol

[aespielberg]

Hi all,

On Ubuntu 20.04, Anaconda python 3.9.7, cudatoolkit 11.3. I followed the installation in the guide and I am getting the following error. Why is this happening?

$ python examples/example_raycast.py 
Traceback (most recent call last):
  File "/home/user/scratch_space/warp/warp/examples/example_raycast.py", line 23, in <module>
    wp.init()
  File "/home/user/scratch_space/warp/warp/warp/context.py", line 1097, in init
    runtime = Runtime()
  File "/home/user/scratch_space/warp/warp/warp/context.py", line 470, in __init__
    self.core = warp.build.load_dll(warp_lib)
  File "/home/user/scratch_space/warp/warp/warp/build.py", line 285, in load_dll
    dll = ctypes.CDLL(dll_path)
  File "/home/user/anaconda3/envs/warp/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/user/scratch_space/warp/warp/warp/bin/warp.so: undefined symbol: _ZN2wp24hash_grid_rebuild_deviceERKNS_8HashGridEPKNS_4vec3Ei

[mmacklin]

Hi @aespielberg , did you build the library yourself or use the release package? If you built it yourself could you confirm if it was built with CUDA support enabled?

[etaoxing]

I think I ran into similar issues on Ubuntu 21.04, CUDA 11.4, both when building warp (with export CUDA_PATH=/usr/local/cuda set properly) and using the release package.

It seems Omniverse officially supports Ubuntu 18.04. A workaround to this is to run warp in Docker, I used the Isaac Sim container with some modifications:

FROM nvcr.io/nvidia/isaac-sim:2021.2.1

RUN apt-get update \
  && apt-get install -y -qq --no-install-recommends \
    wget \
    git \
    vim \
    sudo \
    build-essential \
    software-properties-common \
  && apt-get clean && rm -rf /var/lib/apt/lists/*

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin \
  && mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 \
  && apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub \
  && add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
RUN apt-get update \
  && apt-get -y install cuda-toolkit-11-4 \
  && apt-get clean && rm -rf /var/lib/apt/lists/*
ENV PATH /usr/local/cuda/bin:$PATH
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:$LD_LIBRARY_PATH

RUN mkdir /root/.local/share/ov/pkg/ && ln -s /isaac-sim /root/.local/share/ov/pkg/isaac_sim-2021.2.1
RUN ln -sf /isaac-sim/kit/python/bin/python3 /usr/bin/python
ENV PATH /isaac-sim/kit/python/bin:$PATH

[aespielberg]

Thank you both for the replies. Yes, I foolishly forgot to set the CUDA_PATH. I have now set it to /home/user/anaconda3/envs/warp/, and cuda objects were built. I also set LD_LIBRARY_PATH to $CONDA_PREFIX/lib.
however, now on runtime it is a new error:

$ python examples/example_raycast.py 
Traceback (most recent call last):
  File "/home/user/scratch_space/warp/warp/examples/example_raycast.py", line 23, in <module>
    wp.init()
  File "/home/user/scratch_space/warp/warp/warp/context.py", line 1097, in init
    runtime = Runtime()
  File "/home/user/scratch_space/warp/warp/warp/context.py", line 536, in __init__
    raise Exception("Warp Initialization failed, CUDA not found")
Exception: Warp Initialization failed, CUDA not found

Not sure how to debug this error. 🤔

[mmacklin]

This usually means that the CUDA driver is not installed correctly on your system. If you run nvidia-smi from the command line do you see visible CUDA devices?

[aespielberg]

Wow, crazy, those drivers were there just two days ago.

That was indeed the problem. Thank you!