[BUG] Embedded Python interpreter: RuntimeError: Function "cuInit" not found #12862
Assignees
Labels
bug
Something isn't working
Comments
dagardner-nv
added
Needs Triage
|
Hi @dagardner-nv -- could you confirm if you see the same behaviour with a more recent version of cuDF (e.g., 23.02 or 23.04?)). There was an important bug fixed in CUDA Python 11.7.1 around initialization, and I see that this environment is still using 11.7.0. At the very least, you should not see the segfault on more recent versions of CUDA Python. |
|
Can you check the transitive links of libcudf and also cuda-python? It would also be interesting to get a backtrace from cuda-python's attempts to resolve Something like: (Needs your gdb to be built with python support, which most of them are these days). Would be interesting to try and understand why that resolution fails. |
|
@wence- if I understand the issue correctly, the problem is that its loading the stub library and not the real lib when the conda env contains gdb output: If instead I renamed |
|
@shwina I tried to create a simple repro of this issue outside of Morpheus, to then make it easy to test against newer versions. However I was unable to do so. Feel free to close this, since I have a work-around. |
|
Thanks for the backtraces. That does look odd. The conda-provided nvcc is just a shell script wrapper: So my cloudy crystal ball suggests something is wrong with the |
|
@wence- You're right it is a shell script. I double-checked my |
|
I found a better work-around, presumably based on the info from @wence- something in the stack (?cuda-python?) was using the location of For now my work-around is to do: Prior to building |
dagardner-nv
added a commit
to dagardner-nv/Morpheus
that referenced
this issue
Mar 21, 2023
|
Thanks. My only guess would be the config that sets things up for the pybind build, but not sure. |
|
We can close this issue. Its unrelated to cudf and is more related to this issue: NVIDIA/cuda-python#17 (comment). Basically, To fix this, there are a few options:
|
Thanks, I agree, though I think that only works on windows and WSL (the logic that leads to that code-path is dependent on libdxcore.so existing which will only be the case on WSL), so the fallback hard-cordes I've opened a bug with the cuda-python package so that it is updated to explicitly dlopen |
|
cf. #12708 |
|
@wence- Can you share the issue you opened with |
|
The relevant issue was filed on the internal cuda-python repo where development occurs, so it cannot be linked here. Closing as resolved as discussed above. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I ran into a weird variation on the old
RuntimeError: Function "cuInit" not foundissue inside of a conda env with cudf22.10.00.From inside of the env I can do:
python -c "import cudf"just fine.However from C++ if I do:
pybind11::module_::import("cudf")I get :
I double checked my cmake logs and its picking up the correct version of python. In chatting with a team-mate who didn't have the problem, the only difference between my setup and their's is that my
nvccwas coming from the conda env, and his was coming from the OS.I renamed
nvcctonvcc.oldallowing it to pickup the OS's copy, did a rebuild and now the problem went away.Environment details
This is from within a Morpheus conda dev env:
The text was updated successfully, but these errors were encountered: