"module EXLA.NIF is not available" What does this mean?

[severian1778]

Hi. I am not a genius like Jose Valim. I do not have 10 years in Docker experience. I am just a statistician and cannot spend days of my life learning how to compile things.

Can someone give a clear and concise, plain english description on what is happening here? It has destroyed my app and I can no longer run it after upgrading.

[seanmor5]

@severian1778 is it possible for you to share the Dockerfile? Are you running on CPU or GPU? What versions? The IPC stuff is a relatively recent addition

[polvalente]

Are you using Docker? Can you share the Dockerfile by any chance?

This is most likely related to a CUDA flag we have in the build process.

Also, which version of Nx and EXLA are you running?

[severian1778]

@seanmor5 I am not using a docker, I have a computer at my house with ubuntu on it. I am a simple mathematician, not an advanced computer scientist. But looking to learn.

EXLA/NX version:

GPU/CUDA version

I just need to get my app running again and I am permenantly stuck with this error which is rough!

[polvalente]

Could you rm -rf the cache and _build directories and rebuild?

[severian1778]

@polvalente As far as I understand the cache is in the umbrella root's _build directory? I did remove this as suggested and peformed mix deps.compile and it simply returned the same error again.

[polvalente]

I think the cache might be under the deps/exla directory instead. The hypothesis here is that on upgrading you kept outdated .o files that didn't recompile

[seanmor5]

So there are a few things that could be happening here. The symbol should be present in a shared object library called libcudart.so. One thing to try is to first find the directory this shared object is in (typically it's like usr/lib/cuda or similar) and then add that directory to the LD_LIBRARY_PATH environment variable.

It's also possible this is a build issue from recent changes, but I haven't looked into this enough yet to know if that's the case

Another thing that will probably work as a temporary fix is that you can downgrade to EXLA 0.8 and it should work.

Also worth nothing that you will probably want to use both Nx and EXLA from the same version to avoid any issues

[severian1778]

@polvalente @seanmor5 ok thank you that is helpful guidance. I will try to use this to fix things up and report back. Thank you very much for the guidance.

[severian1778]

@seanmor5 sadly I think the build is broken Note:

and the path is clearly added in .bashrc

Downgrading to version 0.6.0 worked. From my research it seems that the libexla.so in the priv folder is somehow asking for that ipc symbol but for some reason it can't seem to resolve it when I point it to to the shared lib via LD_LIBRARY_PATH.

I know that the file contains the symbol via nm command, but the init function in EXLA.NIF is puking because the symbol cant be resolved. Pardon if I seem basic, I am not a real computer scientist, so its all new to me.

[polvalente]

What does nvcc --version output for you?

[polvalente]

@severian1778

If nvcc shows 12.1 or later, that's fine. Otherwise, you should upgrade CUDA to that and libcudnn to version 9. I suspect you're somehow using CUDA 11, which isn't supported by newer versions anyway.

Also, look for a message like "Using libexla.so from #{cached_so}" in the mix compile logs. You should delete that file, the deps and _build directories, and then upgrade the EXLA version. This should ensure that you're starting from a clean slate.

If the problem persists, we can investigate linking/path issues.

[severian1778]

@polvalente that makes sense I will try this guidance. I suspected there was a disconnect between Cuda files and libexla.so and this seems like a clear solution. In the future would it be appropriate to have better error messaging that reflects minimum cuda release?

[polvalente]

The minimum CUDA version is already pretty clear in XLA, which is pointed to from EXLA.

Checking at compile time would be possible, but given that the supported CUDA version is given indirectly by openxla/xla via elixir-nx/xla, there isn't much we can do. However, if you have a suggestion on how to use openxla/xla's checks at EXLA's compile time, we can converge on a plan of attack and PRs are always welcome.

[severian1778]

@polvalente I just updated to the most modern cuda version and still the same problem. I guess the problem is that its hard to understand what libexla.so is looking for. It just refuses to grab the symbol from libcudart.so. I am totally at a loss. I was operating on version 12.2 as the image post defines, so I think its not the cuda version that is the problem. although I did upgrade to 12.6

[polvalente]

Have you deleted the build and cache directories as suggested? Please do that and share the resulting logs.

What I'm looking to see is which XLA version is being pulled and whether libexla.so is getting recompiled.

Also, please share the output for nvcc --version