undefined symbol: cudnnBackendPopulateCudaGraph, version libcudnn_graph.so.9

[universalwow]

Hello everyone. When I was running the elixir nx demo in orin nano, I encountered this error and don't know how to solve it。

10:58:06.377 [error] Process #PID<0.220.0> raised an exception
** (RuntimeError) Failed to load NIF library.
Follow the steps in the :exla README Troubleshooting section for more information.

:load_failed
Failed to load NIF library /home/uni/cuda_demo/_build/dev/lib/exla/priv/libexla: '/home/uni/cuda_demo/_build/dev/lib/exla/priv/xla_extension/lib/libxla_extension.so: undefined symbol: cudnnBackendPopulateCudaGraph, version libcudnn_graph.so.9'

(exla 0.10.0) lib/exla/nif.ex:13: EXLA.NIF.__on_load__/0
(kernel 10.5) code_server.erl:1391: anonymous fn/1 in :code_server.schedule_on_load/4

10:58:06.382 [notice] Application exla exited: EXLA.Application.start(:normal, []) returned an error: shutdown: failed to start child: EXLA.Logger
** (EXIT) an exception was raised:
** (UndefinedFunctionError) function EXLA.NIF.start_log_sink/1 is undefined (module EXLA.NIF is not available)
(exla 0.10.0) EXLA.NIF.start_log_sink(#PID<0.219.0>)
(exla 0.10.0) lib/exla/logger.ex:12: EXLA.Logger.init/1
(stdlib 7.2) gen_server.erl:2276: :gen_server.init_it/2
(stdlib 7.2) gen_server.erl:2236: :gen_server.init_it/6
(stdlib 7.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

10:58:06.421 [notice] Application nimble_pool exited: :stopped

10:58:06.381 [warning] The on_load function for module Elixir.EXLA.NIF returned:
{%RuntimeError{
message: "Failed to load NIF library.\nFollow the steps in the :exla README Troubleshooting section for more information.\n\n:load_failed\nFailed to load NIF library /home/uni/cuda_demo/_build/dev/lib/exla/priv/libexla: '/home/uni/cuda_demo/_build/dev/lib/exla/priv/xla_extension/lib/libxla_extension.so: undefined symbol: cudnnBackendPopulateCudaGraph, version libcudnn_graph.so.9'\n"
},
[{EXLA.NIF, :on_load, 0, [file: ~c"lib/exla/nif.ex", line: 13, ...]}, ...]}

[polvalente]

Do you have cudnn installed? Which version?

Would also be helpful to know xla, elixir, erlang and cuda versions

[universalwow]

Do you have cudnn installed? Which version?

Would also be helpful to know xla, elixir, erlang and cuda versions
The new orin nano officially comes with ubuntu 22.04。
elixir: 1.19.4 erlang:28.3 xla 0.9.1 cuda 12.6 cudnn 9.3

should i install cuda 13？

[polvalente]

Could you also try recompiling with XLA_TARGET=cuda12 set? Just to eliminate the chance you're using a CPU-only .so

[universalwow]

Could you also try recompiling with XLA_TARGET=cuda12 set? Just to eliminate the chance you're using a CPU-only .so

i set export XLA_TARGET=cuda12 to the .bashrc file. it seems like a cudnn version problem

[polvalente]

Yes, it could be, as cuda 12.6 is also not the latest cuda.

[universalwow]

Yes, it could be, as cuda 12.6 is also not the latest cuda.

Yes, it could be, as cuda 12.6 is also not the latest cuda.

It seems that orin nano is not allowed to upgrade cuda directly, but needs to be upgraded through jetpack. I attempted to install cuda 13.1 and its cudnn 9.17, and then attempted to upgrade nvidia-driver but failed. Currently, the official built-in driver version is 540.4, which has led to a new error again。

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1767075693.254837 15625 se_gpu_pjrt_client.cc:1101] Using BFC allocator.
I0000 00:00:1767075693.255027 15625 gpu_helpers.cc:136] XLA backend allocating 4793983795 bytes on device 0 for BFCAllocator.
I0000 00:00:1767075693.255154 15625 gpu_helpers.cc:177] XLA backend will use up to 3195989196 bytes on device 0 for CollectiveBFCAllocator.
E0000 00:00:1767075693.260476 15625 cuda_dnn.cc:534] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E0000 00:00:1767075693.260639 15625 cuda_dnn.cc:538] Memory usage: 4768473088 bytes free, 7989972992 bytes total.
E0000 00:00:1767075693.260822 15625 cuda_dnn.cc:548] Possibly insufficient driver version: 540.4.0
E0000 00:00:1767075693.261582 15625 cuda_dnn.cc:534] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E0000 00:00:1767075693.261628 15625 cuda_dnn.cc:538] Memory usage: 4768473088 bytes free, 7989972992 bytes total.
E0000 00:00:1767075693.261708 15625 cuda_dnn.cc:548] Possibly insufficient driver version: 540.4.0

14:21:33.262 [info] XLA service 0xffff613035d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:

14:21:33.272 [info] StreamExecutor device (0): Orin, Compute Capability 8.7
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
I0000 00:00:1767075696.810658 15639 cuda_executor.cc:508] failed to allocate 4.46GiB (4793983744 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
E0000 00:00:1767075698.983521 15629 cuda_dnn.cc:534] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E0000 00:00:1767075698.983797 15629 cuda_dnn.cc:538] Memory usage: 1349734400 bytes free, 7989972992 bytes total.
E0000 00:00:1767075698.984083 15629 cuda_dnn.cc:548] Possibly insufficient driver version: 540.4.0
E0000 00:00:1767075699.003292 15629 cuda_dnn.cc:534] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E0000 00:00:1767075699.003491 15629 cuda_dnn.cc:538] Memory usage: 1349582848 bytes free, 7989972992 bytes total.
E0000 00:00:1767075699.003735 15629 cuda_dnn.cc:548] Possibly insufficient driver version: 540.4.0
** (RuntimeError) DNN library initialization failed. Look at the errors above for more details.
(exla 0.10.0) EXLA.NIF.mlir_compile(#Reference<0.3372175841.2832859140.232705>, #Reference<0.3372175841.2832859140.232774>, [%EXLA.Typespec{type: {:u, 8}, shape: {47040000}}], 1, 1, true, 0)
(exla 0.10.0) lib/exla/mlir/module.ex:114: EXLA.MLIR.Module.compile/5
(stdlib 7.2) timer.erl:599: :timer.tc/2
(exla 0.10.0) lib/exla/defn.ex:239: anonymous fn/13 in EXLA.Defn.compile/7
(exla 0.10.0) lib/exla/mlir/context_pool.ex:10: anonymous fn/3 in EXLA.MLIR.ContextPool.checkout/1
(nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
(exla 0.10.0) lib/exla/defn/locked_cache.ex:36: EXLA.Defn.LockedCache.run/2
iex:2: (file)

[polvalente]

Try CUDA 12.8

[universalwow]

Try CUDA 12.8

The gap between elixir exla and orin nano is truly suffocating. I installed cuda12.9 and its compatibility with cudnn, but now a new error has been reported. Is there a way for me to upgrade the driver? The current version 540.4 seems a bit low。

final_params = MNIST.train(fun, train_images, train_labels, params, epochs: 10)
** (RuntimeError) CUDA Runtime error: Failed call to cudaGetFuncBySymbol: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device
(exla 0.10.0) EXLA.NIF.mlir_compile(#Reference<0.2437853237.80347137.210615>, #Reference<0.2437853237.80347137.235104>, [%EXLA.Typespec{type: {:f, 32}, shape: {784, 128}}, %EXLA.Typespec{type: {:f, 32}, shape: {128}}, %EXLA.Typespec{type: {:f, 32}, shape: {128, 10}}, %EXLA.Typespec{type: {:f, 32}, shape: {10}}, %EXLA.Typespec{type: {:f, 32}, shape: {30, 784}}, %EXLA.Typespec{type: {:u, 8}, shape: {30, 10}}, %EXLA.Typespec{type: {:f, 32}, shape: {}}, %EXLA.Typespec{type: {:f, 32}, shape: {}}, %EXLA.Typespec{type: {:s, 32}, shape: {}}], 1, 1, true, 0)
(exla 0.10.0) lib/exla/mlir/module.ex:114: EXLA.MLIR.Module.compile/5
(stdlib 7.2) timer.erl:599: :timer.tc/2
(exla 0.10.0) lib/exla/defn.ex:239: anonymous fn/13 in EXLA.Defn.compile/7
(exla 0.10.0) lib/exla/mlir/context_pool.ex:10: anonymous fn/3 in EXLA.MLIR.ContextPool.checkout/1
(nimble_pool 1.1.0) lib/nimble_pool.ex:462: NimblePool.checkout!/4
(exla 0.10.0) lib/exla/defn/locked_cache.ex:36: EXLA.Defn.LockedCache.run/2
iex:12: (file)

[polvalente]

I suggest searching for the cuda-specific error messages such as the one you got as the Jetson ecosystem in general is very finicky with CUDA installs.

There's likely a minimum driver version you need to support CUDA 12.9

[universalwow]

I suggest searching for the cuda-specific error messages such as the one you got as the Jetson ecosystem in general is very finicky with CUDA installs.

There's likely a minimum driver version you need to support CUDA 12.9

I already know the supported versions, but I don't know how to update them. The official said that dgpu and jetson are separate and can only be updated through jetpack. But I'm already using the latest version of jetpack