Latest commit on cudarc seems to have broken running the examples codes

[hololite]

$ cargo run --example quantized --release --features cuda -- --which 7b-open-chat-3.5 --prompt interactive
Updating crates.io index
Finished release profile [optimized] target(s) in 1.83s
Running target/release/examples/quantized --which 7b-open-chat-3.5 --prompt interactive
avx: true, neon: false, simd128: false, f16c: true
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
thread 'main' panicked at /home/hikari/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.11.1/src/driver/sys/mod.rs:38:71:
called Result::unwrap() on an Err value: DlOpen { desc: "libcuda.so: cannot open shared object file: No such file or directory" }
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

If you checkout the branch at "01794dc1 - 2024-05-05 - Laurent Mazare - Use write rather than try-write on the metal rw-locks. (#2162)", then you can run the example fine.

[LaurentMazare]

We recentrly updated the cudarc crate to a version that includes changes on how cuda is discovered, I guess that it's what is causing the issue. Maybe you can adjust your LD_LIBRARY_PATH to include the directory in which your libcuda.so is located? Fwiw this works well on my setup.

[hololite]

Ok I have added /usr/local/cuda/targets/x86_64-linux/lib/stubs to my LD_LIBRARY_PATH, and confirmed the libcuda.so exists under that dir.

However I am still getting into error, like this:

$ RUST_BACKTRACE=full cargo run --example quantized --release --features cuda -- --which 7b-open-chat-3.5 --prompt interactive
Finished release profile [optimized] target(s) in 0.10s
Running target/release/examples/quantized --which 7b-open-chat-3.5 --prompt interactive
avx: true, neon: false, simd128: false, f16c: true
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: thread panicked while processing panic. aborting.
Aborted

---

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

[LaurentMazare]

Ok, I've actually tweaked candle to be back at using dynamic linking rather than dynamic loading for cudarc in #2176 as I was able to reproduce the issue in some setups. Could you try it again with this?
Cudarc recently changed its default to dynamic loading as detailed in this issue coreylowman/cudarc#197 , candle might have to do the same at some point but that seems a bit early to do so.

[Cifko]

I just tested it, and it still fails to load cuda.dll on my windows machine.
The error is the same called 'Result::unwrap()' on an 'Err' value: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." } }

[coreylowman]

@Cifko do you know where cuda.dll is located on your machine? And also is it called cuda.dll, or is there something called nvcuda.dll on your machine?

@hololite are the cuda shared libraries located anywhere else on your machine (e.g. maybe there are some sym links that the stubs directory you shared links to)?

[hololite]

@coreylowman @LaurentMazare
Per @coreylowman's suggestions, I have checked my system, it appears that there are 2 locations in my WSL ubuntu that have the libcuda.so:

$ find /usr -name libcuda.so
/usr/local/cuda-12.4/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/lib/wsl/lib/libcuda.so

The first dir which I tried (and failed running the examples codes) is for the Cuda toolkit.
The second dir is for the Cuda drivers which is installed as part of Windows.
Both are running the same cuda version: 12.4

Using the latest commit (d9bc5ec), I am still getting the error.
I tried setting either cuda dir in my LD_LIBRARY_PATH, but both failed with the same error:

$ cargo run --example quantized --release --features cuda -- --which phi3 --prompt interactive
Finished release profile [optimized] target(s) in 0.14s
Running target/release/examples/quantized --which phi3 --prompt interactive
avx: true, neon: false, simd128: false, f16c: true
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: thread panicked while processing panic. aborting.
Aborted

[Cifko]

I had to take couple files and rename them (they are in path)
c:\Windows\System32\nvcuda.dll -> cuda.dll
c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cublas64_12.dll -> cublas.dll
c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\curand64_10.dll -> curand.dll
Now I can run the example.

[Nie-Tianyi]

I got same error when i try to run the mnist sample code in the tutorial with CUDA enabled:

called `Result::unwrap()` on an `Err` value: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "找不到指定的模块。" } }

my CUDA version is 12.4

[hololite]

I also have issue running this on ec2 instance with A10.
The error is different though, it's related to building cudarc:

$ RUST_BACKTRACE=1 cargo run --example quantized --release --features cuda -- --which 7b-open-chat-3.5 --prompt chat
Compiling cudarc v0.11.1
error: failed to run custom build command for cudarc v0.11.1
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
process didn't exit successfully: /home/ubuntu/candle/target/release/build/cudarc-12c4a369cb2cb021/build-script-build (exit status: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-env-changed=CUDA_ROOT
cargo:rerun-if-env-changed=CUDA_PATH
cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR

--- stderr
thread 'main' panicked at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.11.1/build.rs:54:14:
Unsupported cuda toolkit version: 11050. Please raise a github issue.
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: build_script_build::cuda_version_from_build_system
3: build_script_build::main
4: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

[coreylowman]

@hololite Huh looks like the header file (cuda.h) on that system has a CUDA_VERSION of 11.5 (11050) even though nvcc supports 12.4? The issue here is that cudarc only includes headers of 11.7 onwards atm ( we could generate bindings for pretty much any version though).

We could either add in 11.5/11.6 headers, and/or address this difference in what nvcc reports the supported cuda version as.

[stalek71]

I was able to build a sample with CUDA support, but it looks like there is an issue on Windows with using the CUDA libraries. The system is unable to find cuda.dll, which is not available on my machine. It performed lookup operations, but all lookups failed for this cuda.dll file.

On Windows, CUDA libraries have names following this pattern:

Has anyone been able to run Candle on Windows with CUDA support?

[Cifko]

@stalek71 yes, I explained above. And I also added it to FAQ in candle. You basically need to rename the libraries. What I did I copied to the project folder so I don't pollute my windows or program files directory. They will eventually fix it.
The cuda.dll you are looking for is the nvcuda.dll in your c:\windows\system32.

[stalek71]

Thanks! I just did it the same way :)
There is some issue yet. I ran a very simple code a few times building it for Gpu and Cpu.
When I run it for cpu I receive every time proper random values.
When I run it for Gpu it looks numbers are the same all the time.
Both tensors are allocated on proper device (or at least I expect they are allocated there)...

This is my simple code (I run it on Quadro RTX 4000: - cap. 7.5)

fn main() -> Result<(), Box<dyn std::error::Error>> {
    //let device = Device::new_cuda(0)?;
    let device = Device::Cpu;

    let a = Tensor::randn(0f32, 1., (2, 3), &device)?;
    let b = Tensor::randn(0f32, 1., (3, 4), &device)?;

    let c = a.matmul(&b)?;

    println!("{c}");
    
    Ok(())
}

This is the result for cpu:

This is the result for Gpu:

[Cifko]

@stalek71 this question shouldn't be in this ticket. But if you look at the source code, there is a fixed seed set.
https://github.com/huggingface/candle/blob/main/candle-core/src/cuda_backend/device.rs#L153
Just set some random seed by yourself.

[stalek71]

@Cifko, thanks for the quick response.
Now everything is clear.

[coreylowman]

I was able to build a sample with CUDA support, but it looks like there is an issue on Windows with using the CUDA libraries. The system is unable to find cuda.dll, which is not available on my machine. It performed lookup operations, but all lookups failed for this cuda.dll file.

On Windows, CUDA libraries have names following this pattern:

Has anyone been able to run Candle on Windows with CUDA support?

We have a tracking issue in cudarc for this (coreylowman/cudarc#219). Its not fully clear to me all the patterns that are possible on windows. The original issue reporting this mentioned nvrtc64_120_0, which is different than the names you report. It seems like we could fallbacks to <lib name><32 or 64>_<11 or 12>.dll and then maybe <lib name><32 or 64>_<110_0 or 120_0>.dll. I'll work on getting this added for the 11.2 release of cudarc

[hololite]

On Windows's WSL Ubuntu/linux, this problem still persists.
Adding the dir containing the libcuda.so to the LD_LIBRARY_PATH did not help.
I have 2 different laptops with WSL/Ubuntu having similar problems.

Also on AWS EC2 instance with A10G, the problem also exists although with different error messages.

The example codes used to work fine on all of these 3 machines prior to the recent cudarc changes.

[Helios113]

Same issue on a native Ubuntu install.
Cargo finds the oldest version installed on the system.

--- stderr
thread 'main' panicked at /nfs-share/pa511/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.11.1/build.rs:54:14:
Unsupported cuda toolkit version: 10010. Please raise a github issue.
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

this is with
RUSTFLAGS='-L /usr/local/cuda-12.1/targets/x86_64-linux/lib/stubs, and
/usr/local/cuda-12.1/targets/x86_64-linux/lib/stubs added to LD_LIBRARY_PATH

I have tried both independently as well