CUDA.jl v4 installs CUDA runtime despite version=local

[oschulz]

This seems to happen even with Julia v1.9, but only on certain systems. For example on ORNL Ascent, with Julia v1.9.0, CUDA.jl insists on installing the CUDA runtime despite an "environments/v1.9/LocalPreferences.toml"

[CUDA_Runtime_jll]
version = "local"

if "CUDA_Runtime_jll" is not part of the environment. So with

module load julia/1.9.0
module load cuda/11.4.2

This works as it should (uses local CUDA)

echo -e '[CUDA_Runtime_jll]\nversion = "local"' > "/path/to/julia-depot/environments/v1.9/LocalPreferences.toml"
julia -E 'import Pkg; Pkg.add(["CUDA_Runtime_jll", "CUDA"])'

But with (no explicit "CUDA_Runtime_jll") instead

echo -e '[CUDA_Runtime_jll]\nversion = "local"' > "/path/to/julia-depot/environments/v1.9/LocalPreferences.toml"
julia -E 'import Pkg; Pkg.add(["CUDA"])'

CUDA.jl immediately installs it's own runtime (tested with a fresh Julia depot multiple times).

CC @williamfgc , I ran into this during the ORNL Julia HPC training session today.

[maleadt]

I think this is a limitation of Preferences; the package should be listed as a dependency somehow. Preferences can be layered though, so it should be possible to provide one in an upper depot. It should also be possible to set the dep in the [extras] section, i.e., it doesn't need to be an actual dependency.

[oschulz]

Currently we only warn about this on VERSION <= v"1.6.5" || VERSION == v"1.7.0"

CUDA.jl/lib/cudadrv/version.jl

Line 59 in d3b1363

@warn """Due to a bug in Julia (until 1.6.5 and 1.7.1) your environment needs to directly include CUDA_Runtime_jll for this to work."""

Should we encourage users to add "CUDA_Runtime_jll" to their environment in general? People can use CUDA.set_runtime_version! of course, but at this point they've already downloaded a few GB of CUDA runtime. :-)

[vchuravy]

There are two different kinds of "add to environment".

If you use the Preference API it will automatically add it to [extras] which is distinct from the notion of ]add e.g. dependencies.

I tried to give a bit off guidance on this here https://juliaparallel.org/tutorials/preferences/

[maleadt]

Should we encourage users to add "CUDA_Runtime_jll" to their environment in general?

The call to Preferences above should automatically add it to the [extras] section of the active environment, so I don't think this would be needed.

It's true that for some users, artifacts will have been downloaded by then. Truly lazy artifacts would be a nice to have, but would need to be added to JLLWrappers (AFAIU the current architecture doesn't allow that). Hopefully you can avoid that on clusters by providing a global preference.

---

Closing this because I don't think this a CUDA.jl bug / something we should change here, but feel free to keep the discussion alive if anything is unclear.

[oschulz]

Closing this because I don't think this a CUDA.jl bug / something we should change here, but feel free to keep the discussion alive if anything is unclear.

Oh, sure - thanks for the help!

@vchuravy regarding Preferences.jl - it would be nice being able to provide setup scripts to users for specific systems that need a local CUDA (without downloading a whole CUDA runtime first). Is there a way to make Pkg add a package ("CUDA_Runtime_jll" in this case) under [extras], so one can then do Preferences.set_preferences!(uuid, "version" => "local") or so? I didn't find a way of using Preferences.set_preferences! without having "CUDA_Runtime_jll" installed first, at least.

It's kind of a chicken and egg problem - Preferences.set_preferences! would add CUDA_Runtime_jll to [extras] - but it doesn't to anything if the package isn't already part of the environment in some form:

julia> Preferences.set_preferences!(Base.UUID("76a88914-d11a-5bdc-97e0-2f5a05c973a2"), "version" => "local")
ERROR: Cannot set preferences of an unknown package that is not loaded!

[vchuravy]

It's kind of a chicken and egg problem - Preferences.set_preferences! would add CUDA_Runtime_jll to [extras] - but it doesn't to anything if the package isn't already part of the environment in some form:

File an issue with Preferences.jl? Should be easy enough https://github.com/JuliaPackaging/Preferences.jl/blob/e8082bc48a40dc27ed427e5ac9d17398d9949a14/src/Preferences.jl#L256-L264 (and it's not a stdlib so we can update it easily)

[oschulz]

File an issue with Preferences.jl?

Done. :-)