Reactant with Nvidia GPU on x86_64-linux-gnu taking forever to precompile on 0.2.153 vs 0.2.152

[yolhan83]

Hello, I tried to up Reactant and it seems like it now takes forever to precompile, didn't manage to do so ( specifing I'm on wsl so not sure it is the case on linux). Also the case for main.

Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 20 × 12th Gen Intel(R) Core(TM) i7-12700H
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 20 default, 0 interactive, 10 GC (on 20 virtual cores)

[giordano]

not sure it is the case on linux

No, linux precompilation seems to be working fine on CI, here and in other related repos.

[yolhan83]

weird it does precompile on 0.1.152 though

[giordano]

Can you tell where it gets stuck? Perhaps it's during a precompilation workload, so some of the new compiler passes are taking longer than expected? You should be able to disable that with https://julialang.github.io/PrecompileTools.jl/stable/#Package-developers:-reducing-the-cost-of-precompilation-during-development

[yolhan83]

It hangs forever, even after cloning and removing Precompile.jl in source completly so the issue may be elsewhere, also even if I run
julia --compiled-modules=no it just hangs forever when doing using Reactant

[yolhan83]

Ok so it works on PR "Fix enzyme.make_zero! (#1544) (@wsmoses)" but fail on PR "Update Project.toml (#1546) (@wsmoses)" and I just realize I get a Failure artifact: Reactant after this one and even on main, it doesn't crash but after this precompilation takes forever, so I think its something about Reactant_jll = "0.0.230" having some weird behavior with wsl. Also if I add the jll itself it hangs forever at using Reactant_jll, maybe some part of the compilation reconize my system as windows and other parts as linux

[giordano]

@yolhan83 just to confirm, you have a GPU system, right? using Reactant_jll should be sufficient to trigger the issue on a Nvidia GPU machine, but CPU-only systems should be fine.

[yolhan83]

Yes I have an Nvidia GPU 4060 and it works (on wsl) with all Reactant version both on cpu and gpu before that one. And doing the using Reactant_jll just hang forever

[giordano]

So, the problem is the switch to Clang for the x86_64-linux-gnu CUDA builds in JuliaPackaging/Yggdrasil#11718, reverting to GCC solves the issue for me (at least the CUDA 12.8 build, haven't tested others), but upstream may not support GCC in the future, so this is a complicated situation.

[yolhan83]

Oh ok, its ok, I will continue on the older version for now until Reactant is fully available on windows and then this won't be an issue ( unless you know smething else than wsl using x86_64-linux-gnu obviously). Thank you !

[wsmoses]

Independently, can you let us know if the non cuda native (non wsl) build of reactant works?

Simply because I don't have a windows machine to test that on

[yolhan83]

for now no but this may be because of a check in Reactant ? :

julia> using Reactant
┌ Warning: Reactant_jll isn't availble for your platform Platform("x86_64", "windows"; libgfortran_version = "5.0.0", julia_version = "1.11.6", cuda_version = "12.8", gpu = "cuda", mode = "opt", cxxstring_abi = "cxx11")
└ @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Reactant.jl:267

julia> Reactant.devices()
ERROR: TypeError: in dlopen, expected Union{String, Symbol, Base.Libc.Libdl.LazyLibrary}, got a value of type Missing
Stacktrace:
 [1] dlopen(x::Missing)
   @ Base.Libc.Libdl .\libdl.jl:448
 [2] #MakeCPUClient#11
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\PJRT\Client.jl:145 [inlined]
 [3] MakeCPUClient
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\PJRT\Client.jl:135 [inlined]
 [4] CPUClient(; checkcount::Bool, kwargs::@Kwargs{…})
   @ Reactant.XLA.PJRT C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\PJRT\Client.jl:125
 [5] initialize_default_clients!(state::Reactant.XLA.PJRTBackendState)
   @ Reactant.XLA C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:199
 [6] getproperty
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:55 [inlined]
 [7] default_backend
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:87 [inlined]
 [8] devices()
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Devices.jl:9
 [9] top-level scope
   @ REPL[3]:1
Some type information was truncated. Use `show(err)` to see complete types.

julia> A = Reactant.to_rarray(rand(10))
ERROR: AssertionError: Client is null
Stacktrace:
 [1] addressable_devices(client::Reactant.XLA.PJRT.Client)
   @ Reactant.XLA.PJRT C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\PJRT\Client.jl:48
 [2] default_device(client::Reactant.XLA.PJRT.Client)
   @ Reactant.XLA C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\Client.jl:16
 [3] ConcretePJRTArray(data::Vector{…}; client::Nothing, idx::Nothing, device::Nothing, sharding::Reactant.Sharding.NoSharding)
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Types.jl:193
 [4] to_rarray_internal(x::Array{<:Union{…}}, track_numbers::Type, sharding::Any, runtime::Any, device::Any, client::Any)
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Tracing.jl:1903
 [5] #to_rarray#150
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Tracing.jl:1837 [inlined]
 [6] to_rarray(x::Any)
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Tracing.jl:1827
 [7] top-level scope
   @ REPL[4]:1
Some type information was truncated. Use `show(err)` to see complete types.

julia> Reactant.set_default_backend("cpu")
ERROR: KeyError: key "cpu" not found
Stacktrace:
 [1] getindex
   @ .\dict.jl:477 [inlined]
 [2] client(backend::String)
   @ Reactant.XLA C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:85
 [3] set_default_backend
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:104 [inlined]
 [4] set_default_backend(backend::String)
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Reactant.jl:285
 [5] top-level scope
   @ REPL[5]:1

[wsmoses]

it should be available, but you need to use the latest jll / main

[wsmoses]

you also need to specify no gpu

[yolhan83]

Where should I specify that ?

(@v1.11) pkg> st
Status `C:\Users\yolha\.julia\environments\v1.11\Project.toml`
  [3c362404] Reactant v0.2.153 `https://github.com/EnzymeAD/Reactant.jl.git#main`
  [0192cb87] Reactant_jll v0.0.230+1 `https://github.com/JuliaBinaryWrappers/Reactant_jll.jl.git#main`

julia> using Reactant_jll

julia> using Reactant

julia> Reactant.set_default_backend("cpu")
ERROR: KeyError: key "cpu" not found
Stacktrace:
 [1] getindex
   @ .\dict.jl:477 [inlined]
 [2] client(backend::String)
   @ Reactant.XLA C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:85
 [3] set_default_backend
   @ C:\Users\yolha\.julia\packages\Reactant\yzj53\src\xla\XLA.jl:104 [inlined]
 [4] set_default_backend(backend::String)
   @ Reactant C:\Users\yolha\.julia\packages\Reactant\yzj53\src\Reactant.jl:285
 [5] top-level scope
   @ REPL[15]:1

PS : Maybe this check should be done at the set_default_backend("gpu") and use cpu as the default for windows ?

[wsmoses]

like this, but GPU -> none: https://enzymead.github.io/Reactant.jl/dev/introduction/configuration#Installing-Reactant-on-GPU-Servers-without-Internet