[CI] Move tests on aarch64 linux to GitHub Actions

[giordano]

Let's see how this fares. The idea is to reduce a little bit pressure on the juliaecosystem runners.

[giordano]

Queue is quite long at the moment, we aren't going to gain much for the time being 🥲

[mofeing]

A lot of people might be trying to use it too. Let's give a couple of days and trigger another run then.

[imciner2]

It doesn't have the 64 suffix. So, the long wait time is because it is trying to run a job on a runner that doesn't exist.

[giordano]

🤦

[giordano]

https://github.com/EnzymeAD/Reactant.jl/actions/runs/12820824033/job/35751035540#step:9:778

     CondaPkg Creating environment
             │ /home/runner/.julia/artifacts/528c740035ea96b4091a42660cac663aa0a3f036/bin/micromamba
             │ -r /home/runner/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
             │ create
             │ -y
             │ -p /tmp/jl_BCzCK3/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             │ uv[version='>=0.4']
             └ -c conda-forge
info     libmamba ****************** Backtrace Start ******************
debug    libmamba Loading configuration
trace    libmamba Compute configurable 'create_base'
trace    libmamba Compute configurable 'no_env'
trace    libmamba Compute configurable 'no_rc'
trace    libmamba Compute configurable 'rc_files'
trace    libmamba Compute configurable 'root_prefix'
trace    libmamba Get RC files configuration from locations up to HomeDir
trace    libmamba Configuration not found at '/home/runner/.mambarc'
trace    libmamba Configuration not found at '/home/runner/.mamba/mambarc.d'
trace    libmamba Configuration not found at '/home/runner/.mamba/mambarc'
trace    libmamba Configuration not found at '/home/runner/.mamba/.mambarc'
critical libmamba filesystem error: status: Permission denied [/home/runneradmin/.config/mamba/mambarc.d]
info     libmamba ****************** Backtrace End ********************

wut

[mofeing]

current failing tests are due to some bug in the CUDA integration. the buildkite job was failing before this PR, but the one "CI / Julia 1.11 - integration - ubuntu-24.04-arm - aarch64 - packaged libReactant - assertions=false" could be a spurious error?

[giordano]

I don't think that's spurious, I've got similar errors locally on my laptop the other day but didn't have the time to investigate.

[mofeing]

cc @avik-pal

[giordano]

Same error happens also on main on Buildkite, looks very real to me: https://buildkite.com/julialang/reactant-dot-jl/builds/3529#01947d51-fb10-4f9c-b3ed-7e4897ed7dd8/286-1084. However it doesn't seem to systematic.

[wsmoses]

@maleadt @vchuravy @gbaraldi if you have any insights here, for some reason GPUCompiler.load_runtime of a cuda config is throwing "Not implemented" on aarch64

[giordano]

My understanding is that it's happening at line 29 of

Reactant.jl/test/integration/cuda.jl

Lines 20 to 31 in 32762fb

	@testset "Square Kernel" begin
	oA = collect(1:1:64)
	A = Reactant.to_rarray(oA)
	B = Reactant.to_rarray(100 .* oA)
	if CUDA.functional()
	@jit square!(A, B)
	@test all(Array(A) .≈ (oA .* oA .* 100))
	@test all(Array(B) .≈ (oA .* 100))
	else
	@code_hlo optimize = :before_kernel square!(A, B)
	end
	end

in the !CUDA.functional() branch, which makes sense since we don't have a GPU in this setup.

[wsmoses]

yeah but we should still be able to compile the code successfully (and this works in a similar no gpu case in x86)

[vchuravy]

No not really. Are you doing any world-age shenanigans? You might be executing in a world before CUDA.jl got loaded?

[wsmoses]

well for one thing CUDA.isfunctional() is false [as there is no GPU on the machine]. but no CUDA should've already been loaded (and as an example we've already GPUCopiler.compile'd to generate LLVM from CUDA.jl]

[vchuravy]

You may be in a scenario where GPUCompiler is loaded and thus most of the compiler functionality is there, but you are then calling something that CUDA.jl is supposed to implement, and you are executing in a world before CUDA.jl was loaded?

That's really the only way you could get a the "not implemented" error

[maleadt]

That's really the only way you could get a the "not implemented" error

There's a bug in 1.10/1.11 that can result in this happening (as observed in SciML code). JuliaLang/julia#57077 should fix it.

[giordano]

Oh, cool, I can see if I can find an aarch64 machine to reproduce this on and see if that PR fixes it. Thanks for the heads up!

[giordano]

I was able to successfully run the CUDA integration tests with JuliaLang/julia#57077 12 times in a row on a Grace Grace system, while they failed at the first try with Julia v1.11.2, so it looks like it was right that one.

I guess this PR is ready to go then, failure is unrelated (also because it was happening with buildkite too).

[wsmoses]

@giordano can you add a guard around the cuda tests if aarch64 and Julia version wouldn’t have the fix, don’t run them?

[mofeing]

lgtm

[wsmoses]

CI has some semeingly new red?

[giordano]

• Julia 1.10 - core - ubuntu-24.04-arm - aarch64 - packaged libReactant - assertions=false failed to check out the repository, will restart it 🫠
• Julia 1.11 - core - ubuntu-20.04 - x64 - packaged libReactant - assertions=false (note it's x86-64, not aarch64) is failing a test not touched here:

  fn = tanpi: Test Failed at /home/runner/work/Reactant.jl/Reactant.jl/test/basic.jl:1012
    Expression: #= /home/runner/work/Reactant.jl/Reactant.jl/test/basic.jl:1012 =# @jit(fn.(x_ra)) ≈ fn.(x)
     Evaluated: ConcreteRArray{Float32, 2}(Float32[-0.21030955 6.786836 … 25.12034 -14.385436; -0.19476584 1.6611387 … -0.19030015 14.128226; -19.919302 4.171895 … 0.16302171 2.0547147; 0.40969867 0.09946294 … 1.0452943 1.3147937]) ≈ Float32[-0.21030961 6.7868366 … 25.120296 -14.385441; -0.19476582 1.6611385 … -0.19030032 14.128206; -19.919342 4.1718936 … 0.16302171 2.0547144; 0.40969867 0.099462934 … 1.0452943 1.3147937]
  

[wsmoses]

ah hm, maybe we should lower the tolerance for the tan test or something?

[mofeing]

ah hm, maybe we should lower the tolerance for the tan test or something?

yeah, i agree