A suggested step to cut CI time on the Ryzen-AI (npu-xrt) tests: split compile from device execution

[atassis]

I spent some time profiling why the buildAndTestRyzenAI jobs are the long pole, and I think the framing
matters, so I wanted to put the numbers somewhere trackable rather than in a drive-by PR. Opening this per
the tracking discussion that was suggested; happy to fold it into #3198.

The finding. The capacity-1 npu-xrt lit group serializes each device test end to end -- host
aiecc/xclbin generation and the device invocation, not just the device milliseconds. On this hardware
the generation dominates: for the simple Chess-backed tests the compile is ~95-100% of the test's
wall-clock (e.g. a 26 s test whose compile-only twin is 26 s), and across the serialized lane the honest
planning interval is ~85-95% compile. So the lane is compile-bound, and compilation touches no device.

The mechanism, framed as de-loading. I know coordination has been tried and deliberately backed out
here (the device lock experimented under #2914 and removed in #2992 with "the problem is not one of
concurrency", and the merge-queue concurrency reverted for job ejections). I am not proposing to add
coordination. The opposite: an opt-in, per-test compile/execute split that moves the compile half OUT
of the capacity-1 group so it runs in parallel with everything else, and leaves ONLY the device execution
in the serialized lane that #2737 added to protect the shared XRT hw_context pool. Strictly less on the
serial lane, no new lock, no cross-job handoff.

Concretely it is a marker on each RUN line -- a build-phase prefix and a run-phase prefix that expand to
nothing in a normal run and to a shell no-op in the phase that should skip. A test opts in by carrying the
markers; everything else is unchanged and stays serialized. Redirect/pipe/FileCheck/JIT tests that have no
clean compile/execute boundary are simply not converted.

Evidence (on a Krackan AIE2P box, single-tenant). I built a matched-commit toolchain and ran the
converted npu-xrt tests three ways -- whole test, compile phase, execute phase -- and compared verdicts:

• Behaviour-identical: across the full runnable Peano subset (39 tests) the split gives the same
  verdict as the whole test, 0 mismatches on a clean run. The one transient failure I saw was an
  equal-value mismatch (5 != 5), i.e. the known host/device read race the workflow already retries on,
  not a split artifact (it passes in isolation in both modes).
• Faster: whole ~54 s serial vs ~9.7 s parallel compile + ~3.8 s serial execute on that subset. That
  is a lower bound on the win -- these are fast Peano compiles (~1-3 s); the real check-aie sink is the
  Chess compiles at 26-77 s each, which the split relocates identically, so the compile-bound jobs benefit
  far more.

Two smaller, orthogonal levers I also measured, for completeness: the on-device workflow has no
paths: filter (the mlirDistro one does), and PR-synchronize churn cancels ~8.8% of device-job time.
Path filters are a safe, zero-coverage-loss config change; dropping pull_request to merge_group-only is a
real spend/pre-merge-signal trade I would leave to your judgment, since the suite does catch real
pre-merge failures.

One further idea, raw and unmeasured -- floating it for the tracking record. If the split lands, the same
compile-vs-execute seam generalizes across OS. The device payload (.xclbin + npu-insts) is a silicon
binary, so in principle it could be built ONCE on a Linux host (reusing that one build cache) and executed on
both the Linux and Windows NPU runners, with the Windows host harness cross-compiled on Linux (clang
targeting Windows -- cross-compilation, not emulation). The Windows leg would then skip recompilation and
share the Linux cache. The load-bearing question is one you would know far better than I do: does the
.xclbin carry any host-OS-specific bits, or is the AIE2P payload OS-independent? If it is independent,
"compile once on Linux, execute on each OS's runner" removes a whole duplicate compile. Not proposing it --
just recording it in case it is useful later.

The split is already up as a small, individually-reviewable draft RFC (#3403) -- plumbing plus a handful of
converted tests -- if it helps to look at something concrete. Separately, #3402 narrows the on-device retry
so a transient failure re-runs only the failed tests instead of the whole job (the mechanism behind the
occasional ~1-hour outliers). Mostly, though, I wanted the compile-bound attribution on the record so the
optimization aims at the right half.

[atassis]

@hunhoffe I suppose you might want to have a look at this, since CI looks being painful a little

[hunhoffe]

@atassis One piece for why the compilation was originally included, is that some chess builds can be very memory intensive too, so we were oomkilling the CI runners if we didn't limit concurrency specifically for chess quite a bit. We might need to be careful to keep the chess tests serial.

@thomthehound Tagging you to make sure you see the windows notes!

[atassis]

Thanks, and this is the part I most wanted to say out loud, since it's the thread under everything I've sent here.

All of it started from one question: what does it cost me to test an idea, a good one or a dumb one? My first PRs were recompile-time cuts for exactly that, making it cheap to try a hypothesis and throw it away. CI is the same cost one level up, except everyone pays it: the longer a build or device step takes, the fewer things anyone bothers to try. So the compile/execute split, the narrowed retry, the chess attribution, the cross-OS artifact idea are one lever, not four. Cut the time it takes to test a theory. The graph framing is just the mechanical version of that: wherever two paths would rebuild the same thing, share the edge.

On the native-host side you're right, and I'll take your read over mine. I've built cross-arch binaries in CI before, but that was a gentler world than MSVC-vs-GCC strictness, and I left Windows years ago so I genuinely can't judge how much of the harness survives a cross-build. The half I'm confident in is the artifact: if the .xclbin and npu-insts are OS-independent, they're built once and both runners just execute them, no second compile. The host 1:1 is yours to weigh, and it sounds like the human cost is the real variable there, not the machine time. (One honest-ignorance question, since you'd actually know: is any of the Windows execution leg reachable under something like Wine, or does the driver/XRT stack make that a dead end? Not proposing it, just genuinely unsure.)

On xchesscc, your read matches what I found from outside: the compiler swap is the easy half, the aiesim/ISS dependency is the real Chess anchor. I'll scope that in the separate discussion hunhoffe asked for rather than pull this thread off course.

[atassis]

I have just posted the discussion, too, #3414

[thomthehound]

Thanks for the good work! CI is (jokingly) my arch-nemesis, so I appreciate anything that gives it a good comeuppance.

As for Wine... you know, I've never tested that. For compile-only, I don't think it should be a problem. It's just ordinary Windows code, after all. But actually testing that the resulting programs work is another matter. I think the XRT runtime is going to need the Windows driver stack, and I strongly doubt Wine can fully provide or translate that without a lot of potentially impossible tweaking. Still, this would work as a compile-check.

Separately, I haven't even looked at the WSL/Windows boundary in months, so there may be some cross-domain path and process nonsense that has accumulated without being addressed. So "HIC SUNT DRACONES"

Unless it causes a problem, I think it would be best to defer the Windows considerations for now (while perhaps leaving space for them in the future) and continue leaning into the main strengths of the ideas you've proposed here. It would be very easy to burn yourself out on this part while the Windows CI is still in its embryonic stage.

[atassis]

I have tried to work on WSL (both 1,2). And I was amazed at time in a good way what Windows has created, but that wasn't enough, still, to make me stick back to it. I say that, yes, you might wanna keep things working there, but not to run things there for windows.

Per Windows considerations- I definitely get your point. All things said, I still have a dual-boot for windows reasons (not even gaming, but for some maybes). So, if I even come to some ideas windows-wise - they'd be only after stronger claims and ideas become true.

[jgmelber]

Windows CI is still in its embryonic stage

Hopefully moving this forward this week. We have more momentum and Windows users now internally. Thanks for pioneering this @thomthehound

[atassis]

@hunhoffe thanks, that memory point is the key constraint and I don't want to fight it. Agreed the Chess compiles have to stay bounded.

I think the split is compatible with keeping them bounded, because it separates two limits that are currently fused. The capacity-1 npu-xrt group protects the shared XRT hw_context pool, which only the device execution touches. Chess compile memory is a different resource. So the split lets each have its own limit: device execution stays capacity-1 as today, and the compile phase gets its own concurrency cap sized to RAM (its own lit parallelism group, or a dedicated -j for the compile pass). Chess compiles can stay as concurrency-limited as they are now for OOM safety; they just stop waiting behind the device lock while they do it. Nothing forces the compile phase to run unbounded.

Where that turns concrete: I can't run Chess locally, so I built a temporary, self-guarding validation job to gather the real numbers on a Chess runner. It runs whole vs split and asserts they match, plus two deliberate-failure controls (execute-only on a wiped root must fail every test; a mutated kernel must fail in both modes), encoded as CI assertions so a silent regression fails the job rather than passing quietly. Running it on a Chess-capable runner would give the split==whole correctness on Chess, the timing, and the compile memory footprint, so the right compile concurrency cap is a measured number rather than a guess.

Would you be open to running that, or would you prefer I wire it a particular way? Draft RFC is #3403; happy to add a memory-sized compile group to it so the Chess-serial property is explicit.

[hunhoffe]

@atassis I've been thinking through #3404, and I'm still a little bit conflicted. The two things I'm concerned about are:

1. Complexity, generally -- debugging the CI is already hard, does this make it more difficult to explain when/where compilation errors etc are occurring?
2. Pairing between compilation artifacts and what is run in tests: stale artifacts and timing issues with build trees are both common problems we've had with the CI; I'm worried about this introducing more of these types of problems.

Part of me thinks we should just add CI machines with more granular jobs to make it go faster without adding more infrastructure complexity.

[atassis]

Generally speaking, I haven't deeply analyzed the whole CI workflow as a whole, I was just trying to fix only some parts of it to reduce local complecity. CI itselves is difficult to debug, because there are many places which serve similar purpose in it, that's why anyone can get lost in it.
I can suggest 2 things:

1. Reduce local complexity or duration in specifix workflows, which would benefit in making faster code iterations (commit checks, PR checks, merge queue checks and else)
2. Invest into making CI both linear and task specific (make each part of it serve exact purpose, unix-way). For example, make linters first, unit-tests second, build step third, e2e tests (NPU tests) after that and so on. Right now many workflows mix the behavior, and also have a N*M matrix, which contributes to number of simultaneous actions. Some of them might be concatenated, I think, but this is just a speculation I did here, not a researched opinion.

Per CI machines- you might want to do that, if you can request more resources or make payable github org, if you don't have one. But it won't help long term, maybe an opposite. You might wanna add both paid github org and more self hosted runners. Well, self hosted NPU runners are needed more than default ones. Still, an issue with device being stuck should be resolved (maybe by making a service on a server, which will ping the NPU via driver to check if it is wedged, if health probe fails N times- reboot. I am rebooting 1-2 times a day my own laptop because of this issue).

[atassis]

1. I think the split makes this easier rather than harder. Today a compile error and a device error land on the same lit line, so the verdict doesn't tell you which half broke. Split, a compile failure appears in the compile pass, with the test named and no device involved, and a device failure appears in the execute pass. The failure surface gets narrower, not wider.
2. That was the risk I was most worried about, too. That's why I built the negative controls before I measured any timing:

• Fresh-root canary: execute-only from a wiped root is 0/39 PASS. Nothing can ride a stale artifact because there is nothing to ride.
• No-masking fault injection: mutate a kernel so its output is wrong. Whole and split both FAIL identically, and split still FAILs when a previous good split has left good artifacts in the exec root, so the compile phase overwrites them rather than the execute phase reusing them.

One case those do not cover, which I would rather name than have you find: if the compile phase fails, it leaves the previous artifact untouched, and the execute pass would then run against it. That is why the execute pass has to be gated on the compile pass passing, a requirement rather than a nicety.

This is also why I started looking through the caching system. Deleting a cache and rebuilding locally is annoying but tolerable; 1h+ of waiting for a PR to merge sounds like hell, when it could be 5-10 minutes. I have to spend time on my actual job, so I cannot get into the CI deeply on my own, but maybe I can finish this in parallel.

[hunhoffe]

I'm game to try the split, but we'll probably have to review changes here a little bit extra incrementally :)