main is not verified by its own CI: every run is cancelled, and 5 tests fail under ASan/UBSan

[localai-bot]

Two findings, and the second is why nobody has noticed the first.

1. Five tests fail under ASan/UBSan

From the sanitize-cpu (address,undefined) job on
PR #259
(run),
99% tests passed, 5 tests failed out of 366:

  9 - test_load_direct_upload   (Failed)
 43 - test_llama_embedding_fold (Failed)
 58 - test_laguna_nvfp4_loader  (Failed)
233 - test_openai_api_server    (Failed)
237 - test_capi                 (Failed)

Not starvation: the run completed in 390 s with the other 361 passing in ~0.27 s
each, and it has now reproduced on two consecutive runs of that PR.

Not caused by that PR. Its entire diff is records/docs plus two test files
plus six comment-only lines in src/vllm/v1/engine/logprobs.cpp — the object
file is byte-identical to main's (md5 verified while the PR was being prepared),
so libvllm.a is bit-identical. A comment change cannot break a loader test, an
embedding fold, an NVFP4 loader or the C API. These are pre-existing on main.

test_capi is independently corroborated: a heap-corruption failure there was
already observed during the Jetson Thor sm_110 bring-up.

2. main's own CI never completes, so this was never going to be caught

Every recent main run is cancelled:

31395680680 ci completed/cancelled
31394968996 ci completed/cancelled
31394705099 ci completed/cancelled
31394011582 ci completed/cancelled
31393661001 ci completed/cancelled
31393588909 ci completed/cancelled
31393519175 ci completed/cancelled
31389045693 ci completed/cancelled

Merges land faster than a CI run takes, and each push cancels the in-flight run
for its concurrency group. The sanitizer job is ~27 minutes, so it is
structurally the least likely job to ever finish on main. The practical effect
is that main is not verified by its own CI — regressions are only visible
on PR branches, where they read as "someone else's problem" and get waved through
as flakes.

This also makes PR review harder in a specific way: with no green main
baseline, a contributor who sees sanitize-cpu red cannot tell whether they
caused it, and the cheap move is to assume they did not.

Suggested split

Worth treating as two pieces of work: triage the five sanitizer failures (each
likely its own root cause), and decide what a verified main should mean —
e.g. a post-merge run that is not cancellable, or a scheduled full run on a
quiet cadence, so there is a known-good baseline to diff against.

Found while holding #259 for a red sanitize-cpu rather than assuming it was
noise.

[localai-bot]

Corroborated: the same job is failure on main itself.

When I filed this I could not get a control, because every main CI run is cancelled before it finishes. An independent run on an unrelated branch (row/LORA-RUNTIME-W2, PR #282 — a LoRA change touching nothing in these tests) has now supplied one:

• sanitize-cpu (address,undefined) on that PR: 99% tests passed, 5 tests failed out of 369 — exactly the five tests listed above, with neither LoRA binary among them.
• sanitize-cpu (thread) green on the same run.
• The same job is failure on main at 68f434e5.

So this is no longer 'inferred pre-existing from diff scope'. It is observed on main directly, and reproduced across two unrelated PR branches (#259 and #282).

That closes the attribution question and leaves the two pieces of work this issue proposed:

1. Triage the five. Each likely has its own root cause; test_capi heap corruption was already seen during Jetson Thor sm_110 bring-up, so at least one predates this window by a long way.
2. Decide what a verified main means. The cancellation pattern is structural, not incidental — merges land faster than a ~27-minute sanitizer job, and each push cancels the in-flight run for its concurrency group. A post-merge run that is not cancellable, or a scheduled full run on a quiet cadence, would give the known-good baseline that currently does not exist.

The practical cost is already visible in this session: two separate rows had to spend a build cycle proving a red sanitize-cpu was not theirs. Without a green baseline that cost is paid by every PR, and the cheap way out — assuming it is someone else's — is exactly how a real regression would slip through.

[localai-bot]

Evidence refresh from 2026-08-11 — the count in the title is now 6, not 5, and the thread arm is red too.

Two runs on the same day, one of which is a docs-only PR and therefore cannot have caused any of it:

Job	PR #311 (docs-only, docs/FEATURES.md prose)	PR #324 (five model TUs + records)
sanitize-cpu (address,undefined)	test_load_direct_upload, test_llama_embedding_fold, test_laguna_nvfp4_loader, test_llm_engine, test_openai_api_server, test_capi — 6/369	the identical 6
sanitize-cpu (thread)	test_lmcache_connector, test_llm_engine — 2/369	test_llm_engine — 1/369

Two things this pins down:

1. test_llm_engine belongs on this issue. It is red under ASan/UBSan and under TSan, on a PR that changed only markdown prose. The body lists five tests; it is six.
2. sanitize-cpu (thread) is in the same state as the ASan arm and is not currently named here at all. test_llm_engine is red on it consistently; test_lmcache_connector is intermittent (red on docs(FEATURES): Vulkan is at parity with llama.cpp, not behind it (#310) #311, green on fold(FUSION-DENSE-MIGRATE): route the 5 no-blocker dense SwiGLU MLPs onto the merged-GEMM seam (#299) #324), which is the signature of a race that reports on some schedules and not others — exactly what a TSan gate exists to catch, so it should not be dismissed as flake.

Because the ASan failure set is byte-identical between a docs-only PR and a code PR, it stays useless as a signal: a real regression introduced by any PR would be invisible against this floor. That is the argument the issue already makes, restated with a controlled comparison rather than an assertion.