Skip to content

Four pre-existing CUDA ctest failures on main, attributed but unowned: test_capi (SIGSEGV), test_qwen3_apc_e2e, test_minicpm3_paged_engine, test_llama_paged_engine #248

Description

@localai-bot

Four CUDA ctest failures reproduce identically on unmodified origin/main
same assertion, same line, same pass/fail counts — measured by a direct A/B on the
same box and toolchain. They are pre-existing and unowned. .agents/specs/triton-aot-default-on.md
records each as "owed its own issue"; none was filed. This is that record.

They are grouped here because they were attributed by one measurement pass, not
because they share a cause. Each needs its own investigation.

test failure
test_capi SIGSEGV at tests/capi/test_capi.cpp:480 — the ABI v8 custom-logits-processor case on a synthetic engine (MakeSyntheticEngine(), :243-248). 47/47 assertions passed before the crash; 3 of 4 cases green.
test_qwen3_apc_e2e REQUIRE(anchor_ok) at :375, 59 assertions / 58 passed. Automatic prefix caching.
test_minicpm3_paged_engine REQUIRE(first_div < 0) at :224, 26 assertions / 25 passed.
test_llama_paged_engine attributed to main by the same arm.

Notes that will save the next person time

test_capi is not the documented flake. .agents/environment.md:189 calls
test_capi a -j-parallelism flake on this box, but this run was ctest -j 1,
so that explanation is unavailable and the SIGSEGV needs a real diagnosis. There is
prior art worth checking: a heap-corruption failure in test_capi was previously
seen on a different arch during sm_110 bring-up, so this may not be new.

Two of these are token-drift assertions against committed goldens
(test_qwen3_apc_e2e, test_minicpm3_paged_engine). Under "correctness first,
always" those deserve priority over the crash, since a silently wrong golden is
worse than a loud segfault.

Related, already tracked

Reproducing

Clean CUDA build on sm_121a, then ctest -j 1. Use -j 1ctest -j 4
OOM-rebooted this box (unified memory stacks concurrent model gates into host RAM).
Prefer in-suite runs: a standalone model gate here ran ~50x slower and reached
59 GB RSS on a 119 GB box, because the suite leaves the checkpoint warm in page
cache.

Found while gating #219 / #232; none is caused by that change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions