record(ORACLE-LLAMACPP-REPIN-STOCK): the pin named a tree that exists nowhere (#857)

[localai-bot]

FOLLOWING_AGENTS_PROTOCOL

Repins the llama.cpp oracle from a local fork to stock upstream and enumerates
every measurement the fork contaminated. Owning issue
#857; the re-measurement is
#1003.

Records and a spec only. No product behaviour changes.

The pin named a tree that exists nowhere

.agents/oracles/llama-cpp.md recorded pin = 237ad9b96, pin_label = b9892,
gateable = yes. All three were wrong:

• 237ad9b96 is on no remote. GitHub returns No commit found for SHA for
  both mudler/llama.cpp and ggml-org/llama.cpp. It is a local-only commit,
  and the working tree carried 27 uncommitted files on top of it, so builds from
  that directory measured a tree in no repository at all.
• b9892 is not a label, it is a collision. git rev-list --count returns
  9892 for the fork, and upstream tag b9892 (ee445f93d) is ALSO 65 commits
  past b9827 and ALSO counts 9892. Neither is an ancestor of the other. So a
  reader checking out b9892 to reproduce a number silently gets stock upstream.
  That already happened in-tree: rpi5-a76-llamacpp-20260806.md used stock
  b9892 while cpu-x86-llamacpp-20260811.md built the fork under the same
  label. Two evidence files, one label, two trees.
• The 65 fork commits are our own performance work — native NVFP4 W4A4
  prefill GEMM, Marlin-style W4A16 grouped MoE GEMM, MoE-decode CUDA graph,
  decode-graph reuse, fused residual-add plus RMSNorm, chunked GDN prefill, a
  TTFT prefill-first scheduler mode. Six of them touch ggml/src/ggml-cpu/, and
  570aadd7a emits its fused GDN and discriminated SSM_CONV ops default-on
  with a CPU reference implementation, so the CPU floor built GGML_CUDA=OFF hit
  them too.

The oracle file defined this comparator as "the numbers a user can actually get
today" and then pinned a local fork carrying 65 of our own optimisations. Both
cannot be true.

Now pinned to stock b10451 / 10bf611e5, reachable, with the tag bound to
the commit on the remote — a check the old label would have failed.
gateable = no, because stock has never been built or run on this hardware, with
#857 owning the measurement. Carrying yes across the repin would have been the
defect this row exists to remove.

Enumeration by sweep, because hand-enumeration failed three times

Round one listed the set by hand and missed the Vulkan MET and the Pi 5 RSS.
Round two missed Muse Glimmer. Round three missed README.md because the sweep's
path set excluded the repo root.

The spec now carries a runnable three-stage sweep with git ls-files and no
path arguments — no include list to fall behind the tree, no exclusion list to
justify. Stage 1 marks a line when it carries a comparison token and llama.cpp is
attributed to it: the line names it, or a line within three does, or the line is
a table body row whose HEADER does. That third clause is what no git grep -C
can express, and it is what catches | Prefill | 223.8 tok/s | 177.3 | 1.18x |
whose own text never says llama.cpp. Stage 2 filters by the page's own rule at
docs/BENCHMARKS.md:32-34. Stage 3 reads each survivor's revision off its
evidence record.

Twelve contaminated measurements, seven favourable verdicts, ordered by
margin against their own noise floor rather than by margin size. That ordering
matters: the Vulkan decode MET at 4.36 versus 4.35 is a 0.23% margin inside
its own 0.69% spread, so it is far more fragile than the 18% prefill, and
README.md states it publicly as "matches llama.cpp".

Five llama.cpp revisions, and one of them is not a revision

The record twice asserted a universal the same page disproved. Measured, the
tree carries the fork, stock b9892, stock 704485942, stock 030ebb5, and
a Poolside fork on branch laguna with no SHA recorded anywhere. A branch is
a moving reference, so Laguna's 27.8 tok/s was never reproducible and cannot
simply be re-run: re-taking it means choosing a revision first. That figure
drives the campaign's "15x to 18x behind" statements.

Contamination reaching a shipped default

gguf_keep_quant.cpp quotes llama.cpp denominators in source as the
justification for VT_GGUF_KEEP_F16 shipping default ON. The owning A/B has
three axes and two regress: prefill about 10% worse, decode about 1.4% worse,
bought for 1.05 GiB. The recorded tie-break is stated in the floor's own terms,
"comfortably above the competitor floor".

So the default is owed a decision, not just a re-take, and #1003 says so. If a
re-taken stock pp128 lands above 204 t/s, that clause fails and the default's
only recorded justification is gone. The default itself is unchanged here, since
changing a shipped default on a contaminated justification would be the same
error in the other direction.

Two figures in that comment, pp128 176.6 and 205 t/s, appear in no recorded
run. They are self-consistent (205/176.6 = 1.16), so they were computed rather
than mistyped. Recorded as untraceable rather than reconciled to the spec's
173.2/1.18x, because picking one would invent a provenance.

The instrument was wrong five times, and the fifth is stated rather than fixed

Each round hardened the sweep and the next found a new hole, so the record now
says to assume a residual rather than trust the green.

• The path set excluded the repo root. Removed entirely.
• × was present but silently dead: × is not \w, so \b after it needs a
  FOLLOWING word character, and only 3.9×decode matched. A token list can be
  WRONG as well as short.
• Stage 2 spelled the same rule with no trailing \b, so two expressions
  disagreed and only the narrower one gated. Stage 2 is now compiled from the
  same dict.
• Stage 2's \b[1-9] never excluded sub-1.0 ratios, because in 0.204x the
  boundary is satisfied after the decimal point. Two defects were cancelling, and
  the rule was wrong anyway since 0.758x is the Pi 5 RSS win.
• A live token is not a covered shape. The self-test now proves every one of
  19 alternatives is live, 43 mutations all red. It does NOT prove shape
  coverage: narrowing the ratio branch to require a decimal passes green and
  silently drops 32 candidates, and narrowing the attribution window to
  range(n, n+1) passes green while deleting 64% of the instrument's output,
  because every control was a string and the attribution half is not a string.
  Controls added for the two live shapes and for the attribution half; the
  residual is stated with its measured table and carried under ## Owed.

Evidence

scripts/agent-preflight.sh: 77 results, 77 ok, 0 FAIL, 0 SKIP, exit 0,
gated against origin/main d1b0ea3a8, named in both range headings.

The sweep is reproducible from the committed markdown: extract the fenced block
and run it. 1263 candidates in 145 files, 1033 favourable at this head; the
self-test exits 1 naming the strings it can no longer see.

test_cpu_x86_llamacpp_floor (#618) is load-dependent and passed here at an idle
box. It has now been observed failing at a one-minute loadavg of 11.99, below
three loads at which it passed, so the threshold reading of that list is retired.

Owed

• Every recorded llama.cpp floor was measured against a local-only fork carrying 65 of our own performance commits, and must be re-taken against the stock pin #1003 owes the twelve re-takes, ordered by fragility. Re-taking prefill
  alone does not discharge it.
• The keep-f16 default owes a decision, not only a number.
• Laguna's Poolside figure owes a chosen revision before it can be re-taken
  at all.
• The sweep's residual: restore a leading \b to the ratio branches
  (measured cost 18 lines either way), and decide whether per-shape controls
  belong on every alternative or whether the instrument should stop living in a
  document.
• .agents/environment.md still carries the recipe that rebuilds the superseded
  fork; it is marked, and recipes are a surface the sweep does not cover by
  design.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]