record(BENCH-QWEN38-27B-FOUR-WAY): the SGLang blocker was stale and the oracle had already run (#979)

[localai-bot]

FOLLOWING_AGENTS_PROTOCOL

Scopes the four-way benchmark campaign (us, vLLM, llama.cpp, SGLang) for
Qwen/Qwen3.8-27B on GB10. Issue
#979.

Records and a spec only. No product code, no measurement taken here.

This is a reconciliation, not a new row

The campaign already had owners. competitive-benchmarks.md defines
BENCH-CUDA-VLLM, BENCH-CUDA-SGLANG, BENCH-CPU-LLAMA and BENCH-VK-LLAMA
with a canonical serve-low workload, and cuda-sglang-low-concurrency.md
owns three backend rows. Opening a parallel row would have competed with its own
record, so this makes the existing records true and scopes the new subject onto
them.

The recorded blocker was stale

BACKEND-GATE-CUDA-SGLANG was BLOCKED on SERVE-ASYNC-LLM, reason "HTTP
TTFT/ITL cannot be measured honestly yet". That reason no longer holds, on four
independent lines:

• serving_completion.h:9-11 names the live pull-based SseStream over
  AsyncLLM as the production path, with the buffered LLMEngine constructor a
  test seam.
• api_server.cpp:971-981 drives set_chunked_content_provider off
  SseStream::next, one chunk at a time.
• run_serve_low.py:296-310 enforces it, refusing a probe with no
  token-bearing event, a wrong chunk count, first_chunk_s >= total_s, or a
  sub-floor spread.
• Demonstrated on hardware: the 2026-07-28 run measured c16 mean TTFT 2980 ms
  against mean ITL 154.4 ms over 128 tokens. First byte preceded completion by
  about 20 s, which a buffered server cannot do.

Moved to PARTIAL, not DONE. Residuals named: c1/c2/c4, the in-series vLLM
arm, 35B, SGLANG-ORACLE-CORRECT, paired traces, P2 exact-equivalence
classification, and every Qwen3.8-27B point.

The oracle said gateable = no for two and a half weeks after it had run

sglang-matrix.md:162 and docs/STATUS.md:180 record CLAIM-SGLANG-PERF-BENCH
on 2026-07-28: the v0.5.15-cu130 arm64 image pulled and ran 27B-NVFP4 on GB10
sm_121a, CUDA graphs captured, 3 reps at c8/c16, 80x128 tokens, 0 errors. The
oracle file still said "no SGLang run has been recorded on this project's
hardware". Two files in one tree contradicting each other is the failure the
oracle record exists to prevent. Now gateable = yes, discharging the SGLang
third of #647.

That run is not voided by #931: the keepalive needed 15 s of per-request
collector silence, and the worst recorded p99 TTFT was 7220 ms. The argument is
made on silence duration alone, so it does not depend on whether
sglang.bench_serving shares vllm bench serve's strip-before-parse weakness.
It is independently closed by the recorded exact 80x128 output tokens with 0
errors, since a mid-stream comment frame causes silent truncation that a full
token count excludes.

Two of six engine pairs are not comparable, on the container

vLLM at pin 555967922 has no in-tree GGUF at all: 6635279d8
(vllm#39612) migrated it to an out-of-tree plugin, leaving LoadFormats with no
gguf entry, no gguf in quantization/__init__.py, and
setup.py:1300 pointing at vllm-gguf-plugin. llama.cpp reads only GGUF.
So no artifact both can open exists at any quantization, and both
vLLM vs llama.cpp and SGLang vs llama.cpp are closed on that ground.

The deliverable is therefore a matrix with a stated common denominator per pair,
never one headline ratio, and no pair is compared across quantizations.

Three source claims were wrong and are withdrawn here

Each was caught in review and re-derived at the pins:

• llama.cpp does have NVFP4. GGML_TYPE_NVFP4 = 40, the loader ftype, type
  traits, the CUDA MMQ instance and the ModelOpt-to-GGUF converter are
  upstream; only the two sm_121a prefill GEMMs are fork-local. The earlier
  "llama.cpp has no NVFP4" reason for both not-comparable verdicts was false,
  which is why they are re-derived on the container above.
• SGLang's conv1d fails loudly, not silently. qwen3_5.py:199 passes
  quant_config=None, so linear.py:176-179 assigns UnquantizedLinearMethod
  and GGUFConfig.get_quant_method is never reached. The withdrawn wording
  claimed the opposite polarity. What replaces it is an assertion about an
  absent guard, which is directly checkable: loader.py:2149-2153 writes the
  key with no None check, weight_utils.py:1280,1321 skip an unmapped tensor
  with no else and no counter, qwen3_5.py:1359-1412 never compares
  loaded_params against params_dict, and loader.py:2184-2186 discards the
  return value. Which parameters fail to resolve stays an open question with the
  experiment named.
• DSpark is not absent from the pinned tree. docs_new/index.mdx links the
  2026-07-06 blog, three days before the pinned commit f63458b5 of 2026-07-09,
  and SGLang ships a plugin registration API for out-of-tree speculators. The
  true claim is scoped to python/sglang/srt/speculative/, and the conclusion
  that a drafted arm needs a deliberate pin advance now rests on a stated
  argument rather than on a false absence.

The vendor claim, quarantined, and our own measurement beside it

The retrievable source is the NVIDIA forum thread, which reports "~34 tok/s
real-world, 38.0 average on eval-style workloads, 46.7 peak". 38.28 does
not appear in it. The X post returns HTTP 402, so its text comes from a search
index and its date is derived from the status snowflake. Both limits are stated.
Drafter size, TTFT exclusion, batch size and the spread are labelled UNVERIFIED.

Set against it, our own measured head-to-head on byte-identical
Qwen3.6-27B-NVFP4 weights, same box, same corpus, both arms raw: SGLang output
throughput 40.8 tok/s against ours 90.3 at c16, a 2.21x win, and 1.44x
at c8, with TTFT 6 to 12x lower. TPOT and ITL remain an open gap, attributed to
batch composition rather than per-token kernel cost, with the shipped default
unchanged so the deficit stands at the measured configuration.

Without that beside it a reader met "SGLang published 38.0" and "ours is 10.756"
with drafted-versus-raw as the only objection, and could leave believing SGLang
is simply faster.

A missing row, filed

BACKEND-GATE-CUDA-LLAMACPP, INVENTORIED. No existing row covers llama.cpp on
a current CUDA card: the CPU, -LEGACY Pascal/Volta/Turing, Metal, Vulkan and
the eleven BACKEND-GGML-* breadth rows are all distinct, and
bench-27b-five-way.md:12 records the orphan. Ratchet 82 to 83 following the two
Tenstorrent precedents, with a mutation case keyed to its own row and a guard
that the new ID is a strict prefix of -LEGACY. Removing the row yields 3 FAIL
plus 1 ERROR, reproduced in review.

A correction made before it could become permanent

An earlier commit on this branch recorded .agents/issue-index.md:266 as
uncorrectable, on the grounds that the index is append-only and a landed row is
never edited. The row had not landed. 17187f134 is not an ancestor of
origin/main, whose index carries zero #979 rows. The append-only rule's own
rationale is that a union merge duplicates an edited line, which requires the row
to exist at the merge base. It did not, and no other branch carried it.

Corrected in place, with both index gates green afterwards. Had this merged
first, main would have permanently asserted two claims this same branch proves
false, in the round whose purpose was retiring false recorded facts, and only
then would the window have actually closed.

Gates

scripts/agent-preflight.sh: 76 ok, 0 FAIL, exit 0, all five block headings
executed. The Commit trailers vs origin/main block genuinely ran rather than
silently skipping, which it does when origin/main is not an ancestor of HEAD
while still printing "All gates green.".

check-agent-record.py: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83.
check-issue-index-append-only.py: OK after the in-place edit.

Owed

• Qwen3.8-27B: gate NVFP4 and Q4_K_M with mmproj #821 blocks the quantized arms: our NVFP4 and Q4_K_M arms for this
  checkpoint, including the separate clip projector.
• Qwen3.8-27B (bf16): the token gate and every speed axis are OWED — the checkpoint loads and runs, and has never been adjudicated or measured #915's c1 and c8 output-throughput cells stay WITHHELD until re-run under
  the paired interleaved protocol on a binary carrying Our OpenAI server fails requests under concurrency where vLLM does not (1/6 at c1, 12/48 at c8 on Qwen3.8-27B), and the failures silently corrupt every throughput ratio taken against it #931's fix.
• An NVFP4 three-way needs a layout-equivalence proof, since
  conversion/base.py:654-676 repacks rather than re-quantizes. Two assertions
  are owed: no e4m3 scale byte with its sign bit set, and the GDN head
  permutation shown value-preserving.
• The provenance of the third-party vLLM figures is not established, and they
  are not used as a denominator until it is.
• agent-preflight.sh reports "All gates green." while silently skipping its
  trailer block on a branch behind main. Found during Our OpenAI server fails requests under concurrency where vLLM does not (1/6 at c1, 12/48 at c8 on Qwen3.8-27B), and the failures silently corrupt every throughput ratio taken against it #931, unfiled because an
  issue with no owning row reds the unowned ratchet. It needs its own row.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]