docs(ENG-EXPERT-STREAM): the three expert-streaming knobs are a deployment surface, not an allowlist entry (#995) by localai-bot · Pull Request #997 · mudler/vllm.cpp

localai-bot · 2026-08-16T09:36:14Z

Closes #995.

scripts/agent-preflight.sh was RED on origin/main itself, so every branch cut from it inherited a failure its own diff did not cause. Re-verified at HEAD (4496ef196) in a clean detached worktree before changing anything, because main moves several times an hour:

ERROR: production env var(s) read from src/+include/ are neither documented in
docs/ENVIRONMENT.md nor on scripts/env-doc-allowlist.txt:
  - VT_MOE_EXPERT_STREAM
  - VT_MOE_EXPERT_STREAM_SLOTS
  - VT_MOE_EXPERT_STREAM_SLOT_BYTES

with tests/scripts/test_check_env_doc.py failing consequentially on the same three names. They arrived with the ENG-EXPERT-STREAM W4 wiring commit 3005447f8 (#993). Same shape as #965 and #968: a pre-existing red presenting to each author in turn as their own.

Documented, not allowlisted

The allowlist is for kernel-internal micro-tuning switches whose meaning is tied to one kernel. These are not that. VT_MOE_EXPERT_STREAM changes which decode path runs — it turns OFF the default-on grouped MoE, because that path stages the whole tower and cannot stream — and the other two set a host memory budget the process reserves up front and never grows. An operator serving an mmap'd MoE tower larger than RAM has to be able to choose those numbers.

What each entry claims, and the read site it came from

Read from the implementation, not guessed from the name.

Knob	Default	Derived from
`VT_MOE_EXPERT_STREAM`	off	`qwen3_5.cpp:5145` — read ONCE per process and only its FIRST character examined (`v[0] != '0' && v[0] != '\0'`), a deliberate divergence from the page's general flag convention, stated as such. CPU-only guard at `:5234`; grouping disabled with a stderr notice at `:5313`
`VT_MOE_EXPERT_STREAM_SLOTS`	`64`	`qwen3_5.cpp:5195` — `std::atol`, applied only when `> 0`, so unset / empty / zero / negative / unparseable all keep `64`. Exhaustion semantics from `expert_slot_cache.h:74-88`: every slot acquired in a step is protected until `EndStep`, so a budget under one step's working set returns slot `-1` and the caller reads the tower directly (`:5168-5179`) — correct, slow, counted
`VT_MOE_EXPERT_STREAM_SLOT_BYTES`	the size of the FIRST slice streamed	`qwen3_5.cpp:5189` — units confirmed as BYTES rather than inferred from the name (`HostExpertSlotStore` sizes `buf_` as `slots * slot_bytes` and reports `resident_bytes()`). `std::atoll`, applied only when `> 0`. A later, larger slice is REFUSED by name through `VT_CHECK` (`:5160-5164`, a thrown `std::runtime_error`), never truncated or silently routed back to the mmap path

The section also records that the lever is still experimental: the W4 measurement in .agents/specs/expert-streaming.md is token-identical output and 4.5x on TTFT with no steady-state decode gain, because the slot is filled by copying from the mapping and so still takes the fault it was meant to avoid. A doc entry that recommended the knob would be inventing a result the spec explicitly refutes.

Gate

Red-before at 4496ef196: the block quoted above, check-env-doc exit 1; test_check_env_doc AssertionError: Lists differ: ['VT_MOE_EXPERT_STREAM', ...] != [], exit 1.
Green-after: OK: all 340 production env vars are documented or classified kernel-internal (exit 0); test_check_env_doc 8 tests OK (exit 0); full scripts/agent-preflight.sh green.
Mutation: deleting only the VT_MOE_EXPERT_STREAM_SLOTS table row (uniqueness-asserted anchor — that name occurs on exactly one line) REDs the checker naming exactly that knob, and fails test_shipped_tree_is_fully_covered with ['VT_MOE_EXPERT_STREAM_SLOTS'] != []; both exit 1. Restored byte-for-byte (sha256 16f3b14f…e913c9 before and after) and both gates green again. The entries are load-bearing, not merely present.

No other docs/ projection is owed: no row changed lifecycle state, no measurement was accepted, and no feature, command, config key or C API changed — the knobs already shipped in 3005447f8; this only describes them.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…yment surface, not an allowlist entry (#995) `scripts/agent-preflight.sh` was RED on `origin/main` itself, so every branch cut from it inherited a failure its own diff did not cause. Measured in a clean detached worktree at `4496ef196`: ERROR: production env var(s) read from src/+include/ are neither documented in docs/ENVIRONMENT.md nor on scripts/env-doc-allowlist.txt: - VT_MOE_EXPERT_STREAM - VT_MOE_EXPERT_STREAM_SLOTS - VT_MOE_EXPERT_STREAM_SLOT_BYTES and `tests/scripts/test_check_env_doc.py` failing consequentially on the same three names. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` (#993). Same shape as #965 and #968: a pre-existing red that presents to each author in turn as their own. WHY DOCUMENTED AND NOT ALLOWLISTED. The allowlist is for kernel-internal micro-tuning switches whose meaning is tied to one kernel. These are not that. `VT_MOE_EXPERT_STREAM` changes which decode path runs (it turns OFF the default-on grouped MoE, because that path stages the whole tower and cannot stream), and the other two set a host memory budget the process reserves up front and never grows. An operator serving a mmap'd MoE tower larger than RAM has to be able to choose those numbers, which is what `docs/ENVIRONMENT.md` is for. WHAT EACH ENTRY CLAIMS, AND THE READ SITE IT WAS DERIVED FROM. Read, not guessed from the name. * `VT_MOE_EXPERT_STREAM`, off by default, `qwen3_5.cpp:5145` (`Qwen35ExpertStreamRequested`). Read ONCE per process and only its FIRST character is examined (`v[0] != '0' && v[0] != '\0'`), which is a deliberate divergence from this page's general flag convention and is stated as such. CPU only: `KqExpertSlice` (`:5234`) guards on `is_cpu()`. Disables grouping at `:5313` with a one-line stderr notice. * `VT_MOE_EXPERT_STREAM_SLOTS`, default `64`, `qwen3_5.cpp:5195`. `std::atol`, applied only when `> 0`, so unset / empty / zero / negative / unparseable all keep `64`. Exhaustion is documented from `expert_slot_cache.h:74-88`: every slot acquired in a step is protected until `EndStep`, so a budget under one step's working set returns slot `-1` and the caller reads the tower directly (`:5168-5179`), which is correct, slow, and counted. * `VT_MOE_EXPERT_STREAM_SLOT_BYTES`, in BYTES (confirmed, not inferred from the name: `HostExpertSlotStore` sizes `buf_` as `slots * slot_bytes` and reports `resident_bytes()`), `qwen3_5.cpp:5189`. `std::atoll`, applied only when `> 0`. Its default is the size of the FIRST slice streamed, because `Qwen35ExpertStream::Get(bytes)` constructs the singleton with the current slice's size; a later larger slice is REFUSED by name through `VT_CHECK` (`:5160-5164`, a thrown `std::runtime_error`) rather than truncated or silently routed back to the mmap path. The section also records, rather than omits, that the lever is still experimental: the W4 measurement in `.agents/specs/expert-streaming.md` is token-identical output and 4.5x on TTFT with NO steady-state decode gain, because the slot is filled by copying from the mapping and so still takes the fault it was meant to avoid. A doc entry that recommended it would be inventing a result the spec explicitly refutes. GATE. Red-before verbatim above at `4496ef196`. Green-after: `check-env-doc` prints `OK: all 340 production env vars are documented or classified kernel-internal` (exit 0), `test_check_env_doc.py` runs 8 tests OK (exit 0), and full `scripts/agent-preflight.sh` is green. MUTATION. Deleting only the `VT_MOE_EXPERT_STREAM_SLOTS` table row (a uniqueness-asserted anchor; that name occurs on exactly one line) REDs the checker naming exactly that knob and fails `test_shipped_tree_is_fully_covered` with `['VT_MOE_EXPERT_STREAM_SLOTS'] != []`, both exit 1. Restored byte-for-byte, `sha256 16f3b14f9c990e06f6a277d6f906a008ad108f3e047cac633c92f56351e913c9` before and after, and both gates green again. So the entries are load-bearing rather than merely present. No other `docs/` projection is owed: no row changed lifecycle state, no measurement was accepted, and no feature, command, config key or C API changed — the knobs already shipped in `3005447f8`, this only describes them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…reconciled BY HAND (#995) `origin/main` advanced to `3ce1cf7c7` (#992) while this branch's gate was running. Merged at that pinned SHA rather than at a moving ref. `.agents/issue-index.md` was the only overlapping path, and its `merge=union` driver had already produced a clean auto-merge. That auto-merge was DISCARDED: the file was taken from `3ce1cf7c7` wholesale and this branch's single row re-appended at the end, then asserted byte-for-byte that the target's 114534 bytes are a strict PREFIX of the 115617-byte result. Union-clean is not union-correct on an append-only record — it is exactly the interleaving that has had to be repaired repeatedly — so the invariant is checked rather than assumed. Row count 252, no row edited and none deleted. Nothing else conflicted: `docs/ENVIRONMENT.md` is untouched by `3ce1cf7c7`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…rmed (#1022) (#1025) Closes #1022. `main` at `45b022cdc` is RED on `check-agent-record.py` — the CI job `agent-record` — for **every** branch cut from it: ``` ERROR: .agents/issue-index.md: issue #995 listed twice. Under `merge=union` a duplicate is what two branches appending the same issue look like ``` Reproduced on a clean detached worktree at `45b022cdc`, not the shared checkout (which reports differently on this surface). Independently corroborated by the A2-Q2a helper, which hit the same red at the same base. ## Three defects, one cause, one repair | # | Defect | Caught by | |---|---|---| | 1 | duplicate key `#995` (lines 270 and 272) | `check-agent-record.py` | | 2 | line 272 lacks its trailing `\|` — a 3-cell row in a 4-column table | **nothing** | | 3 | line 270 has an **unescaped** `\|` inside a code span — 5 rendered cells | **nothing** | Line 270 arrived with `332aed738` (#996), whose author recorded the env-doc red they hit while gating #986. Line 272 arrived with `45b022cdc` (#997), the fix for that same red. **#997 did the right thing** — it discarded the union driver's clean auto-merge and re-appended against its pinned base `3ce1cf7c7`, asserting the prefix property by hand. But `main` then advanced to `332aed738`, which had *already* added a `#995` row, and the squash-merge applied #997's append on top without seeing it. Neither author could have seen the other's row when they wrote theirs. A cell-count histogram over the whole file reads `{6: 252, 7: 1, 5: 1}` — **the only two malformed rows in the index were the two `#995` duplicates.** Merging them by key repairs all three defects at once. The merged row carries BOTH sides' content, per the keyed-record rule: the discoverer's provenance (found while gating #986, proved pre-existing with a matched-arm check, deliberately not fixed in flow and why) and the fixer's resolution (documented in `docs/ENVIRONMENT.md` rather than allowlisted, and why). ## Exception argued, not waived This change **deletes a row and edits another**, which is precisely what `scripts/check-issue-index-append-only.py` forbids. That gate is RED on this branch and I am not weakening it — there is no waiver registry, so the argument lives in the commit message where it is attached to the diff it excuses. The append-only contract **cannot** repair a duplicate: appending a third `#995` row makes `check-agent-record` angrier, and the file only becomes well-formed again by removing one. The two gates are in genuine contradiction on this tree, and that contradiction is the defect. The rule is preserved in substance, verified mechanically rather than by eye: - 254 → 253 rows; `#995` appears exactly **1x** - all 253 rows are 6-part (well-formed) — up from 252 - **252 unrelated rows byte-identical AND in the same order**, compared as a list - the survivor sits at the **earlier** of the two positions, so every subsequent row keeps the offset it was appended against - no key added, no key lost That gate is preflight-only and not in CI, so this costs no CI red. Once merged, later branches diff a `main` with no duplicate and see no removal in their range — the violation is confined to this branch. ## Scope note This is the **duplicate** case, which the checker does catch. It is *not* #1002 (append-only checker blind to an **interleave**, which preserves uniqueness and fails only the PREFIX property). #1002 stays open on its own terms. Both are consequences of a record surface every PR must write — the shape AGENTS.md warns about under "No surface that every PR must write". ## Gates - `check-agent-record.py` RED-before at `45b022cdc` (verbatim above) → **green after** - `check-env-doc.py` green (confirms #997's fix survives) - `scripts/agent-preflight.sh`: 2 failures, both accounted for — `issue-index append-only` (argued above) and `test_cpu_x86_llamacpp_floor` (exit 4 `NO_QUIET_WINDOW`, box load average **106**; diff is 1 markdown file and **0** code files, so it cannot be mine by construction) Diff is `.agents/issue-index.md`, +1/-2. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

) (#1030) FOLLOWING_AGENTS_PROTOCOL Fixes [#998](#998). Files [#999](#999) and [#1000](#1000). `scripts/agent-preflight.sh` reported `All gates green.` while silently skipping gates it never ran. It now reports a third state, `SKIP`, and the banner is unreachable when anything was skipped. ## What it did The commit-trailer block ran only when `origin/main` was an ancestor of `HEAD`. Otherwise it was skipped **silently**, and the run still printed the banner. It fired three times in one session: 1. On a branch five commits behind `main`: `74 ok, 0 FAIL, All gates green.`, trailer block never run, caught only by reading the script. 2. After merging `origin/main` and re-running: green **again**, because `origin/main` advanced mid-run. The remote-tracking ref is shared by every worktree of this checkout, so the guard was true at start and false when evaluated. 3. On an operator gate run: `All gates green.` at **74 ok instead of 76**, exactly the two trailer checks missing. The count drops and the banner does not change. An audit of all nine conditionals found a **second** block with the same shape: `Committed range vs origin/main`, whose unresolvable-ref arm silently dropped three more gates. Measured end to end, an unresolvable base costs **five**. ## The fix `skip()` mirrors `run()`, prints `SKIP` with its reason, and appends to a `skipped` array. `All gates green.` is guarded on both `failed` and `skipped` being empty, and it is the only banner in the file. `BASE_SHA` resolves `origin/main^{commit}` once before the first gate, both range blocks compare against that SHA, and both headings print it. **Exit code stays 0 on a skip, and `--fail-on-skip` is the opt-in that changes it.** `scripts/agent-ready.py` passes it, because that gate read preflight by exit code alone and would otherwise print `READY: local and live PR/CI evidence are green` over gates that never ran. That was this row's own thesis failing at its one machine consumer. The default is not a preference. `check-test-registration.py` executes preflight under a `git` shim that fails every call, so five gates skip on every invocation and it requires `rc == 0`. Flipping the default would red a checker that is itself in `CHECKERS`. Measured, not argued: as shipped `rc = 0`; with the default flipped, `rc = 1`. An earlier draft argued from `--role-only` as precedent. That argument is withdrawn in the spec: `--role-only` is a narrowing the caller opted into, and a skip is imposed on a caller who asked for a full run. The two are not the same kind of partial. ## A regression this branch introduced, found in review, and fixed here The first repair folded stderr into the value: `RANGE_COUNT="$(git rev-list --count "${BASE_SHA}..HEAD" 2>&1)"`. When git exits **0** and also writes to stderr, the count is non-numeric, `[ "$RANGE_COUNT" -gt 0 ]` errors and reads false, and both range blocks fall to the empty-range arm: ``` Committed range vs origin/main 51612d06…: empty, HEAD adds no commits. Commit trailers vs origin/main 51612d06…: empty, HEAD adds no commits. All gates green. rc=0 ``` Reproduced with a repo whose `.git/objects/info/alternates` names a missing path, so git warns and still exits 0. Counted three ways on one topology: pre-repair **77 ok**, shipped-with-regression **72**, repaired **77**. Five gates, and unlike the two original bugs this one failed in the **dishonest** direction. The spec's claim that all of them failed honestly is corrected. The repair has two halves and both earn their place: `2>/dev/null` keeps stderr out of the value, and a `case` predicate validating `^[0-9]+$` gives a non-numeric value an arm to land in. Reverting either reddens a different test and neither covers the other. The git message is not discarded, only kept out of the value. `RANGE_ERROR="$(... 2>&1 >/dev/null)"` captures it separately, matching the `ANCESTRY_ERROR` discipline two lines above, so an unborn HEAD still reports `fatal: ambiguous argument …` rather than `printed [] on stdout`. The rule is message-not-a-value, not no-message. ## Evidence Red-before on the original defect, one HEAD and four ref settings: | `origin/main` | before | after | |---|---|---| | ancestor | 76 ok, 0 SKIP, green | 77 ok, 0 SKIP, green | | divergent | 74 ok, 0 SKIP, **green** | 75 ok, **2 SKIP**, no banner | | unresolvable | 71 ok, 0 SKIP, **green** | 72 ok, **5 SKIP**, no banner | | empty range | 71 ok, green | 72 ok, green | An empty range is deliberately not a skip: a verdict over zero commits withholds nothing, and reporting it would fire on every freshly cut branch. `tests/scripts/test_agent_preflight_skip_report.py` is 17 cases. Every one is armed by at least one mutation. The suite runs the script from a scratch repo under a stub `python3` so a preflight-testing suite does not recurse, and the stub logs `"$@"` so the five base-taking gates are asserted to receive the pinned SHA and no ref. Before that log, changing `--base "$BASE_SHA"` to `--base origin/main` at three sites left the suite green. ## Two instrument failures worth recording Both were caught by hash checks rather than by reading results, and both are recorded in the spec. A `sed` mutation with a colliding delimiter exited non-zero, edited nothing, and the suite reported `OK` — a mutation that never applied wearing a passing test. The harness now refuses a result when the file hash is unchanged. A later harness restored with `git checkout --`, which reverts to the **committed** file and so deleted the uncommitted repair. Two mutations then measured a script with no fix in it and reddened a case that should have stayed green. Read at face value that is a finding about the code. It was entirely the instrument. ## Gates `scripts/agent-preflight.sh` on the merged head, gated against `origin/main` `4f2d91756`, named in both range headings: **74 ok, 3 FAIL, 0 SKIP**. The run skipped nothing. All three failures are inherited from `main` and none is touched by this branch: - `check-agent-record` and `test_agent_record` fail because `.agents/issue-index.md` on `main` carries **two rows for #995**. Reproduced on a clean detached worktree at `4f2d91756`. PR [#1025](#1025) repairs it. This branch does not touch that file outside the merge. - `test_cpu_x86_llamacpp_floor` is [#618](#618), load-dependent, at loadavg 37.65 here. Its assertions carry the harness's own reason: `waiting for quiet: busy=110%`. `check-env-doc` was red earlier in this work and is now green: #997 landed. ## Owed - [#999](#999) — `check-commit-style.py:125` raises on a non-ancestor base where `check-commit-trailers.py:331` takes a merge base. Until that is repaired, the ancestry guard cannot simply be dropped, which would be the better fix than reporting a skip. - [#1000](#1000) — `main` red on `check-env-doc` when filed, since fixed by #997. Note it duplicates #995's filing of the same defect; both name `ENG-EXPERT-STREAM` and the index is append-only, so neither row is edited. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 2 commits August 16, 2026 09:27

localai-bot mentioned this pull request Aug 16, 2026

spec(LTX25-DECODE-SPEED): the decode has no device arm, and the 60 GiB was never in it #1018

Closed

localai-bot merged commit 45b022c into main Aug 16, 2026
24 of 26 checks passed

localai-bot deleted the row/FIX-ENV-DOC-995 branch August 16, 2026 11:21

localai-bot mentioned this pull request Aug 16, 2026

spec(LTX25-DECODE-SPEED): the decode has no device arm, the 60 GiB was never in it, and lever 2 landed mid-review #1038

Merged

Navigation Menu

docs(ENG-EXPERT-STREAM): the three expert-streaming knobs are a deployment surface, not an allowlist entry (#995) - #997

docs(ENG-EXPERT-STREAM): the three expert-streaming knobs are a deployment surface, not an allowlist entry (#995)#997
localai-bot merged 2 commits into
mainfrom
row/FIX-ENV-DOC-995

localai-bot commented Aug 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 16, 2026

Documented, not allowlisted

What each entry claims, and the read site it came from

Gate

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants