head: parallel batched bound walk + per-request k (the insitu ledger's first catch) - #43
Merged
Merged
Conversation
…n's verdict, answered The insitu ledger's first production run caught the bounded head losing: 469 ms/call against the dense head's 68, evaluating ~91% of the vocab. Two causes, both fixed: 1. The tile walk was SERIAL while the dense path it replaced was rayon-wide. Tiles now evaluate in parallel batches (results merge into the heap serially, the threshold advances between batches; the stop rule is unchanged, so exactness is preserved — a batch can only overshoot the serial stopping point by its own size). Batch size ramps from exactly-enough-to-fill-the-heap to 64 tiles, so a small k prunes from the second batch onward and wide parallelism returns within a few doublings (a flat 64-tile first batch was measured evaluating an entire small test vocab before any threshold existed). 2. k = 1024 cannot prune this model — the 1024th logit sits deep in the bulk, so the threshold clears almost no tile bound. But the one caller that knows the sampler also knows greedy consults ONLY the argmax: serve now scopes a per-request k (RequestTopK guard — fetch_max under overlap, drop falls back to the safe env default) of 1 for temperature-0 requests and the request's own top_k otherwise. At k = 1 the threshold is the running max, which is where the norm bound actually bites. 391 lib tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
physics515
force-pushed
the
claude/head-request-k
branch
from
August 28, 2026 16:00
4da06b4 to
0db8530
Compare
physics515
pushed a commit
that referenced
this pull request
Aug 28, 2026
Three conflicts, all the same shape: this branch's side was the whole-tree rustfmt, main's side was real work landing under it (PR #43's bounded head + in-situ ledger, PR #46's shim fail-fast, and the KV-f16 byte pricing). Took main's code in every case and re-derived this branch's two contributions on top of it — the `try_to_vec` / `try_into_vec` rename re-applied across the tree, then `cargo fmt --all`. Also dropped the narrow `packed_gemv` re-export in `nn/mod.rs`: main's wider one (adding `Q4GemmOps` / `Q4HeadOps` / `try_q4s_head`) supersedes it. Verified on the merge result, not on either parent: `cargo check --all-targets --keep-going` green, `cargo fmt --check` clean, 391 lib tests + 6 mummu-mix pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The in-situ ledger's first production run (post-#42, the 27B on the box) measured the bounded head at 469 ms/call vs the dense head's 68, evaluating ~91% of the vocab — the serial tile walk threw away the dense path's 16-way parallelism, and k=1024's threshold sits too deep in the bulk to prune this model.
Fixes, both measured-cause-first:
RequestTopKper request — 1 for greedy (temperature 0; the sampler reads only the argmax, and at k=1 the threshold is the running max, where the Cauchy–Schwarz bound actually prunes), the request's owntop_kotherwise. Overlapping requests combine viafetch_max; the guard's drop falls back to the safe env default, so a concurrent request can never see a k smaller than its need.391 lib tests green. The serve-side A/B on the 27B follows in the benchmark run this PR unblocks.
🤖 Generated with Claude Code