Skip to content

head: parallel batched bound walk + per-request k (the insitu ledger's first catch) - #43

Merged
physics515 merged 1 commit into
mainfrom
claude/head-request-k
Aug 28, 2026
Merged

head: parallel batched bound walk + per-request k (the insitu ledger's first catch)#43
physics515 merged 1 commit into
mainfrom
claude/head-request-k

Conversation

@physics515

Copy link
Copy Markdown
Owner

The in-situ ledger's first production run (post-#42, the 27B on the box) measured the bounded head at 469 ms/call vs the dense head's 68, evaluating ~91% of the vocab — the serial tile walk threw away the dense path's 16-way parallelism, and k=1024's threshold sits too deep in the bulk to prune this model.

Fixes, both measured-cause-first:

  • Parallel batched bound walk: tiles evaluate across the rayon pool in batches; the threshold advances between batches; the stop rule is unchanged so exactness is preserved (overshoot bounded by one batch). Batch size ramps from ceil(k/TILE) to 64 so small-k pruning engages immediately and parallel width returns within a few doublings.
  • Per-request k: serve scopes RequestTopK per request — 1 for greedy (temperature 0; the sampler reads only the argmax, and at k=1 the threshold is the running max, where the Cauchy–Schwarz bound actually prunes), the request's own top_k otherwise. Overlapping requests combine via fetch_max; the guard's drop falls back to the safe env default, so a concurrent request can never see a k smaller than its need.

391 lib tests green. The serve-side A/B on the 27B follows in the benchmark run this PR unblocks.

🤖 Generated with Claude Code

…n's verdict, answered

The insitu ledger's first production run caught the bounded head losing:
469 ms/call against the dense head's 68, evaluating ~91% of the vocab.
Two causes, both fixed:

1. The tile walk was SERIAL while the dense path it replaced was
   rayon-wide. Tiles now evaluate in parallel batches (results merge into
   the heap serially, the threshold advances between batches; the stop
   rule is unchanged, so exactness is preserved — a batch can only
   overshoot the serial stopping point by its own size). Batch size ramps
   from exactly-enough-to-fill-the-heap to 64 tiles, so a small k prunes
   from the second batch onward and wide parallelism returns within a few
   doublings (a flat 64-tile first batch was measured evaluating an
   entire small test vocab before any threshold existed).

2. k = 1024 cannot prune this model — the 1024th logit sits deep in the
   bulk, so the threshold clears almost no tile bound. But the one caller
   that knows the sampler also knows greedy consults ONLY the argmax:
   serve now scopes a per-request k (RequestTopK guard — fetch_max under
   overlap, drop falls back to the safe env default) of 1 for
   temperature-0 requests and the request's own top_k otherwise. At k = 1
   the threshold is the running max, which is where the norm bound
   actually bites.

391 lib tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@physics515
physics515 force-pushed the claude/head-request-k branch from 4da06b4 to 0db8530 Compare August 28, 2026 16:00
@physics515
physics515 merged commit 784ca8d into main Aug 28, 2026
physics515 pushed a commit that referenced this pull request Aug 28, 2026
Three conflicts, all the same shape: this branch's side was the whole-tree
rustfmt, main's side was real work landing under it (PR #43's bounded head +
in-situ ledger, PR #46's shim fail-fast, and the KV-f16 byte pricing). Took
main's code in every case and re-derived this branch's two contributions on
top of it — the `try_to_vec` / `try_into_vec` rename re-applied across the
tree, then `cargo fmt --all`. Also dropped the narrow `packed_gemv` re-export
in `nn/mod.rs`: main's wider one (adding `Q4GemmOps` / `Q4HeadOps` /
`try_q4s_head`) supersedes it.

Verified on the merge result, not on either parent: `cargo check --all-targets
--keep-going` green, `cargo fmt --check` clean, 391 lib tests + 6 mummu-mix
pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant