bench(rocmfpx): exploratory bench results + tooling fixes#1072
Merged
Conversation
…lts/handoff - generate_results_json.py: ingest server-ab/*.json into a Tier B/C SUMMARY section (decode t/s, draft acceptance %, concurrency aggregate/per-stream/TTFT); Tier-A table gets short model names + a quant column. Previously SUMMARY could not display any server_ab (MTP/concurrency) output. - handoffs: ROCmFPX exploratory-bench results + next-session handoff. Records the runner blockers (per-request MTP override ignored, cumulative GPU mem leak/OOM, stale Tier-A seam), production decode numbers (27B ~29 / 35B ~76 t/s), the draft-KV experiment, the 140-t/s quant explanation (bandwidth-bound), and a model/quant-per-slot + stack-layout research task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The on-box bench pass disproved two premises this tooling shipped on (rocmfpx-bench-results-2026-07-05.md, blockers #1 and #5): - Per-request speculative.* is SILENTLY IGNORED by the ROCmFPX runner at 7aa484a — every --mode mtp cell read the launch-time config (n_max=1 and n_max=4 returned identical draft_n/accepted). --mode mtp now RELAUNCHES per (n_max,p_min) via [server].extra_args (restores after); --per-request opts back into the dead path only for a future build that honors it. - Early-EOS mismeasurement: a strict chat template (the 35B's Froggeric) emits EOS after ~1 token on a raw /completion prompt, collapsing predicted_n to 1 and making decode t/s meaningless. _completion now sends ignore_eos=True for throughput cells (off for the reuse trace, which only reads prompt timings). Also correct the runbook: finding 0.6 (restart-free sweep) marked DISPROVEN; depth axis 128k -> <=24k (128k is OOM-infeasible for 27B+MTP here, slots are ctx-bounded to 40960); §4 delta 1 updated. profile-matrix Tier B guidance notes relaunch-every-<=4-cells (runner leaks GPU mem across MTP requests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ky49xV79kYjmiYMp4uzHMw
…' into claude/llamacpp-strix-halo-flags-cyzyo8
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ky49xV79kYjmiYMp4uzHMw
This was referenced Jul 5, 2026
thinmintdev
added a commit
that referenced
this pull request
Jul 5, 2026
* release: prep v0.9.1 — bump version + CHANGELOG - pyproject 0.9.0 → 0.9.1. - CHANGELOG: promote [Unreleased] → [0.9.1] documenting the merges since v0.9.0 (ROCmFPX runner quant-detect/#1068, seed profiles/#1069, quantize skill/#1071, bench tooling/#1072, updater prepare/commit + signed notes/#1075, seed cleanup + auto-fallback/#1076, release-notes producer/#1078, continuous batching). Uses ### Highlights / ### Migrations subsections so `hal0 update` renders the seed-cleanup auto-fallback as a callout (dogfooding #1078). Fresh empty [Unreleased] restored. Not the tag itself — reviewed separately, then tag v0.9.1 triggers release.yml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * profiles: normalise seed intent labels to structural tags Drop served-model names and filler from seed 'intent' labels (e.g. 'VKFPX · ROCmFPX MoEQuality 35B-A3B · Vulkan0 + MTP' → 'VULKFPX · MOE · MTP'; 'ROCm · basic GPU LLM' → 'ROCm'). Structural runner/arch/feature tags only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * profiles: add vkfpx-dense seed (VULKFPX · DENSE · MTP, Vulkan0) Completes the FPX lane×arch matrix — Vulkan0 lane for dense ROCmFP4, for prefill-bound workloads (Vulkan wins ~+24% PP); complements the decode-optimal ROCm0 rocmfpx-rocm. mtp=True (runner-capability gate; params stay on the model). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the on-box ROCmFPX exploratory bench results and fixes the two
server_ab.pybugs the bench exposed. Combines the box session's branch (bench/rocmfpx-explore-2026-07-05) with the tooling corrections so the blockers and their fixes land together.Bench results (docs)
handoffs/rocmfpx-bench-results-2026-07-05.md— results appendix. Headline: the 35B-A3B MoE on Vulkan decodes ~2.6× faster and prefills ~3× faster than the dense 27B on ROCm (3B active vs 27B dense; decode is bandwidth-bound). Production numbers: 27B ~29 t/s / 57% accept (real task); 35B ~76 t/s / 74% accept. Seeds already match the vendor cards — the only real levers are HIP env (now shipped) and an optional 27B draft-KV tweak.handoffs/rocmfpx-bench-handoff-next-2026-07-05.md— next-session handoff (model-selection research grounded in the decode ≈ bandwidth ÷ (active_params × bytes/weight) ground-truth).Tooling fixes (
server_ab.py) — address blockers the results doc raisesspeculative.*ignored at 7aa484a.--mode mtpnow relaunches per (n_max, p_min) via[server].extra_args(restores after), instead of the silently-ignored per-request JSON.--per-requestretained as an opt-in for a future runner build that honors it._completionnow sendsignore_eos=Truefor throughput cells (the 35B's strict Froggeric template otherwise stops after ~1 token → meaningless decode t/s); off for the reuse trace.SUMMARY display
generate_results_json.pynow ingestsserver-ab/*.jsonand renders a Tier B/C section (decode t/s · draft accept % · concurrency aggregate/per-stream/TTFT) — previously it showed only Tier-A llama-bench and could display none of this bench's output. Adds short model names + aquantcolumn. (Plus a ruff UP017datetime.UTCfixup for CI.)Runbook corrections
Not in this PR (box-side, not version-controlled)
The live-slot config (venv
SEED_PROFILESimage override,/etc/hal0/slots/*.tomlHIP env +context_size 40960, thesaber-fpxprofile) stays on the box with the backups noted in the results handoff.🤖 Generated with Claude Code
https://claude.ai/code/session_01Ky49xV79kYjmiYMp4uzHMw
Generated by Claude Code