fix(scripts): auto-fit --max-ctx to prompt size#11
Merged
Conversation
Previously neither script passed --max-ctx to test_dflash, so users testing long-context from the README example (which showcases --max-ctx=131072 with --kv-q4) would routinely oversize the context relative to their actual prompt. Measured on RTX 3090 with Qwen3.5-27B Q4_K_M target + BF16 draft: | Prompt | KV | max_ctx | Prefill | Gen tok/s | |--------|------|---------|---------|-----------| | 32K | Q8_0 | 32768 | 38 s | 42.0 | | 32K | Q4_0 | 131072 | 1035 s | 1.10 | | 64K | Q4_0 | 131072 | 1014 s | 3.97 | | 64K | Q4_0 | 66048 | 126 s | 18.35 | Same prompt, same KV type, same binary — only --max-ctx differs. Output quality is unaffected (last_tok bit-exact); this is a strict perf regression from oversized attention strides. The FA kernel scales with align_up(max_ctx, FATTN_KQ_STRIDE=256), not with the filled kv_len, so allocating 131K for a 32K prompt burns ~100K extra keys of dequant+score per head per step per layer. Auto-fit formula: align_up(prompt + n_gen + 64, 256) — the 64-token pad covers q_len=16 + ddtree budget up to 22 with margin. Users who genuinely want a larger KV budget can pass --max-ctx=N explicitly in run.py; bench_llm.py always auto-fits since it's per-prompt. See Luce-Org#10 for the full repro and root cause.
This was referenced May 13, 2026
oliveagle
pushed a commit
to oliveagle/lucebox-hub
that referenced
this pull request
May 22, 2026
fix(scripts): auto-fit --max-ctx to prompt size
easel
added a commit
to easel/lucebox-hub
that referenced
this pull request
May 29, 2026
…appers
- .github/workflows/{ci,docker,release-luce-bench}.yml: pin
actions/checkout, docker/{setup-buildx,login,metadata,bake}-action,
and astral-sh/setup-uv to immutable commit SHAs with `# vN` comments
so the supply chain is reproducible (#4).
- harness/src/harness/clients/_common.py: replace the external `timeout`
shell-out with `subprocess.run(..., timeout=N)`, return 124 on
TimeoutExpired to match GNU timeout's exit code (#5).
- scripts/build_image.sh: normalize REGISTRY to end in `/` instead of
silently producing `ghcr.io/luce-orglucebox-hub` when the trailing
slash is missing (#6).
- harness/src/harness/clients/pi.py: non-interactive launch now mirrors
run_pi.sh's validated invocation (--provider, --print, --mode json,
--tools, --no-session, --offline) and sets PI_CODING_AGENT_DIR /
PI_CODING_AGENT_SESSION_DIR / PI_OFFLINE (Luce-Org#7).
- docker-bake.hcl: sanitize `+` → `-` in VERSION before composing tags,
since `+` is not a valid Docker tag character (Luce-Org#8).
- harness/src/harness/clients/hermes.py: set HERMES_HOME + the rest of
run_hermes.sh's env wiring and call `chat --provider --model
--accept-hooks --yolo --max-turns --source --query` instead of a bare
positional prompt (Luce-Org#9, Luce-Org#10).
- harness/src/harness/clients/openclaw.py: apply the OpenClaw config
patch via `openclaw config patch --file` before the run, and call
`agent --local --json --model lucebox/<model> --session-id --timeout
--message` instead of a bare positional prompt (Luce-Org#11).
- pyproject.toml: drop the dead dflash/scripts/{prefix_cache,test_server,
tool_memory}.py ruff include pins (those paths were renamed during
the dflash→server rename and then deleted upstream) (Luce-Org#12).
- lefthook.yml: widen the shellcheck/bash-parse glob from `*.sh` to
`**/*.sh` so scripts under nested dirs (harness/clients/*.sh,
scripts/*.sh, server/scripts/*.sh) are linted on commit (Luce-Org#13).
Addresses cubic #4–Luce-Org#13 (P2) on PR Luce-Org#285. Luce-Org#14 was already addressed in
the previous commit alongside the LUCEBENCH_THINK default fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10.
Summary
scripts/run.pyandscripts/bench_llm.pydidn't pass--max-ctxtotest_dflash, so users testing long-context following the README's--max-ctx=131072example would oversize the KV budget relative to their actual prompt. That's a 27× prefill slowdown at 32K with--kv-q4, because the FA kernel strides over unused KV.This patch auto-fits
--max-ctx = align_up(prompt + n_gen + 64, 256)by default. The 64-token pad coversq_len=16plus DDTree budget up to 22 with margin. Users who want a larger budget can still pass--max-ctx=Nexplicitly inrun.py;bench_llm.pyalways auto-fits since it runs per-prompt.Measurements (RTX 3090, Qwen3.5-27B Q4_K_M target + BF16 draft)
--max-ctxOutput quality unaffected (
last_tokbit-exact on all paths). This is strictly a perf fix.Test plan
python3 scripts/run.py --prompt "def fibonacci(n):" --draft models/draft/model.safetensors— short prompt, printsmax_ctx=512, output bit-exact.--kv-q4: prefill 126 s vs 1014 s before.python3 scripts/bench_llm.pyfull suite (HumanEval / GSM8K / Math500) — expected to complete at or above README speeds, since per-prompt max_ctx is now tight.