fix(server): sensible default max_ctx + tokenizer auto-detect + Qwen3.6 bench numbers#13
Merged
davide221 merged 1 commit intoApr 23, 2026
Conversation
….6 bench numbers Three small quality-of-life fixes for the OpenAI-compatible daemon: 1. **Default max_ctx lowered from 131072 → 16384.** Attention work currently scales with --max-ctx rather than the actual kv_len (see issue Luce-Org#10), so a daemon started with --kv-q4 + the old 131072 default burns 20×+ extra compute on every short-prompt request. 16384 fits typical API workloads; users who actually need long context can still bump via --max-ctx. 2. **Tokenizer auto-detect.** Previously hardcoded to Qwen/Qwen3.5-27B. Reads general.basename / general.name from the target GGUF and picks Qwen/Qwen3.6-27B when the target file is a 3.6 checkpoint. Falls back to the historical default on any error so existing setups keep working. --tokenizer override added for sibling models not yet in the registry. 3. **Qwen3.6-27B bench_llm.py numbers added to README.** Full 3-task suite (HumanEval / GSM8K / Math500, 10 prompts, n_gen=256, RTX 3090) with the z-lab/Qwen3.5-27B-DFlash draft still in place: mean 1.98× vs AR (3.5 baseline was 2.54×). Empirically confirms cross-generation draft transferability: draft from 3.5 gives 69.2 mean tok/s on 3.6-27B with zero retraining.
easel
added a commit
to easel/lucebox-hub
that referenced
this pull request
Apr 29, 2026
…ctor) PR Luce-Org#13 (upstream) lowered the daemon default from 131072 → 16384 to avoid the FA-stride / VRAM-cliff trap documented in issue Luce-Org#10. The package refactor in 1a86289 inadvertently restored 131072 when moving the CLI into src/dflash/server/__init__.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 13, 2026
oliveagle
pushed a commit
to oliveagle/lucebox-hub
that referenced
this pull request
May 22, 2026
…-and-tokenizer-autodetect fix(server): sensible default max_ctx + tokenizer auto-detect + Qwen3.6 bench numbers
easel
added a commit
to easel/lucebox-hub
that referenced
this pull request
May 29, 2026
…appers
- .github/workflows/{ci,docker,release-luce-bench}.yml: pin
actions/checkout, docker/{setup-buildx,login,metadata,bake}-action,
and astral-sh/setup-uv to immutable commit SHAs with `# vN` comments
so the supply chain is reproducible (#4).
- harness/src/harness/clients/_common.py: replace the external `timeout`
shell-out with `subprocess.run(..., timeout=N)`, return 124 on
TimeoutExpired to match GNU timeout's exit code (#5).
- scripts/build_image.sh: normalize REGISTRY to end in `/` instead of
silently producing `ghcr.io/luce-orglucebox-hub` when the trailing
slash is missing (#6).
- harness/src/harness/clients/pi.py: non-interactive launch now mirrors
run_pi.sh's validated invocation (--provider, --print, --mode json,
--tools, --no-session, --offline) and sets PI_CODING_AGENT_DIR /
PI_CODING_AGENT_SESSION_DIR / PI_OFFLINE (Luce-Org#7).
- docker-bake.hcl: sanitize `+` → `-` in VERSION before composing tags,
since `+` is not a valid Docker tag character (Luce-Org#8).
- harness/src/harness/clients/hermes.py: set HERMES_HOME + the rest of
run_hermes.sh's env wiring and call `chat --provider --model
--accept-hooks --yolo --max-turns --source --query` instead of a bare
positional prompt (Luce-Org#9, Luce-Org#10).
- harness/src/harness/clients/openclaw.py: apply the OpenClaw config
patch via `openclaw config patch --file` before the run, and call
`agent --local --json --model lucebox/<model> --session-id --timeout
--message` instead of a bare positional prompt (Luce-Org#11).
- pyproject.toml: drop the dead dflash/scripts/{prefix_cache,test_server,
tool_memory}.py ruff include pins (those paths were renamed during
the dflash→server rename and then deleted upstream) (Luce-Org#12).
- lefthook.yml: widen the shellcheck/bash-parse glob from `*.sh` to
`**/*.sh` so scripts under nested dirs (harness/clients/*.sh,
scripts/*.sh, server/scripts/*.sh) are linted on commit (Luce-Org#13).
Addresses cubic #4–Luce-Org#13 (P2) on PR Luce-Org#285. Luce-Org#14 was already addressed in
the previous commit alongside the LUCEBENCH_THINK default fix.
davide221
added a commit
that referenced
this pull request
Jun 7, 2026
The origin/main merge resolved the server/deps/llama.cpp gitlink to the PR's older b896cf69 (llama #12 hip-shim), rewinding it from main's 570d9785 and dropping llama #13 (crash), #14 (pflash-sparse stream-sync / BSA fix) and #15 (fattn-sparse hip cudaStreamDefault). Point the gitlink back to 570d9785 (a strict superset — b896cf69 is its ancestor) so the PR no longer downgrades llama.cpp / drops the BSA fix. Rebuilt + smoke-tested against 570d9785: sg_b100 96/96 bit-exact vs all-GPU, offload 60% 99.3 tok/s, all-GPU 119.5. Co-Authored-By: WOZCODE <contact@withwoz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small daemon quality-of-life fixes bundled for review efficiency.
1. Default
max_ctxlowered from 131072 → 16384Related: #10 (the attention-scales-with-max_ctx perf trap).
Current default of
131072(whenDFLASH27B_KV_Q4=1) causes every short-prompt API request to pay a 20×+ slowdown, because the FA kernel strides over the full allocated context instead of the actual filledkv_len. Reproduced after pulling latest main and the daemon-mode commits (#7): a 32K prompt with--kv-q4 --max-ctx=131072still takes >5 min of prefill vs. 38 s at--max-ctx=32768.16384 fits typical chat / code / RAG workloads without the trap; users who genuinely need long context can still pass
--max-ctx=131072explicitly.Help text now points at #10 so anyone surprised can follow the thread.
2. Tokenizer auto-detect
server.pypreviously hardcodedQwen/Qwen3.5-27B. Now:general.basename/general.namefrom the target GGUF.Qwen/Qwen3.5-27B, Qwen3.6-27B →Qwen/Qwen3.6-27B(both validated to sharetokenizer.ggml.pre = "qwen35"+ identical eos/pad/bos IDs).--tokenizer <repo_id>override added for sibling models that land in the qwen35 arch family but aren't in the registry yet.Startup banner now prints which tokenizer was picked.
3. Qwen3.6-27B
bench_llm.pynumbers added to READMEThe
Qwen3.6 27b experimentalcommit (b373478) added prose but limited numbers. This adds the full 3-taskbench_llm.pysuite on Qwen3.6-27B UD-Q4_K_XL (unsloth Dynamic 2.0, 10 prompts, n_gen=256, RTX 3090 24 GB, auto-fit--max-ctx):For reference, the same harness on Qwen3.5-27B gave 2.87× / 2.21× / 2.56× — cross-generation drop of ~22% uniformly, driven entirely by the draft's hidden-state mismatch against 3.6. Still a clean ~2× with zero retraining.
Test plan
python3 -c "import ast; ast.parse(open('dflash/scripts/server.py').read())"— parses clean._tokenizer_id_from_gguf()resolves Qwen3.5-27B, Qwen3.6-27B, and bogus paths correctly (unit-tested interactively).server.py --helpstill renders with the new flags.DFLASH27B_KV_Q4=1 python3 scripts/server.py --target .../Qwen3.6-27B-UD-Q4_K_XL.ggufboots, serves a request via/v1/chat/completions(tested locally on RTX 3090 before this PR; banner now showstokenizer = Qwen/Qwen3.6-27Bandmax_ctx = 16384).