Skip to content

fix(server): sensible default max_ctx + tokenizer auto-detect + Qwen3.6 bench numbers#13

Merged
davide221 merged 1 commit into
Luce-Org:mainfrom
dusterbloom:fix/sensible-defaults-and-tokenizer-autodetect
Apr 23, 2026
Merged

fix(server): sensible default max_ctx + tokenizer auto-detect + Qwen3.6 bench numbers#13
davide221 merged 1 commit into
Luce-Org:mainfrom
dusterbloom:fix/sensible-defaults-and-tokenizer-autodetect

Conversation

@dusterbloom

Copy link
Copy Markdown
Collaborator

Three small daemon quality-of-life fixes bundled for review efficiency.

1. Default max_ctx lowered from 131072 → 16384

Related: #10 (the attention-scales-with-max_ctx perf trap).

Current default of 131072 (when DFLASH27B_KV_Q4=1) causes every short-prompt API request to pay a 20×+ slowdown, because the FA kernel strides over the full allocated context instead of the actual filled kv_len. Reproduced after pulling latest main and the daemon-mode commits (#7): a 32K prompt with --kv-q4 --max-ctx=131072 still takes >5 min of prefill vs. 38 s at --max-ctx=32768.

16384 fits typical chat / code / RAG workloads without the trap; users who genuinely need long context can still pass --max-ctx=131072 explicitly.

Help text now points at #10 so anyone surprised can follow the thread.

2. Tokenizer auto-detect

server.py previously hardcoded Qwen/Qwen3.5-27B. Now:

  • Reads general.basename / general.name from the target GGUF.
  • Maps Qwen3.5-27B → Qwen/Qwen3.5-27B, Qwen3.6-27B → Qwen/Qwen3.6-27B (both validated to share tokenizer.ggml.pre = "qwen35" + identical eos/pad/bos IDs).
  • Falls back to the historical default on any error so existing configs don't regress.
  • --tokenizer <repo_id> override added for sibling models that land in the qwen35 arch family but aren't in the registry yet.

Startup banner now prints which tokenizer was picked.

3. Qwen3.6-27B bench_llm.py numbers added to README

The Qwen3.6 27b experimental commit (b373478) added prose but limited numbers. This adds the full 3-task bench_llm.py suite on Qwen3.6-27B UD-Q4_K_XL (unsloth Dynamic 2.0, 10 prompts, n_gen=256, RTX 3090 24 GB, auto-fit --max-ctx):

Bench AR tok/s DFlash tok/s AL Speedup
HumanEval 34.90 78.16 5.94 2.24×
GSM8K 34.89 59.65 4.43 1.71×
Math500 35.13 69.77 5.15 1.99×
Mean 34.97 69.19 5.17 1.98×

For reference, the same harness on Qwen3.5-27B gave 2.87× / 2.21× / 2.56× — cross-generation drop of ~22% uniformly, driven entirely by the draft's hidden-state mismatch against 3.6. Still a clean ~2× with zero retraining.

Test plan

  • python3 -c "import ast; ast.parse(open('dflash/scripts/server.py').read())" — parses clean.
  • _tokenizer_id_from_gguf() resolves Qwen3.5-27B, Qwen3.6-27B, and bogus paths correctly (unit-tested interactively).
  • server.py --help still renders with the new flags.
  • End-to-end: DFLASH27B_KV_Q4=1 python3 scripts/server.py --target .../Qwen3.6-27B-UD-Q4_K_XL.gguf boots, serves a request via /v1/chat/completions (tested locally on RTX 3090 before this PR; banner now shows tokenizer = Qwen/Qwen3.6-27B and max_ctx = 16384).

….6 bench numbers

Three small quality-of-life fixes for the OpenAI-compatible daemon:

1. **Default max_ctx lowered from 131072 → 16384.** Attention work
   currently scales with --max-ctx rather than the actual kv_len (see
   issue Luce-Org#10), so a daemon started with --kv-q4 + the old 131072
   default burns 20×+ extra compute on every short-prompt request.
   16384 fits typical API workloads; users who actually need long
   context can still bump via --max-ctx.

2. **Tokenizer auto-detect.** Previously hardcoded to Qwen/Qwen3.5-27B.
   Reads general.basename / general.name from the target GGUF and
   picks Qwen/Qwen3.6-27B when the target file is a 3.6 checkpoint.
   Falls back to the historical default on any error so existing
   setups keep working. --tokenizer override added for
   sibling models not yet in the registry.

3. **Qwen3.6-27B bench_llm.py numbers added to README.** Full 3-task
   suite (HumanEval / GSM8K / Math500, 10 prompts, n_gen=256,
   RTX 3090) with the z-lab/Qwen3.5-27B-DFlash draft still in place:
   mean 1.98× vs AR (3.5 baseline was 2.54×). Empirically confirms
   cross-generation draft transferability: draft from 3.5 gives 69.2
   mean tok/s on 3.6-27B with zero retraining.
@davide221 davide221 merged commit a21dc00 into Luce-Org:main Apr 23, 2026
easel added a commit to easel/lucebox-hub that referenced this pull request Apr 29, 2026
…ctor)

PR Luce-Org#13 (upstream) lowered the daemon default from 131072 → 16384 to
avoid the FA-stride / VRAM-cliff trap documented in issue Luce-Org#10. The
package refactor in 1a86289 inadvertently restored 131072 when moving
the CLI into src/dflash/server/__init__.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
oliveagle pushed a commit to oliveagle/lucebox-hub that referenced this pull request May 22, 2026
…-and-tokenizer-autodetect

fix(server): sensible default max_ctx + tokenizer auto-detect + Qwen3.6 bench numbers
easel added a commit to easel/lucebox-hub that referenced this pull request May 29, 2026
…appers

- .github/workflows/{ci,docker,release-luce-bench}.yml: pin
  actions/checkout, docker/{setup-buildx,login,metadata,bake}-action,
  and astral-sh/setup-uv to immutable commit SHAs with `# vN` comments
  so the supply chain is reproducible (#4).
- harness/src/harness/clients/_common.py: replace the external `timeout`
  shell-out with `subprocess.run(..., timeout=N)`, return 124 on
  TimeoutExpired to match GNU timeout's exit code (#5).
- scripts/build_image.sh: normalize REGISTRY to end in `/` instead of
  silently producing `ghcr.io/luce-orglucebox-hub` when the trailing
  slash is missing (#6).
- harness/src/harness/clients/pi.py: non-interactive launch now mirrors
  run_pi.sh's validated invocation (--provider, --print, --mode json,
  --tools, --no-session, --offline) and sets PI_CODING_AGENT_DIR /
  PI_CODING_AGENT_SESSION_DIR / PI_OFFLINE (Luce-Org#7).
- docker-bake.hcl: sanitize `+` → `-` in VERSION before composing tags,
  since `+` is not a valid Docker tag character (Luce-Org#8).
- harness/src/harness/clients/hermes.py: set HERMES_HOME + the rest of
  run_hermes.sh's env wiring and call `chat --provider --model
  --accept-hooks --yolo --max-turns --source --query` instead of a bare
  positional prompt (Luce-Org#9, Luce-Org#10).
- harness/src/harness/clients/openclaw.py: apply the OpenClaw config
  patch via `openclaw config patch --file` before the run, and call
  `agent --local --json --model lucebox/<model> --session-id --timeout
  --message` instead of a bare positional prompt (Luce-Org#11).
- pyproject.toml: drop the dead dflash/scripts/{prefix_cache,test_server,
  tool_memory}.py ruff include pins (those paths were renamed during
  the dflash→server rename and then deleted upstream) (Luce-Org#12).
- lefthook.yml: widen the shellcheck/bash-parse glob from `*.sh` to
  `**/*.sh` so scripts under nested dirs (harness/clients/*.sh,
  scripts/*.sh, server/scripts/*.sh) are linted on commit (Luce-Org#13).

Addresses cubic #4Luce-Org#13 (P2) on PR Luce-Org#285. Luce-Org#14 was already addressed in
the previous commit alongside the LUCEBENCH_THINK default fix.
davide221 added a commit that referenced this pull request Jun 7, 2026
The origin/main merge resolved the server/deps/llama.cpp gitlink to the PR's
older b896cf69 (llama #12 hip-shim), rewinding it from main's 570d9785 and
dropping llama #13 (crash), #14 (pflash-sparse stream-sync / BSA fix) and
#15 (fattn-sparse hip cudaStreamDefault). Point the gitlink back to 570d9785
(a strict superset — b896cf69 is its ancestor) so the PR no longer downgrades
llama.cpp / drops the BSA fix.

Rebuilt + smoke-tested against 570d9785: sg_b100 96/96 bit-exact vs all-GPU,
offload 60% 99.3 tok/s, all-GPU 119.5.

Co-Authored-By: WOZCODE <contact@withwoz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants