Skip to content

v0.1.8-alpha — DFlash prompt-shape +26.7% (inspired by Lucebox)

Choose a tag to compare

View all tags
@Kaden-Schutt Kaden-Schutt released this 25 Apr 08:09
v0.1.8-alpha
c03dfa3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.1.8-alpha (2026-04-25)

Major perf cycle on the DFlash branch. Headline: Phase 1 prompt-shape
adaptation lifts 27B-3.5 DFlash by +26.7% on PEP-8-style code prompts
(median 161 → 199 tok/s, τ 8.07 → 10.36). Plain DFlash 10-prompt
HumanEval mean at n_gen=256 sits at 146.9 tok/s — alongside Lucebox's
ggml/CUDA published numbers on parallel hardware (RTX 3090). DFlash
work this cycle was substantially inspired by Lucebox
— credit to Davide Ciffa for published targets, n_gen-aware bench
methodology, and pointers at perf opportunities.

Features

  • Prompt-shape adaptation (Phase 1): env-gated \n{3,} → \n\n
    collapse before tokenize. Eliminates rare BPE token 1358 (\n\n\n)
    in favor of HOT token 271 (\n\n) on Qwen3.5/3.6 vocab. Default
    OFF — opt-in via prompt_normalize=true (config TUI / CLI) or
    HIPFIRE_NORMALIZE_PROMPT=1. Impl: engine::tokenizer::maybe_normalize_prompt,
    10 unit tests, wired into 4 entry points (dflash_spec_demo, daemon,
    run, triattn_infer).
  • Token heat diagnostic (Phase 2): HIPFIRE_PROMPT_TOKEN_HEAT=1
    triggers per-position BPE merge-rank heat dump at every encode site.
    HIPFIRE_PROMPT_HEAT_JSON=1 for machine-readable JSON to stdout.
    Standalone tool crates/engine/examples/encode_prompt.rs.
  • EOT-stop fix: Tokenizer::is_terminator(id) + eot_id field.
    Daemon, run, dflash_spec_demo now stop on <|endoftext|> too, not
    just <|im_end|>. Fibonacci attractor loop in raw-text DFlash dead.
  • DFlash drafts on HuggingFace: pullable via hipfire pull —
    qwen3.5:9b-draft, qwen3.5:27b-draft, qwen3.6:27b-draft. Plus
    qwen3.6:27b (15 GB target) added to registry. Files land at
    ~/.hipfire/models/<canonical-name> matching daemon auto-discovery.
  • DDTree task #101 (tree-aware LA kernel): conv1d_silu_split_tree.hip
    • gated_delta_net_q8_tree.hip shipped with parent_indices plumbing
      through spec_step_ddtree_batched. Default ON post-validation.
  • Adaptive-b continuous scheduler: with hysteresis + B_MAX clamp to
    draft-trained block size. Restores dflash_spec_demo's adaptive
    behavior in hipfire serve.

Tools / harnesses

  • scripts/sweep_dflash_full.sh — unified 3 model × 2 mode × 3 genre
    bench harness (post-EOT-fix).
  • scripts/bench_humaneval_dflash.py — HumanEval bench (Lucebox parity
    methodology, n_gen=256).
  • benchmarks/prompts/lru_cache_pep8_strict.txt + lru_cache_single_blank.txt
    — md5-locked canonical bench prompts (CLAUDE.md mandate).
  • crates/hipfire-quantize/src/bin/draft_to_mq4.rs — draft requantizer
    (+7.35% tok/s on 27B draft path).
  • coherence-gate-dflash.sh is now the canonical correctness gate
    (replaces deprecated quality-gate.sh byte-exact baselines).

Kernel work

25 kernel files touched. Major:

  • WMMA fast paths: gemm_hfq4g256_residual_wmma_{k2,k2x32,ksplit}.hip,
    gemm_qkv_hfq4g256_wmma.hip, gemm_qkvza_hfq4g256_wmma.hip,
    gemm_gate_up_hfq4g256_wmma.hip, gemm_mw16_residual_wmma.hip
    (tasks #73-#86): wo_residual 41% → grid-starvation fixed; gate_up
    50% → 75% BW; qkvza 46% → 70% BW; lm_head 32-rows/block.
  • Tree-aware: conv1d_silu_split_tree.hip, gated_delta_net_q8_tree.hip.
  • Fused: fused_rmsnorm_mq_rotate.hip, fused_qk_l2_norm_scale.hip,
    fused_sigmoid_alpha_gate.hip.
  • Batched: embedding_hfq4g256_batched.hip, embedding_q8_batched.hip,
    kv_cache_write_q8_0_batched.hip, rope_partial_interleaved_batched.hip.
  • 55+ hot-path hipMemset → hipMemsetAsync migrations (task #88).
  • Async hipMemcpyHtoD for per-cycle uploads (task #89).
  • wave_reduce intrinsic sweep (task #81).

CLI

  • prompt_normalize: boolean config field — TUI toggle, per-model
    overridable, env-propagated through applyConfigEnv.
  • Bonus: hipfire config set <bool-key> true|false parser fixed —
    was rejecting strings; now coerces "true"/"false" → bool. Incidentally
    fixes pre-existing dflash_adaptive_b, cask, experimental_budget_alert
    set paths.
  • New aliases: qwen3.5-9b:draft, qwen3.5:9b:draft → qwen3.5:9b-draft.

Docs

  • AGENTS.md (new) — tester-focused playbook complementing CLAUDE.md
    rules. Setup, smoke tests, gotchas table, flag reference.
  • CLAUDE.md: "Prompt-structure τ sensitivity" hard rule (one newline
    = 17% τ swing) + "DFlash Coherence Gate" section.
  • docs/methodology/perf-benchmarking.md: prompt-shape section + null-
    result log of attempts that looked like wins one-shell A/B but
    measured no-op on fresh probe.
  • New PRDs: prompt-shape-adaptation.prd, task-93-path-c-trained-draft.prd,
    task-93-path-d-stale-context.prd, humaneval-2026-04-24-beats-3090.prd,
    prompt-structure-tau-discovery-2026-04-24.prd. Plus DDTree post-mortems.

Performance numbers (RX 7900 XTX, gfx1100, asym3 KV)

DFlash by genre (5-run medians, max=120, --no-chatml, normalize ON):

Model code (HE/53) code peak prose (Rome) instruct (sky)
27B-3.5 196.8 (4.45× AR) 218.6 49.6 (1.13×) 44.7 (~tie)
27B-3.6 185.5 (4.19× AR) 186.0 — —
9B-3.5 329.1 (2.65× AR) 346.7 99.4 (0.79× ✗) 246.9 (1.99×)

DFlash is genre-conditional — net loss on 9B prose (-20%) because
draft-target argmax disagreement at high-entropy text. dflash_mode=auto
default handles this automatically for known-loss configs.

Lucebox parity (10-prompt HumanEval @ n_gen=256, plain DFlash linear):
146.9 tok/s mean — alongside Lucebox's published 112.82 (Chain DFlash)
and 135.80 (best DDTree b22 f16) on RTX 3090.

Roadmap (post-0.1.8)

GitHub issues:

Notes

  • Master fixes merged into dflash for release: vision encoder F32 dequant
    fixes (#28), Qwen3.6-A3B HF upload (#34), batched-prefill determinism
    fixes (PR #28 follow-ups in master).
  • Legacy byte-exact quality-gate.sh baselines are stale by design —
    branch-local artifacts, not the canonical gate. Coherence-gate-dflash
    is the gate going forward.
  • DDTree on gfx1100 still has a structural RoPE phase-delta skew bug
    (commit 39aa358). Linear DFlash is the production path until issue
    #41 lands.
Assets 2
Loading