dasLLAMA: K-quant kernel arc — kq v2 integer folds, kq v3 panel-scratch tiles, chain regrain, parity gates by borisbat · Pull Request #3450 · GaijinEntertainment/daScript

borisbat · 2026-07-13T02:52:41Z

What

The K-quant kernel arc: native Q4_K/Q5_K/Q6_K matmuls move to llama.cpp's integer-fold scheme (kq v2), the k5/k6 tiles gain byte-expanded panel scratch (kq v3), K-quant layers are re-admitted to the fused decode chains, and token-level parity gates now guard every kq rail. dasLLAMA-only (plus two one-line AOT infrastructure fixes below).

Results (Qwen3-4B, steady-state median vs llama.cpp CPU)

M1 Max (8T):

pp512	das before	das after	lcpp	ratio
Q4_K_M	151–163	~155–158	172	0.90×
Q5_K_M	110	~150	131	1.15×
Q6_K	111	~141	138.5	1.02×

tg64 at lcpp parity on all three (Q6 +2.1% from the chain regrain).

zen2 / Threadripper 3990X (16 pinned cores both sides):

pp512	das	lcpp	ratio
Q4_K_M	161–180	168.3	~parity
Q5_K_M	~159	87.0	1.84×
Q6_K	~129	98.0	1.32×

tg64 at lcpp parity all three. (lcpp has no AVX2 Q5/Q6 repack; Q4 races their one AVX2 K-quant repack GEMM.)

The pieces

kq v2 (all three formats): integer sub-scale/bsums folds against Q8_K-form activations — per-256 scale + per-16 sums, one float fold per superblock (the lcpp scheme; per-32 Q8_0 activation scales were the v1 mistake). k5 reuses k4's scale-block decode wholesale (byte-identical block form); k6 folds native SIGNED per-16 sub-scales exactly in int32. Dual-form plumbing deleted (net −180 lines).
kq v3 panel scratch: k5/k6 tiles read a byte-expanded weight panel (one byte per weight, zero unpack ALU) that the batch cell unpacks from the packed grp planes once per (group, token-block) — SWAR uint64 deposit, amortized ×TB instead of the tile's ×4. DRAM planes keep the packed 160/192 B/superblock form: the measured alternative (byte-expanded planes everywhere) won the same prefill but cost tg −30%/−20% on Q5/Q6 — decode is model-level DRAM-bound, so plane bytes are decode time.
Chain regrain: the fused decode chains run one activation image per consumed form (Q8_0 for q8 weights, Q8_K for kq); the w2-input requant moves to 256-row stage-0 groups and the wo-input requant to head groups of 256/head_size heads, so every Q8_K quantize covers whole superblocks. Bit-exact vs the per-op path (gated).
Per-format tune families: k4/k5/k6 each own their [tune] grid + gemv/layout companions, so each format's planes take their own winning interleave.
Parity gates (the test gap behind the embed_row desync): token-level arms for Qwen3-4B Q4_K_M/Q5_K_M/Q6_K, qwen3moe-30B Q4_K_M (both prefill paths), and gemma-2-2b Q4_K_M (the softcap+kq rail); a multi-row GEMV stride gate; a fused-vs-per-op kq bit-exactness arm.
3 latent stride bugs fixed (one class: sites missed in the K4_SSB 32→20 move), each with a failing test first: portable k4_rows_kernel scale-row stride, kq_kernel_gen row-tail stride, and cls_softcap_chain feeding kq rows cores the wrong activation form.
Measurement: the M1 prefill "bimodality" was diagnosed via lane-timeline traces as a package-DVFS boost window (all lanes 98–99% utilized in both modes, uniform ×1.113 per-chunk) — benches now report the steady-state median and discard the first-after-idle rep. The zen2 race ran 16 affinity-pinned cores on both sides.

Drive-by fixes (both caught by the preflight full-AOT gate)

dasllama_arch_qwen35.das (dasLLAMA: Qwen 3.5/3.6 support (qwen35/qwen35moe — Gated DeltaNet hybrid) #3448) was missing from the AOT lib list in tests/aot/CMakeLists.txt — every dasLLAMA test failed the full AOT sweep with error[50101] on qwen35 symbols (invisible to per-PR CI, which only builds the tests/language subset; the nightly would have gone red). One-line registration.
utils/aot/main.das (the das AOT driver the cmake rules actually invoke) never set tune_frozen — only the C++ -aot path got the policy in dasLLAMA: size-tiered allocation alignment, tune-rail hardening, M1/zen2 re-baseline #3423. Generation therefore ran in the stamped world while the AOT-consuming runtime compiles gated to the reference tier, desyncing the recorded hashes for any function with a [tune] family in its static call graph. Latent since dasLLAMA: size-tiered allocation alignment, tune-rail hardening, M1/zen2 re-baseline #3423; first exposed by this PR's kq tile gates (the first test-file code calling [tune] functions directly). One line: cop.tune_frozen = true.

Ledgered follow-ups (API_REWORK.md)

zen2 Q6 pp −4.6% before→after (arc otherwise neutral on the maddubs lattice; candidate per-ISA kqBytes gate).
kq tune bench MoE-shaped cell row; k6 fold-structure headroom; k5 NEON tbl/cmtst unpack kernel.

Testing

preflight --full green: 16 passed, 0 failed (format, lint, docs, interp/JIT/full-AOT suites, sequence smoke; full AOT 12400 tests / 0 failed / 0 errors).
dasLLAMA suite 428 tests / 0 failed under -jit (test_kv_paged 2 errors = known master red dasLLAMA: paged KV + q8_0 KV codec + continuous-batching server with prefix caching #3425/dasLLAMA: native K-quant tier (Q4_K/Q5_K/Q6_K) + scheduler audit #3444).
Token parity 50 tests / 0 failed; fused-vs-per-op kq arm ids+logits EXACT; gen_tune_probe grid green on M1 (sdot) and zen2 (maddubs mr8 — first x64-silicon run of the byte panels).

🤖 Generated with Claude Code

Synthetic disk superblocks -> transcode -> per-variant grp<mr> repack, gated against the scalar kq_grp_row_dot oracle in test mode; tune mode adds per-format tile / gemv-stream / gemv-hot tables (diagnostic — the manifest still crowns one q8q8_tile_gen entry). kq tolerance gates against the fixture magnitude: k6's signed sub-scales cancel within a row, so element-relative tolerance trips on fold-order noise at production shapes. First M1 tables say the q8 crown is not kq-optimal: k5 tile trails the kq-best row x1.30, k6 tile x1.16, k4 tile x1.07 (nrsplit2 doubles the weight unpack in the 4-token tile), while k4/k6 gemv prefer the crowned mr8 planes. k5 is compute-bound everywhere (20 GB/s weight stream vs k4's 42.5) — the high-bit unpack is the next lever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The kq kernels were tune_companions riding the q8q8_tile_gen crown, and the M1 iso bench showed that crown (grp8+nrsplit2) is wrong for the kq tiles: nrsplit token-splits the 4-token tile and re-runs the expensive superblock unpack per slice (k5 tile -30%, k6 -16%, k4 -7% vs the best in-grid row). k4/k5/k6q8_tile_gen are now their own [tune] families — per-format grids over (mr, nrsplit) + the x64 dot tiers, gemv + a NEW per-format layout companion per family, one manifest entry per format — so each format's planes take their own interleave. Loader/dispatch read kq_layout_of(fmt); test_kquant's two layout-coupled gates follow. gen_tune_probe gates and benches each family's own grid (tile-best wins the entry; the kq gemv is nrsplit-independent, same-mr rows share its plane). Fallback chains reproduce today's manifest-less behavior on x64 (maddubs mr8 on avx2, busd on vnni tiers — byte-identical codegen to the old ride-along on zen2) and drop the never-hardware-tested bias128+kq combination; arm64 lands on the measured mr4. M1 e2e (Qwen3-4B, 8T, vs llama.cpp build-cpu): pp512 Q4_K_M 127->140 tok/s (0.88x lcpp), Q5_K_M 80->100 (0.77x), Q6_K 78->96 (0.72x); tg128 flat everywhere (decode stays DRAM-bound, as the split's design argued); qwen3moe-30B Q4_K_M pp -3% (expert cells mildly prefer grp8 — ledgered with the kq emitter headroom item). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… at parity Adopts llama.cpp's proven K-quant kernel design for the k4 family (studied against their ARM 8x4 repack GEMMs, which cover Q4_K/Q5_K/Q6_K — the earlier 'no k5/k6 repack on their side' note was stale): - Q8_K-form activations for k4 matmuls: ONE fp32 scale per 256-superblock + per-16 bsums (quantize_q8_k_into + requant_rows_q8k_bs). k5/k6 keep the Q8_0 form until they port; mixed models run dual-form at the shared seams (mm_pre_f k-plane params, per-site k-fills, MoE gathers off a K-form batch image, form-branched hidden requants, mixed-pair guards). - k4 scale planes go verbatim-disk (f16 d/dmin + 12B 6-bit packing + 4B pad, 20B/superblock vs v1's 32B f16 pairs); the grp repack decodes the packing once into u8 sc/mn rows IN PLACE. Plane dequant is now bit-identical to the disk dequant. - emit_block_k4v2: integer sub-scale x int32-dot and bsums x mins accumulation per superblock, one d*d8 fma + dmin*d8 fmls per (token, row-quad) — replaces two float fmas plus f16 pair loads per 32-block. Lattice-generic (the x64 maddubs/busd kq rows emit the same scheme). Iso M1: k4 tile 84->97.7 GMAC/s (lcpp-effective ~98), gemv 56->65; scalar reference 2.4->8.7. - fused decode chains gate k4 layers out (their mid-chain requants slice at 32/head grain; Q8_K needs whole superblocks) — regrain ledgered; tg still improved on the per-op path. - FIX (also latent in the per-format-tune commit): embed_row gathered kq embedding rows at the q8 family's interleave — garbage generation on kq models. The Model now freezes per-format kq_repack_mr{4,5,6} from the new KernelBackend.kq_layout slot; all kq models generate coherently again. (Benches measure speed, not tokens — a kq-model generation test is a gap.) M1 e2e vs llama.cpp build-cpu (8T): Qwen3-4B Q4_K_M pp512 158.0 vs 158.2 (1.00x, was 0.80x pre-arc) tg128 37.7 vs 38.4 (0.98x, up from 36.5); qwen3moe-30B Q4_K_M pp512 176.5 vs 131.6 (1.34x) tg 44.1 vs 43.0. Q5_K_M / Q6_K unchanged (their v2 port + the zen2 race are the ledgered next steps). Gates: gen_tune_probe test/tune green, test_kquant 40/40 (fp64 reference and fixtures follow the k4 form), dasLLAMA suite 397/412 with zero failures from this change (test_kv_paged's 2 errors are a pre-existing master red). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…embed_row desync) Four new arms in test_parity.das: Qwen3-4B Q4_K_M / Q5_K_M / Q6_K (fast tier) and Qwen3-30B-A3B-Instruct-2507 Q4_K_M (large tier, both MoE prefill paths), all forced through the native kq planes (kquant_native pinned around the load). Oracles produced per-file by simple_ids; all four continue the counting fixture identically to the frozen QWEN3_GEN ids. Benches measure speed and test_kquant measures kernels on synthetic rows — the 2026-07-12 embed_row mr-desync generated garbage on every kq model and nothing committed caught it. These arms make that class of bug loud, and gate the upcoming k5/k6 kq-v2 port token-for-token. All four verified green (46/46 with the rest of the parity file; MoE arm under DASLLAMA_PARITY_FULL=1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…v2 plane change) The kq v2 commit moved K4_SSB to 20 but k4_rows_kernel — the portable disk-order GEMV rail a kq-tagged model runs when the backend has no kq stamps — kept striding scale rows at nsb*32: row 0 dotted clean, every later row read the wrong scale rows. Found scoping the k5/k6 port; none of the committed gates exercised multi-row portable GEMVs. Test first: new test_kq_gemv_rows in test_kquant.das gates matmul_kq (all three formats, 8 rows, per-row LCG-perturbed planes) bit-exact against per-row disk dots — k4 arm red before the fix, 44/44-suite green after; k5/k6 arms guard their strides through the coming port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…al-form plumbing deleted Finishes what the k4 rework proved (ledger entry): all three kq families now run kq v2 — Q8_K-form activations (per-256 scale + per-16 bsums), verbatim disk scale planes, integer sub-scale/bsums folds, ONE float fold per superblock per (token, row-quad): - k5 takes the k4 plane treatment verbatim: the Q5_K scale block IS the Q4_K scale block, so transcode keeps the 16B disk block + 4B pad (K5_SSB 32->20), the grp repack decodes the 6-bit packing into u8 sc/mn rows in place, and the emitter/scalar folds are byte-identical to k4's — only the high-bit OR in the quant walk differs (kept from v1, flush cadence unchanged). - k6 planes stay as they were (already verbatim, 18B): the fold goes integer — SIGNED per-16 sub-scales sext'd, iacc += sc0·lo + sc1·hi, bacc += sc0·Sa16lo + sc1·Sa16hi, closed as float(iacc - 32·bacc)·(d·d8) (exact: |iacc| < 2^28, |32·bacc| < 2^28). Replaces two float fmas + f16 widens per 32-block. - emit_block_k4v2 + the v1 emit_block_kq body unify into emit_block_kqv2; the kq gemv/tile impls and every scalar/grp/batch walk read per-256 activation scale rows for all three formats. - The dual-form activation plumbing DIES: mm_pre_f loses its Q8_0 kq side, mm_kq/mm_b_kq always quantize Q8_K, the MoE gate/up hoists fill exactly one image (Q8_K for kq pairs — the dead Q8_0 twin image is gone, halving the 30B MoE parity arm's wall time), moe_pair_kform reduces to the q8/kq mix guard, and the orphaned moe_xbs scratch is deleted. Net -180 lines. - Fused decode chains now gate ALL kq layers out (k5/k6 previously rode them on Q8_0-form; feeding the v2 kernels that form would miscompute) — per-op path until the ledgered regrain. Two more latent k4-rework bugs found and fixed en route (same class as the k4_rows_kernel stride miss): - kq_kernel_gen's row tail (gen backend, d % mr rows) still strode k4 scale rows at 32B over the 20B plane. - cls_softcap_chain (the fused Gemma-class softcap classifier) fed the stamped kq rows cores Q8_0-form activations. No fixture reached that rail — Qwen3 has no softcap — so a NEW gate exists: gemma-2-2b-it Q4_K_M (local llama-quantize requant of the on-disk Q8_0), oracled with simple_ids, token-for-token through the kq planes + softcap classifier. Gates: test_kquant 44/44 (all fixtures follow the Q8_K form; the k4/k5 plane gates now assert bit-exactness vs the FILE dequant — the old f16-rounded expectation only passed by numerical coincidence); gen_tune_probe test green (every k5/k6 perm vs the scalar oracle); kq parity arms token-for-token: Qwen3-4B Q4_K_M / Q5_K_M / Q6_K, gemma-2 Q4_K_M (softcap), qwen3moe-30B Q4_K_M (both prefill paths, DASLLAMA_PARITY_FULL); full dasLLAMA suite 426 tests / 410 passed / 0 failed (test_kv_paged's 2 errors are the known pre-existing master red). Q5/Q6 e2e vs lcpp re-race is next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e-unpack lever Interleaved A/B on M1 8T (Parsec running — ratios only): Q5_K_M pp 0.83x lcpp (was 0.77x), Q6_K pp 0.80x (was 0.72x), tg at parity for both off the per-op path (the v2 gemv covered the chain loss, as with k4); Q4_K_M unregressed. New ledger entry: the residual Q5/Q6 gap is the per-(blk, j) weight unpack — our k5/k6 tiles run ~0.69 of our k4 tile where lcpp's Q5/Q6 hold ~0.86 of their Q4 — with repack-side candidate fixes and an iso-first plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…s a load artifact Re-race with warm page cache on a quiet box (the earlier 'Parsec running' flag was wrong — parsecd is the idle login daemon; the real dirt was cold file cache and background load): lcpp pp512 = 172 / 131 / 138.5 on Q4/Q5/Q6 (their binary unchanged since Jul 3 — the session-2 lcpp reads were depressed ~9%), das = 151-163 (bimodal run-to-run, suspect E-core lane placement; lcpp holds +-0.3%) / 110 / 111. True ratios: Q4 0.88-0.95x (the 158-vs-158 'parity' was two mutually-depressed reads coinciding), Q5 0.84x, Q6 0.80x. Iso grid bench (same box, same run): tile k4 97.0 / k5 65.2 / k6 57.6 GMAC/s, gemv 64.3 / 27.5 / 34.5 — the Q5/Q6 e2e gap is fully the kernel unpack, and the k5 GEMV is its own sub-item (0.43x of k4, 19.4 GB/s weight stream). Ledger entry updated with the confirmed numbers; lcpp repack coverage source-verified (Q5_K/Q6_K repack is NEON-only, no AVX2 arm — reconciling the zen2 ~2x wins with the M1 deficit). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… planes The k5/k6 kernel-unpack lever, landed both-sides-win: the kq tile now reads a BYTE-EXPANDED weight panel (one byte per weight — zero unpack ALU in the kernel) that the batch cell unpacks from the packed grp planes once per (group, token-block) via unpack_kq_panel_grp (SWAR uint64 deposit), so the 5/6-bit deposit amortizes over TB=128 tokens instead of the tile's 4. The DRAM planes KEEP the packed 160/192 B/superblock form — the decode path (model-level DRAM-bound) is byte-identical to kq v2. Measured (M1 Max, Qwen3-4B, warm files, quiet box, 3 alternated reps): - Q5_K_M pp512 110 -> 150-168 t/s (lcpp 131 -> das 1.14-1.28x) - Q6_K pp512 111 -> 140-143 t/s (lcpp 138.5 -> das 1.01-1.03x) - tg64 at v2 parity: Q5 ~33.7, Q6 ~30.5, Q4_K_M control untouched - iso tile: k5 65 -> 89, k6 58 -> 76 GMAC/s (probe x16 amortization; production TB=128 is x32); gemv unchanged at the v2 numbers The measured dead end (recorded in the ledger, do not re-learn): pure byte-expanded DRAM planes win the same pp but cost tg -30%/-20% on Q5/Q6 — decode reads the whole blob per token, so plane bytes ARE decode time. k4 stays nibble-packed everywhere for the same reason. emit_block_kqv2 grows a kqBytes mode (verbatim wlo/whi loads; tile stamps only — the gemv keeps the in-kernel unpack); kq_grp_row_dot_b is the tile wrappers' byte-panel reference body. The gemv/rows/portable/MoE-groupn rails are untouched kq v2. The tile-vs-gemv bit-exactness gate now also covers the unpack; the tune probe benches tiles in the amortized production shape (per-group unpack included). Suite 426/410/0-failed (test_kv_paged 2 errors = known master red); parity 41/50-run 0-failed; tune crowns unchanged (mr4 all three formats). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ow, not code Lane-timeline traces (DASLLAMA_PREFILL_TRACE) of a fast (168 t/s) vs slow (150) Q5 pp512 run: all 8 lanes 98-99% utilized in BOTH modes, identical chunk counts, and a UNIFORM x1.113 per-chunk slowdown flat across run deciles — one P-cluster clock step, run-scoped. First-run-after-idle rides a ~3 s boost window; back-to-back runs sit at the sustained clock at llama-bench stability (+-0.6%). 45 s cool-downs recover only partially; pmset shows no thermal warnings. E-core lane placement EXONERATED. Method rule (ledger + tune doc): report the steady-state MEDIAN of >=3 back-to-back reps, discard the first-after-idle rep — best-of-N picks the boost outlier. Steady-state scoreboard (M1 Max, Qwen3-4B vs lcpp steady): Q5 ~150 vs 131 = 1.15x, Q6 ~141 vs 138.5 = 1.02x, Q4_K_M ~155-158 vs 172 = 0.90x. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…decode chains The chains now run one activation image per consumed form (the per-op mm_pre_f contract): q8 projections read the Q8_0 image, kq projections the Q8_K image (kxq/kxs/kxbs). The two mid-chain requants move to Q8_K-legal granules: the FFN w2-input requant to 256-row stage-0 groups, the attention wo-input requant to HEAD GROUPS of 256/head_size heads (head_size 128 = head pairs) in both the single-pass and flash-combine shapes. Fuse gates: dim % 256 for any kq weight, hidden % 256 for a kq down-proj, qd % 256 + head-size divisibility for a kq wo — every real kq model passes. Bit-exact vs the per-op kq path: test_fused_decode grew a kq arm (Qwen3-4B Q5_K_M = k5+k6+head-pairs; gemma-2-2b Q4_K_M = k4+k6+softcap/ post-norm deferral) — ids AND full logits exact. Measured (M1 Max, steady-state median): Q6_K tg 30.5 -> 31.1 (+2.1%), Q5_K_M ~flat (33.8), Q4_K_M control flat; decode_prof confirms attn_chain + ffn_chain carry ~90% of the kq decode. The dead Q8_0-bs chain arms and requant_rows_q8_bs go with it. Suite 428/412/0-failed (test_kv_paged 2 errors = known master red); parity 41/50-run 0-failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…maddubs ABBA on zen2 (Threadripper 3990X, 16 affinity-pinned cores both sides, before=52a22a39b after=3d78ca8ef, Qwen3-4B): das beats lcpp pp512 1.84x on Q5_K_M (159 vs 87) and 1.32x on Q6_K (129 vs 98) — lcpp has no AVX2 repack for those; Q4_K_M ~parity (161-180 vs 168.3) against their one AVX2 K-quant repack GEMM. tg64 at lcpp parity on all three. The kq v2+v3 arc itself measures ~neutral on the maddubs lattice (the M1 unpack win was an sdot-lattice property), with one cost: Q6 pp -4.6% (unattributed between the v2 signed-fold chain shape and the v3 second verbatim load; candidate fix = per-ISA kqBytes gate, ledgered). The zen2 kq tune sweep was not run (fallback maddubs-mr8 rows served both legs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…#3448) Every dasLLAMA test that requires the transformer umbrella failed the full AOT sweep with error[50101] on dasllama_arch_qwen35 symbols — the module landed without its tests/aot/CMakeLists.txt registration (per-PR CI only builds the tests/language subset, so the gap was invisible until the nightly/preflight full sweep). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…into AOT stubs The cmake AOT rules drive generation through utils/aot/main.das, whose CodeOfPolicies never set tune_frozen — only the C++ -aot path (utils/daScript/main.cpp) got the policy when #3423 introduced it. The das-driven generation therefore compiled in the STAMPED world (manifest or fallback= [tune] stamps + full variants registries) while the AOT-consuming runtime (policies.aot, JIT off) compiles gated to the reference tier — the recorded hashes desync and any function with a [tune] family in its static call graph fails error[50101] at link. Latent since #3423; first exposed by test_kquant's kq tile gates (the first TEST-file code calling [tune] functions directly — everything else reaches them through KernelBackend function-pointer slots, which no stub hash depends on). One line: cop.tune_frozen = true in updateCOP, the das twin of main.cpp's -aot policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR advances dasLLAMA’s native K-quant (Q4_K/Q5_K/Q6_K) execution path by aligning kernels with llama.cpp’s integer-fold scheme, adding byte-expanded panel scratch for k5/k6 tiles, and re-enabling K-quant layers in fused decode chains with new parity gates. It also includes two small AOT infrastructure fixes to prevent tune-stamp hash drift and to ensure qwen35 arch modules are included in full-AOT builds.

Changes:

Implement kq v2 activation contract (Q8_K-form activations) with integer-fold accumulation and per-format kq layout companions (k4/k5/k6 can choose different grp interleaves).
Add kq v3 “panel scratch” path for k5/k6 tiles (unpack once per (group, token-block), tiles read byte-expanded panel).
Expand test coverage with token-parity gates for real kq models and additional synthetic correctness gates (multi-row GEMV, fused-vs-std bit-exactness).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/aot/main.das	Freeze `[tune]` selection for AOT stub generation to avoid cross-box stamp leakage / hash drift.
tests/aot/CMakeLists.txt	Register qwen35 arch module in AOT test build list.
modules/dasLLAMA/dasllama/dasllama_quant.das	Add Q8_K activation quantizer used by kq v2 kernels.
modules/dasLLAMA/dasllama/dasllama_math.das	Add backend slot for per-format kq layout and update kq stride/constants for v2.
modules/dasLLAMA/dasllama/dasllama_math_default.das	Update portable kq dots/dequants to v2 folds and new 20B k4/k5 scale stride.
modules/dasLLAMA/dasllama/dasllama_math_gen.das	Introduce per-format kq tune families, layouts, byte-panel tile plumbing, and new unpack helper(s).
modules/dasLLAMA/dasllama/dasllama_gemm_gen.das	Update LLVM tile emitter to kq v2 integer folds and support kqBytes (byte-expanded) tile loads.
modules/dasLLAMA/dasllama/dasllama_gguf.das	Switch k4/k5 transcode to verbatim disk scale block (20B) and update plane dequant references.
modules/dasLLAMA/dasllama/dasllama_common.das	Wire Q8_K activation planes into decode/prefill paths, per-format repack mrs, and fused-chain regrain.
modules/dasLLAMA/harness/gen_tune_probe.das	Extend tuner to gate/bench kq families and write per-format manifest entries.
tests/dasLLAMA/test_kquant.das	Update synthetic kq gates for v2 activation form and add multi-row GEMV correctness gate.
tests/dasLLAMA/test_parity.das	Add token-parity tests for real kq ggufs (Qwen3 Q4/Q5/Q6, gemma2 Q4 softcap rail, MoE arm).
tests/dasLLAMA/test_fused_decode.das	Add fused-vs-std bit-exact gate for kq layers under auto backend.
modules/dasLLAMA/tune_for_this_box.md	Document per-format kq tune families and steady-state measurement methodology updates.
modules/dasLLAMA/API_REWORK.md	Ledger updates documenting shipped kq regrain/panel-scratch work and measurement findings.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

…d moe_xq image stale kq v2 quantizes the MoE layer input ONLY in Q8_K form when the routed gate/up stacks are K-quant; the q8 shared-expert path (moe_ffn_core) still reads the hoisted moe_xq/moe_xs image, which stayed stale/zero — the shared expert silently dropped out on every such layer. Only mixed q8/kq files hit it (kq-uniform layers ride the fused decode chains): Qwen3.6-35B UD-Q4_K_M degenerated to space-token spam. Fill the q8 image alongside Q8_K when the layer has a (non-dense) shared expert. 35B counting fixture: 11/40 -> 40/40. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n-fixes dasLLAMA: qwen3next arch (Qwen3-Coder-Next 80B) + 35B serial-hole fixes + #3450 shexp-dropout fix

…shexp fix + >2^31 wscale The 122B (arch qwen35moe, zero new arch work) exposed three latent bugs, each the first time a size threshold or rail was crossed: - ffn_moe_prefill_grouped on K-quant layers filled ONLY the Q8_K activation image; the q8 shared-expert batch GEMMs (step 4) read the STALE Q8_0 image — the shexp silently corrupted on EVERY kform+shexp model's grouped prefill (#3450's prefill twin; that fix covered decode only). CN/35B survived the counting fixture by luck; the 122B flipped at token 0 (generated ordered letters instead of digits). Fix: fill the q8 image alongside when the layer has a non-dense shexp. gemma4's twin is clean (its dense shexp runs mm_b_f off float rows). - wscale_convert_f16 resized the f16 scale plane through int(): Coder-Next Q8_0 is the first model past 2^31 block scales (2.478e9) — the resize wrapped negative. int64 resize; the rest of the q8 rail was already 64-bit clean. - Native multi-part (split) GGUF loading: gguf_shard_paths (llama.cpp shard-name expansion, normalized from any shard index), gguf_map_shards (recursive fmap, all mappings alive), parse_gguf_meta_shards (union directory, per-tensor shard, borrowed @do_not_delete bases), with_tensor_view choke point in every tensor reader. Single-file rail byte-identical (46-fixture sweep; split-vs-merged produced identical outputs during the bug hunt). All five big-model downloads ship split — merge-first would churn >1TB of disk. Debug ladder that localized the shexp bug: simple_ids oracle transfers to the 122B (llama-cli chat-mode does not) -> merged-file A/B exonerated the split loader -> kq-native-off green -> per-family fmt-force bisects -> reference-prefill green -> grouped prefill code read. Fixtures: Coder-Next Q8_0, 122B Q4_K_M (3-shard), 122B Q8_0 (4-shard, ~130GB) — huge-tier gated; _model_tier tiers split models on their TOTAL size. test_kquant gates parameterized by shape (n, and d/nreg for groupn) + a new batch-groupn gate (pins the x64-gen backend). skills/strings.md: Result<T;E> is a structural tuple (._is_ok/._value/._error), not a variant — corrected the try_to_int example. Parity green across the board: kquant 81/81, small tier 47, CN Q4_K_M + Q8_0, 35B, 122B Q4_K_M + Q8_0 — token-for-token vs llama.cpp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

borisbat and others added 14 commits July 12, 2026 18:28

Copilot AI review requested due to automatic review settings July 13, 2026 02:52

Copilot started reviewing on behalf of borisbat July 13, 2026 02:53 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread modules/dasLLAMA/dasllama/dasllama_common.das Outdated

dasLLAMA: kq_repacked comment names the per-format mr fields (review)

ea046ab

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 13, 2026 02:58

Copilot started reviewing on behalf of borisbat July 13, 2026 02:59 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

borisbat merged commit 6924e28 into master Jul 13, 2026
39 of 40 checks passed

borisbat deleted the bbatkin/dasllama-kquant-v3 branch July 13, 2026 04:28

This was referenced Jul 13, 2026

extended_checks ser/deser sweep SIGSEGV'd once on linux (non-reproducible; PR #3450 first run) #3451

Open

dasLLAMA: qwen3next arch (Qwen3-Coder-Next 80B) + 35B serial-hole fixes + #3450 shexp-dropout fix #3457

Merged

borisbat added a commit that referenced this pull request Jul 13, 2026

Merge pull request #3457 from GaijinEntertainment/bbatkin/qwen3next-o…

93f8550

…n-fixes dasLLAMA: qwen3next arch (Qwen3-Coder-Next 80B) + 35B serial-hole fixes + #3450 shexp-dropout fix

borisbat mentioned this pull request Jul 14, 2026

dasLLAMA: Qwen3.5-122B — native split-GGUF loading + grouped-prefill shexp fix + >2^31 wscale #3468

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

dasLLAMA: K-quant kernel arc — kq v2 integer folds, kq v3 panel-scratch tiles, chain regrain, parity gates#3450

dasLLAMA: K-quant kernel arc — kq v2 integer folds, kq v3 panel-scratch tiles, chain regrain, parity gates#3450
borisbat merged 15 commits into
masterfrom
bbatkin/dasllama-kquant-v3

borisbat commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

borisbat commented Jul 13, 2026

What

Results (Qwen3-4B, steady-state median vs llama.cpp CPU)

The pieces

Drive-by fixes (both caught by the preflight full-AOT gate)

Ledgered follow-ups (API_REWORK.md)

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants