AProjQ4: imatrix-guided Q4_K dense attention projections - #952
AProjQ4: imatrix-guided Q4_K dense attention projections#952GiorgioOppo wants to merge 190 commits into
Conversation
The AProjQ4 DeepSeek V4 Flash GGUFs quantize the five dense attention projections per layer (attn_q_a, attn_q_b, attn_kv, attn_output_a, attn_output_b) as Q4_K instead of Q8_0. Loading already accepted them (tensor_expect_dense_quant_layout), but the decode graph read the Q4_K blocks through the hardcoded Q8_0 kernels and generated garbage (BOS loops), and the CPU reference died with "expected a 2D Q8_0 tensor". - Metal decode graph: gate the fused Q8_0 q_a/kv pair kernel and the plain Q8_0 matvec fallbacks on the actual tensor type, dispatching through the existing generic dense-quant path for Q4_K. Q8_0 models keep the exact kernels they used before, so their output stays bit-identical. - CPU reference: add a dense Q4_K matvec/matmul family (activations prequantized to Q8_K, rows reduced with ds4_vec_dot_q4_K_q8_K), with grouped, decode-scratch and prefill-batch variants, and dispatch the attention projection call sites on tensor type. - CUDA decode-TP attention output split now refuses non-Q8_0 output projections with a clear error instead of computing garbage. Verified on Apple M1 Pro 16 GB with --metal --ssd-streaming on DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix: coherent greedy output, --decode-consistency max_abs=0 rms=0, and no regression on the AProjQ8 gguf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DS4_METAL_STREAMING_EXPERT_NOCACHE=1 serves the streaming expert preads from a second F_NOCACHE descriptor (reopened by path: a dup would share the file description with the mmap-backed fd) and turns off the F_RDADVISE readahead hints, which only warm page cache the preads no longer consume. On tight-RAM machines the ~1 GB/token of routed-expert churn was flowing through the page cache and kept evicting the ~7 GiB of mapped dense weights that decode re-reads every token; once the dense set fell out, generation collapsed to SSD-fault speed and could never recover because the next run's expert traffic flushed it again. A/B on Apple M1 Pro 16 GB, AProjQ4 gguf, greedy 32 tokens, cold page cache: baseline 0.29/0.29 tok/s (stuck across runs); with the flag the dense set survives the expert traffic and warms across runs — 0.29, 2.09, 1.93, 1.91 tok/s. Generated tokens bit-identical to baseline in all runs; --decode-consistency max_abs=0 rms=0. Opt-in because on the >=96 GB target machines everything fits in RAM and cached preads are strictly better (second touch is free). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DS4_METAL_STREAMING_EXPERT_PREAD_SPLIT=N expands every expert slab pread into up to N disjoint 16 KB-aligned ranges read concurrently by the existing pread pool. Decode misses queue only a handful of slabs per layer (~4 experts x 3 slabs) while NVMe drives reach their random-read ceiling around ~24 requests in flight: splitting deepens the queue at identical bytes. Results are folded back per original slab so callers keep per-task ok/bytes/ms. Default 1 = historical path. Interleaved A/B on Apple M1 Pro 16 GB, AProjQ4 gguf, greedy 32 tokens, warm state, with DS4_METAL_STREAMING_EXPERT_NOCACHE=1: split=1 1.94/1.92/1.93 tok/s, split=4 2.24/2.20/2.24 tok/s (+16%). Tokens bit-identical in all runs; --decode-consistency max_abs=0 rms=0. Same trick as the Swift port's DS4_PREAD_SPLIT, where 4 also measured best. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The AProjQ4 GGUFs load fine on CUDA but every dense Q4_K projection hit the "matmul_quant: unsupported type 12" error. Add the missing path: - matmul_q4_K_dense_kernel + cuda_matmul_q4_K_tensor: activations are quantized to Q8_K rows with the existing routed-MoE quantizer and each weight row is reduced with the shared dev_dot_q4_K_q8_K_block, so the numerics match the already-validated MoE Q4_K path. 8 lanes per row, same idiom as the MoE decode kernels. Wired into ds4_gpu_matmul_quant_tensor as type 12. Every token re-reads the weight rows from device memory: fine for decode, correct but unoptimized for prefill chunks (a dequant+GEMM path can follow if it shows up in profiles). ROCm compiles the same source through HIP. - ds4.c: the specialized Q4_K attention-output low projection now falls through to the generic per-group dense-quant loop when the backend returns 0 (the CUDA/ROCm stub), instead of failing the layer. Metal keeps its fast path; a quick A/B confirms bit-identical tokens. Still unsupported with Q4_K projections on CUDA: the decode-TP attention-output split and the kslice fused paths — both already refuse loudly via the type guards instead of computing garbage. NOT compile-tested on a CUDA machine (authored on a Mac): testers with NVIDIA/ROCm hardware, please build and run the AProjQ4 gguf with --temp 0 plus --decode-consistency, and confirm AProjQ8 stays bit-identical to main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reduce Metal SSD paging and verifier cache churn, remove CUDA/ROCm mid-token fences, and add CUDA DSpark verifier/proposer fast paths with guarded fallbacks. Validation: Apple M1 Pro, Metal/CPU, no model-backed quant run for this final patch; make -B, make cpu, make ds4_test, ./ds4_test --server, 106/106 engine placement checks, ROCm C syntax check, and git diff --check passed. Target model is Flash 0731 AProjQ4. CUDA/ROCm hardware validation remains pending; Metal kernel tests were unavailable because the Codex process exposed no Metal device.
Port antirez#766 commit d99de5c while preserving the local Metal/CUDA DSpark exact-N, tiny-batch, and Q4 paths. Add fail-closed diagnostics, parity tests, and GB10 benchmark documentation.
Reuse canonical Q8_1 scratch for Q4 MMVQ, add grouped attention-A and the exact K1024 persistent path, and cover the new dispatches with rollback gates and parity tests.
(cherry picked from commit c012d5c)
(cherry picked from commit e1147cd)
Use a fixed-route direct kernel for the production Q4 attention output-A shape on Apple M1-M4, avoiding the generic route-map and work-list path. Add fail-closed controls, a production-shape oracle, and a resident GPU-only benchmark.
|
I would like to thank @adamlawi, @iammac2, @trueimage, @kyuz0, @evandhoffman and @OPS-NeoRetro for their help. |
|
Third-party Metal validation of this branch, for the record — no action needed on this PR. Measured on an M5 Max 128 GB at
One caveat worth having if numbers from different people are compared: within a session the ratio narrows from ~1.19 (first rep, cold) to ~1.13 (third), because q4 loses more to drift than q8 (−10.0% vs −4.1%). A cold single-rep run reporting ~1.19 is consistent with these results, not a disagreement. Die temperature was logged throughout and the decline is not thermal — the machine peaks at 78 °C after four minutes and then runs cooler while throughput keeps falling. Full method, per-rep tables, the complete temperature log and raw CSVs: |
…tion # Conflicts: # Makefile # ds4.c # ds4_gpu.h # ds4_metal.m
|
@GiorgioOppo You asked in #621 whether our extended-range CUDA result was noise or solid. Answering here, since this is where the work continues, and carrying over the two findings from that thread that are still useful — one of them for anyone benchmarking this branch, not just for us. Noise or solid: it is solid, and it is negligible. Those are different questions.Statistically solid. 10 measurement windows over 2 hours, 20 paired samples per context point, 160 runs, zero failures, on GB10 / sm_121 at
All four intervals exclude zero (3.2–5.1σ), and the series had converged: the tenth window moved every mean by ≤0.02 pp. It is not noise. Practically negligible. The effects are two to three tenths of a percent. Nobody will feel that, and no decision about this PR should turn on it. If you need one sentence for the PR: on CUDA/GB10, Q4 and Q8 prefill are at parity from 8k to 64k, with Q4 marginally ahead at 8k and marginally behind above it. The honest framing is that we can now resolve an effect too small to matter — which is a fine place to end up, and better than the earlier −1.0% figure that turned out to be an artefact of our own method. The finding worth carrying into this PR: measurement order biases the comparisonOn this box, throughput declines monotonically over a measurement window. In two windows, all 24 arm-to-arm comparisons drifted downward. If the arms always run in the same order, whichever model is loaded later is penalised. Measured directly:
The bias is larger than three of the four effects being measured. At 32768 the sign of the result depends only on which model you load first. This is not thermal on our side: SM clock held 2405 MHz for the full two hours, no throttling flags, 31–61 °C. @evandhoffman reports something consistent above — his ratio narrows from ~1.19 on the first rep to ~1.13 by the third within a session. Same class of effect, different platform. The fix is free: alternate the order between rounds ( What carried over from #621 and still holds at
|
can you also post the decode speed? |
|
@GiorgioOppo Decode numbers below. One caveat first, because it matters for how much weight to put on them: these are not from the balanced series. That series measured pure prefill ( Decode, steady-state t/s, GB10 / sm_121,
Q4 decode leads Q8 by roughly 14% from 4k to 8k, and by about 17.5% at 2048. Two reasons to trust this despite the weaker design:
Scope limits, stated plainly: ctx only up to 8192 (these windows predate the extended sweep), prompt file was the 135 kB one rather than the 405 kB used later, and the figures are from So, combining with the prefill series: on CUDA/GB10, Q4 is at parity with Q8 on prefill and roughly 14% ahead on decode. That matches what @evandhoffman reports on Metal, where the advantage is also decode-only ( If you want decode measured with the balanced design — alternating order, paired ratios, confidence intervals like the prefill table — say so and I will run it. It is about 40 minutes of windows and would give decode the same statistical footing as the prefill numbers. |
Throughput declines across a measurement window, so a fixed arm order penalises whichever arm always runs last. @adamlawi measured this on antirez/ds4#952: the positional bias was larger than three of the four effects being compared, and at one frontier the SIGN of the result depended only on which model loaded first. Their figures are quoted as reported and are not verified here -- the reason to act is the method. Three places, matching the issue's three items. **The suite runner** ran `backends.items()` in the same order every trial, so one backend was always last. `trial_order` now reverses on even trials. Extracted rather than left inline in main(), so the property that matters -- no backend holds the last position in every trial -- is a test rather than a comment. **decode_ab.sh** had the same fixed order. decode_ab_engine.sh has alternated since it was written; this script predates the finding. It now alternates and appends a run-order.txt beside the CSVs. **The row** records `run_position` and `run_arms`. Alternating removes the bias going forward; it does nothing for the 979 rows already written, and those cannot be retro-corrected for a bias nobody wrote down. Absent means the order was not recorded -- deliberately not defaulted to 1, which would claim every existing row ran first, which is the bias being looked for. Neither field is in REQUIRED, for the same reason as client_version: every existing row predates them and `validate` runs on read. With an odd trial count the split is uneven -- 2 of 3 in first position, not 1.5 -- which is better than 3 of 3 and is why the position is worth recording rather than assumed to cancel. 629 passed, 3 skipped. The 5 ruff findings in run.py predate this change.
4 participants
Summary
This PR supersedes #621.
PR #621 remains the development, investigation, and benchmark archive. This PR
contains only the production-ready changes required to support the DeepSeek V4
Flash AProjQ4 layout, together with focused correctness tests,
backend-specific production paths, and the documentation needed to reproduce
the model.
AProjQ4 changes the 215 dense attention-projection tensors from
Q8_0toQ4_K, while preserving:-0731checkpoint;The result is a model that is 2.14 GiB smaller and substantially faster in
decode on Metal, CUDA, and ROCm.
1. Imatrix-guided requantization
Method
The AProjQ4 artifact is produced directly from the matching AProjQ8 GGUF. The
original Hugging Face safetensors are not required.
The conversion selects five tensor families in each of the 43 transformer
layers:
attn_q_aattn_q_battn_kvattn_output_aattn_output_bAll unselected tensors are copied from the source GGUF byte for byte.
How the imatrix is used
The conversion uses the published DS4 activation-importance matrix:
The imatrix supplies a per-input-column importance vector for each selected
tensor. The Q4_K quantizer uses these values to weight quantization error while
choosing block scales, minima, and quantized codes.
As a result, errors affecting frequently used activation columns receive more
weight than errors on columns that are rarely activated by the calibration
workload.
The tensor type remains
Q4_K; the imatrix changes how the available Q4_Krepresentation is selected, not its runtime format.
--imatrix-strictis used so that conversion fails if a selected tensor has:This prevents a selected projection from silently falling back to
non-imatrix quantization.
Reproduction
First validate the complete conversion plan without writing an output file:
Then produce the AProjQ4 GGUF:
The source and output paths must be different. The output should first be
written to a temporary filename and renamed only after checksum, tensor-layout,
and quality validation.
Published artifact
The saved memory can be used as additional KV/context capacity on resident
systems or as a larger routed-expert cache under SSD streaming.
Quality validation
The original same-checkpoint comparison on the 100-case official-continuation
fixture showed no measured quality regression:
avg_nll0.4048112510.8586250.3964324030.860355After the later
mainrealignment, the absolute Q4 figures changed slightlywith the engine:
The complete 100-case TSV remained byte-identical across
9a448bb1,35b87dda, and the final development head6a20b131. The final grouped CUDAquantizer therefore introduced no numerical movement.
Note
The quality fixture contains 100 cases and 2,313 target tokens. It establishes
no measured regression on this fixture; it is not a claim that Q4 is
universally more accurate than Q8.
2. Metal results
Test platform
The final development commit
6a20b131was tested twice, using threeinterleaved Q4/Q8 sweeps from 2,048 to 65,536 context tokens.
Decode
Selected final-head results:
The advantage is largest at short context and decreases gradually as context
grows, but it never reverses.
The
1.155paired median reproduces the earlier result to three decimal placesdespite 226 intervening development commits.
Prefill
1.0031.0210.998The measured range in the final cold-start run was
0.980–1.027. Takentogether, the three independent measurements show that full-model Metal
prefill is currently indistinguishable between Q4 and Q8.
The production claim for Metal is therefore:
It should not be described as a proven Metal prefill win until a balanced test
shows a repeatable difference above the session-order noise floor.
Startup and first token
An earlier isolated matched run on the same M5 Max measured:
This supports a lower time-to-first-token claim, but it should not be described
as faster thermal warm-up.
The final temperature-logged test showed that the machine reached its peak
temperature within roughly two minutes and then became cooler while throughput
continued to decline. The longer-session drift is therefore not explained by
die temperature alone.
A cold single repetition can report a Q4/Q8 decode ratio close to
1.19, whilethe more representative three-repetition paired median is
1.155.Metal conclusion
Development evidence:
final M5 Max test.
3. CUDA results
Test platform
The final development head
6a20b131was tested with balanced model ordering.This is important because the first arm in a benchmark pair has a measured
advantage of approximately
0.38–0.53percentage points on this machine.The final prefill experiment therefore alternated:
It used 10 windows, 20 paired samples per context, 160 total runs, and pure
prefill with
--gen-tokens 0.Full-model Q4 versus Q8 prefill
[+0.13%, +0.54%][−0.47%, −0.21%][−0.35%, −0.09%][−0.41%, −0.16%]The signs are statistically stable, but every difference is only a fraction of
one percent.
The defensible full-model conclusion is:
This balanced result supersedes the earlier
+2.39%Q4/Q8 figure at 8K, whichused a different prompt, generation enabled, and an unbalanced measurement
order.
Gain from the new grouped prefill path
The production grouped Q4 attention-A path was also compared directly with its
rollback path.
0.18–0.35%This is a separate result from Q4-versus-Q8:
implementation inside the Q4 path.
The grouped optimization is a solid approximately 2% implementation win,
even though the complete Q4 and Q8 models are already close to the same prefill
throughput.
Decode
The grouped quantizer is a prefill-only change. In the matched decode sweep,
AProjQ4 retained an advantage of:
over AProjQ8 across the tested short-context frontiers.
Numerical validation
The final grouped Q8_1 quantizer was verified as actually active through
fail-closed path controls.
Its output is byte-identical across:
for all 100 quality cases. This is a scheduling and throughput optimization,
not a numerical approximation.
CUDA conclusion
Development evidence:
4. ROCm results
Test platform
Important
No complete model-backed Q4/Q8 A/B for the final development head
6a20b131has been published yet.The table below is the latest complete, clean, and reproducible ROCm result.
It was measured at
5c5b1d5, before the later K128 staging change.The 2K row is excluded from the steady-state prefill conclusion because Q8
showed a repeatable first-frontier cold-start artefact.
Startup
Correctness
The latest safe-path correctness follow-up confirmed:
speed sweep.
An earlier experimental compounded-WMMA path produced a
+5.3%prefillimprovement, but also introduced a real decode-versus-prefill mismatch. That
path is not part of the safe production result and should not be included in
this clean PR.
K128 status
A later development commit added K128/P144 staging for eligible 256-row
direct-WMMA launches.
That implementation has not yet received a complete final-head model-backed
Q4/Q8 A/B in the public discussion. This PR therefore intentionally reports
the verified approximately 3% prefill gap rather than the informal
approximately 2% estimate.
The ROCm table should be updated after the cleaned branch receives a balanced
K128-enabled test.
ROCm conclusion
Development evidence:
Overall result
Scope of this clean PR
This PR includes:
This PR intentionally excludes:
Validation required before merge
The cleaned branch should be frozen at one SHA and tested again before merge.
gfx1151.main.For throughput differences below 1%, benchmark order must be counterbalanced:
Fixed-order measurements are not sufficient for sub-percent performance claims.