Support AProjQ4 GGUFs: Q4_K dense attention projections

[GiorgioOppo]

The AProjQ4 DeepSeek V4 Flash GGUFs quantize the five dense attention projections per layer (attn_q_a, attn_q_b, attn_kv, attn_output_a, attn_output_b) as Q4_K instead of Q8_0. Loading already accepted them (tensor_expect_dense_quant_layout), but the decode graph read the Q4_K blocks through the hardcoded Q8_0 kernels and generated garbage (BOS loops), and the CPU reference died with "expected a 2D Q8_0 tensor".

• Metal decode graph: gate the fused Q8_0 q_a/kv pair kernel and the plain Q8_0 matvec fallbacks on the actual tensor type, dispatching through the existing generic dense-quant path for Q4_K. Q8_0 models keep the exact kernels they used before, so their output stays bit-identical.
• CPU reference: add a dense Q4_K matvec/matmul family (activations prequantized to Q8_K, rows reduced with ds4_vec_dot_q4_K_q8_K), with grouped, decode-scratch and prefill-batch variants, and dispatch the attention projection call sites on tensor type.
• CUDA decode-TP attention output split now refuses non-Q8_0 output projections with a clear error instead of computing garbage.

Verified on Apple M1 Pro 16 GB with --metal --ssd-streaming on DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix: coherent greedy output, --decode-consistency max_abs=0 rms=0, and no regression on the AProjQ8 gguf.

[GiorgioOppo]

#618

[GiorgioOppo]

DS4_METAL_STREAMING_EXPERT_NOCACHE=1 DS4_METAL_STREAMING_EXPERT_PREAD_SPLIT=4 ./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 448 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf

[GiorgioOppo]

@antirez ,
Could you please review this post in case you accept it or close it and the related post on hf (https://huggingface.co/antirez/deepseek-v4-gguf/discussions/18)?

[adamlawi]

Volunteering as a tester for the CUDA path — this is the configuration your table lists as implemented but untested.

Machine: stock NVIDIA DGX Spark, GB10 (sm_121), 128 GB unified memory, aarch64, Ubuntu 24.04.4, driver 580.173.02, CUDA 13.0.88. Built with make cuda-spark. Not a Mac and not SSD streaming — the model is fully resident, so this exercises the decode path under different pressure than the M1 Pro run.

What I can run, using a harness already in place from measuring ds4f-q2 vs ds4f-q2-q4 on this box:

• --decode-consistency and greedy A/B against the AProjQ8 baseline (correctness first)
• ds4-bench sweeps at 2k/4k/6k/8k, 128 generated tokens per frontier — the AProjQ8 baseline here is 872 t/s prefill / 14.69 t/s decode at 8k
• full-window prefill (194k tokens) with free-memory sampling every 2 s, reporting the minimum at peak
• perplexity on a fixed sample, to put a number on the quality delta on this hardware

Why the memory result is the interesting one here. Resident ds4f-q2 leaves very little headroom on a 128 GB Spark: at --ctx 200000, filling 97% of the window leaves 2.5 GB free at peak; --ctx 220000 leaves 0.94 GB; --ctx 300000 cannot be filled past ~52% (0.7 GB at 155k tokens). The dense tensors account for 8.20 GiB of the startup span phase on this build, so cutting them to Q4_K should show up directly as usable context. If it frees ~2 GiB, that plausibly moves the ceiling from 200k to 250–300k — a benefit orthogonal to the SSD-streaming speedup you measured.

What blocks me: I could not find an AProjQ4 GGUF published (not in antirez/deepseek-v4-gguf, and none under your HF account), and gguf-tools/deepseek4-quantize only builds from --hf DIR with the original safetensors — ~568 GB, which does not fit here alongside the ~78 GB output (644 GB free). Your write-up mentions wiring --requant-gguf to consume a dense imatrix, but I do not see that tool on any of the three branches of your fork.

So: could you publish DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf, or share the requant path that derives it from an existing GGUF? Either way I can turn results around quickly.

Context for the numbers above: #705.

[GiorgioOppo]

Hi, the download is blocked by a pr on hf, https://huggingface.co/antirez/deepseek-v4-gguf/discussions/18 You should be able to download the template from https://huggingface.co/antirez/deepseek-v4-gguf/resolve/5ca1abfe6d0c518ffcd731eae01bc0c88a2e59db/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf?download=true

[adamlawi]

Tested on a stock NVIDIA DGX Spark (GB10, sm_121, 128 GB unified, CUDA 13.0.88, driver 580.173.02), as offered above. Short version: the CUDA path is correct, the memory win is bigger than expected, but prefill regresses ~16x, which is a blocker for resident use on this machine.

Setup: branch aprojq4-dense-attention at 41aefd2c, make cuda-spark, no local changes — builds clean in 2m52s with zero warnings. Model is the GGUF you linked (5ca1abfe, 78.62 GiB). Baseline everywhere below is IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731, run with the same binary, so the comparison isolates the quant rather than the code version.

One caveat up front: the AProjQ4 file is from an older checkpoint (no -0731) and I did not download a matching AProjQ8 baseline, so memory and correctness numbers are clean, but speed and perplexity carry a checkpoint difference that I cannot separate out.

1. Correctness: the CUDA path looks good

--decode-consistency 64:

model	max_abs	rms
AProjQ8 (baseline)	1.476	0.293
AProjQ4	1.118	0.191

Top-1 matches, top-10 sets match. The divergence is smaller than on the existing Q8_0 path, so the new kernel is not introducing numerical error.

2. Memory: a bigger win than the file-size delta suggests

--inspect shows 215 tensors moved Q8_0 → Q4_K (345 → 130 Q8_0, plus 215 Q4_K), everything else byte-identical. Startup spans and the memory plan both drop by exactly the file-size delta:

• startup model preparation: 8.20 GiB → 6.06 GiB, and no time penalty (2.12s → 1.70s)
• planned at --ctx 200000: 85.21 GiB → 83.07 GiB
• aligned artifacts: 474 / 78.71 GiB → 259 / 74.16 GiB (the Q4_K tensors get no repack builder)

But free memory after startup improves by ~11 GB, well beyond the 2.14 GiB the file shrank:

ctx	AProjQ8 free	AProjQ4 free
200000	9.8 GB	21.3 GB
250000	8.7 GB	20.1 GB
300000	8.1 GB	19.1 GB

My guess is that the Q4_K tensors no longer exist simultaneously as an aligned artifact and a mapping, which accounts for roughly 6.7 GiB of it — I have not verified the rest, so treat the exact figure with caution. On GB10 (CPU and GPU share one 128 GB pool) this is a real change: --ctx 300000 now starts with more headroom than --ctx 200000 had before.

3. Prefill: ~16x slower, and this is the blocker

ds4-bench, same prompt file, 128 generated tokens per frontier:

ctx	prefill Q8	prefill Q4	decode Q8	decode Q4
2048	817.32	53.98	17.55	17.28
4096	892.89	54.20	15.08	14.86
6144	883.59	54.18	14.97	14.78
8192	877.75	54.17	14.83	14.65

Decode is essentially free (−1.2%) — impressive for halving the precision of the attention projections. All the cost is in prefill, exactly where your comment above matmul_q4_K_dense_kernel predicts it ("correct-but-unoptimized for prefill chunks"; every token re-reads the weight rows). With a 4096-token prefill chunk this dominates.

Practical effect for a long-context server: a full 194681-token prefill went at a steady 54.6 t/s, i.e. ~59 minutes instead of ~4.5 (I aborted the run at 17%). So on this machine the patch trades a context ceiling we can now afford for one we can no longer fill — a 15k-token page would go from ~17s to ~4.5 minutes.

That makes the dequant + GEMM prefill path you mention in the comment the deciding factor here, rather than a later optimization. Is that something you're planning? If you push a prefill path, I'm happy to re-run this whole sweep on the same machine — the harness is set up and a rebuild plus full sweep is under an hour.

4. Perplexity — reported for completeness only

Same 1882-token sample: AProjQ4 12.730 (avg_nll 2.5439) vs AProjQ8-0731 11.108 (avg_nll 2.4077). Please do not read this as a +14.6% quantization cost — it is dominated by the checkpoint difference, and it is inconsistent with your own +0.99% measured within one checkpoint. I mention it only so the number isn't misread if it surfaces elsewhere. If you publish an AProjQ8 GGUF from the same older checkpoint, I'll rerun and give you a clean quality delta.

Also worth noting for anyone reading this from the DGX Spark side: this is a different issue from #705, which is about routed Q4_K experts having no aligned repack builder. This PR touches dense tensors only and does not OOM.

[GiorgioOppo]

@adamlawi,
Thanks for the feedback, it's hard to test on cuda without it :) Anyway, it should be optimized now and the prefill should be faster. Let me know if the prefill is accelerated or unchanged. In the next few days I'll update gguf with the new version of deepseek that wasn't available.

[adamlawi]

Retested 6c4d769d on the same DGX Spark. Prefill is fixed — ~15.5x faster than before, now within ~4-5% of the Q8_0 baseline. Same binary, same prompt file, same model as my previous comment.

ctx	prefill 41aefd2c	prefill 6c4d769d	AProjQ8 baseline	gap to baseline
2048	53.98	783.60	817.32	−4.1%
4096	54.20	850.37	892.89	−4.8%
6144	54.18	841.37	883.59	−4.8%
8192	54.17	837.14	877.75	−4.6%

Decode is unchanged (17.32 / 14.90 / 14.82 / 14.67 vs 17.55 / 15.08 / 14.97 / 14.83 for Q8_0, i.e. −1.1%), so the MMQ path did not cost anything on the decode side.

Two more checks, since an optimization like this could plausibly trade accuracy for speed — it did not:

• --decode-consistency 64: rms 0.182 / max_abs 1.088, versus 0.191 / 1.118 on 41aefd2c and 0.293 / 1.476 on the Q8_0 baseline. Still better than the baseline path.
• perplexity on the same 1882-token sample: 12.730174, versus 12.729817 before the change — identical to four decimal places.

No ds4_mmq_q4_K_dense returned ... falling back lines appeared in any run, so MMQ accepted every shape in this model.

Memory is unchanged from my previous report, as expected: startup spans 6.06 GiB (vs 8.20 GiB for AProjQ8), 259 aligned artifacts / 74.16 GiB, and ~11 GB more free memory at every context size. So on a 128 GB GB10 this now looks like a straight win: ~2 GiB smaller resident model plus ~11 GB more free memory, for ~4.5% prefill and ~1% decode — and --ctx 300000 starts with more headroom than --ctx 200000 had with AProjQ8.

Nice work — that was a fast turnaround.

One question about the other half of the commit, --requant-gguf: does it work from an existing GGUF as input? Our production model is the -0731 checkpoint, and the blocker for us so far has been that deepseek4-quantize only builds from --hf DIR (~568 GB of safetensors, which does not fit next to the output on this box). If requantizing an existing AProjQ8-...-0731.gguf into AProjQ4 is supported, I'd be glad to try it and report back — that would let us run this variant on the checkpoint we actually serve, and would also give you a clean same-checkpoint quality delta, which my perplexity numbers above cannot provide.

[adamlawi]

Standing offer, so you don't have to ask: when you publish the GGUF for the newer DeepSeek checkpoint, I'll run the full suite on the DGX Spark and post the numbers here — --inspect, --decode-consistency, the ds4-bench prefill/decode sweep, perplexity, and the startup-span / free-memory measurements at 200k / 250k / 300k context. Same harness as the two runs above, so the results will be directly comparable.

That run would also finally give a clean same-checkpoint quality delta, which neither of my perplexity numbers so far can provide (the AProjQ4 file I tested is from the older checkpoint, and I don't have a matching AProjQ8 baseline for it).

The same applies to --requant-gguf whenever you get a chance to answer: if it accepts an existing GGUF as input, I'll requantize our production AProjQ8-…-0731.gguf and report back on that too.

No rush from my side — the machine is a stock DGX Spark that stays available, and a rebuild plus the whole measurement suite takes well under an hour. Just ping me here.

[GiorgioOppo]

The new gguf is online https://huggingface.co/antirez/deepseek-v4-gguf/tree/8268ad9384d11cbb63f472d097e499a9c59709ec. It's made by requantizing the q8. I know it's not the best choice, but on my hardware it's the only thing I can do.

[GiorgioOppo]

try use DS4_CUDA_NO_Q4_ATTN_GROUPED_TOK8=1 DS4_CUDA_NO_Q4_QKV_PAIR=1 in the next run

[adamlawi]

Ran 8b84e878 both ways on the DGX Spark. The two new paths cost ~38% of prefill — setting both env vars restores the 6c4d769d numbers exactly. Same binary, same prompt file, same model throughout.

ctx	A: default (new paths on)	B: both env vars =1	6c4d769d	AProjQ8 baseline
2048	495.15	797.21	783.60	817.32
4096	521.72	851.00	850.37	892.89
6144	518.51	843.21	841.37	883.59
8192	516.13	838.35	837.14	877.75

So DS4_CUDA_NO_Q4_ATTN_GROUPED_TOK8=1 DS4_CUDA_NO_Q4_QKV_PAIR=1 is a clean revert to the previous prefill performance — the regression is fully attributable to those two paths, not to anything else in the commit.

They also hurt numerics, which I did not expect:

variant	--decode-consistency 64 rms	max_abs
A: new paths on	0.307	1.529
B: env vars =1	0.185	0.942
6c4d769d	0.182	1.118
AProjQ8 baseline	0.293	1.476

With the new paths active the divergence is worse than the Q8_0 baseline; with them off it is back to the 6c4d769d value. That looks like a correctness signal on the grouped/paired path, not just a scheduling tradeoff.

The good news — decode improved, and it is not from these two paths:

ctx	decode A	decode B	6c4d769d	AProjQ8 baseline
2048	18.21	18.28	17.32	17.55
4096	15.59	15.64	14.90	15.08
6144	15.50	15.52	14.82	14.97
8192	15.35	15.37	14.67	14.83

Decode is ~+5% over 6c4d769d in both variants, i.e. the gain survives disabling the two paths and comes from the rest of the commit. That is the best decode I have measured on this machine — above the Q8_0 baseline, which is a notable result on its own.

One more thing worth a look: perplexity on the same 1882-token sample is 12.7836 (A) and 12.7904 (B), versus 12.7302 on 6c4d769d. So unlike prefill and rms, ppl does not return to the previous value when the env vars are set — a ~0.47% regression that appears to come from elsewhere in the commit. Also note A and B differ by 0.0068 here, while two runs of 6c4d769d agreed to within 0.0004, so there may be some nondeterminism in this build.

Summary from this machine: keep whatever gave the decode win, and either fix or gate the grouped-tok8 / QKV-pair paths — on GB10 they cost 38% of prefill and worsen decode-consistency. Happy to re-run anything on request.

(And the --requant-gguf question from my previous comment still stands whenever you have a moment — if it accepts an existing GGUF as input, I'd like to requantize our production -0731 model and give you a clean same-checkpoint comparison.)

[GiorgioOppo]

@adamlawi Feel free to try the requantization test

(And the --requant-gguf question from my previous comment still stands whenever you have a moment — if it accepts an existing GGUF as input, I'd like to requantize our production -0731 model and give you a clean same-checkpoint comparison.)

[GiorgioOppo]

@adamlawi could you run ./gguf-tools/quality-testing/score_official to compare the two quantizations?
these are my results on macos
q4.tsv
q8.tsv

[adamlawi]

--source-gguf works, and the result is better than I expected. I requantized our production -0731 model directly from GGUF — no safetensors needed — and then ran score_official on both. Summary: on a same-checkpoint comparison the AProjQ4 quality cost is +0.64% avg_nll, while decode gets faster than Q8_0 and the model frees ~13 GB of memory.

1. Requantization

./deepseek4-quantize \
  --source-gguf  DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
  --template     <same file> \
  --attention-proj q4_k \
  --out          ...-AProjQ4-SExpQ8-OutQ8-chat-v2-0731-requant.gguf

Converted 215 tensors q8_0 -> q4_K, output 84,420,584,032 bytes — 224 bytes off your published AProjQ4 file, so the layout matches exactly. --inspect confirms it: 130 × q8_0 (1.59 GiB), 215 × q4_K (2.41 GiB), everything else byte-identical to the source.

Two notes for anyone repeating this:

• The tool streams 16 rows at a time, so memory use is negligible — it ran fine alongside a live 80 GB ds4-server on the same box, only costing disk I/O.
• No imatrix was used. write_imatrix_kvs only records the imatrix filename and entry count in the KV block, not the vectors, so a requantized GGUF cannot recover them from its source. This is therefore a double quantization (Q8_0 → f32 → Q4_K) with unweighted Q4_K, i.e. a worse case than your imatrix-based path — which makes the quality numbers below a pleasant surprise.

2. score_official — as requested, plus your files for comparison

data/flash/manifest.tsv, ctx 4096, both env vars set to 1 (so the grouped-tok8 / QKV-pair paths from my previous comment are out of the picture):

	avg_nll	top1	first_match	avg_lcp
yours: Q8 (macOS, older ckpt)	0.632957	82.79%	44	3.830
yours: Q4 (macOS, older ckpt)	0.653162	82.88%	41	3.420
mine: Q8 -0731 (CUDA)	0.404811	85.86%	55	5.150
mine: Q4 requant -0731 (CUDA)	0.407383	85.34%	56	5.540

Q8 → Q4 delta: +3.19% for your pair, +0.64% for mine.

Please don't compare the absolute values across our two pairs — the data/flash fixtures were collected from the official DeepSeek V4 Flash 0731 API, so they match my checkpoint and not the older one. That alone explains most of the level difference (0.405 vs 0.633). Only the within-pair deltas are meaningful, and even those differ in backend and checkpoint.

Worth noting: on my pair first_match and avg_lcp actually go up slightly with Q4 (56 vs 55, 5.540 vs 5.150), while top1 drops 0.5 pp — so greedy behaviour is essentially preserved.

Classic perplexity on my own 1882-token sample agrees: 11.108 → 11.316, i.e. +1.87%. That also settles the confusion from my earlier comments — the +14.6% I reported before was almost entirely the checkpoint difference, not the quant.

3. Speed and memory on the requantized -0731

ctx	prefill Q8	prefill Q4	decode Q8	decode Q4
2048	817.32	789.63	17.55	18.41
4096	892.89	850.92	15.08	15.71
6144	883.59	842.99	14.97	15.61
8192	877.75	839.80	14.83	15.44

Decode is ~5% faster than Q8_0 (that gain is from the rest of 8b84e878, as reported earlier), prefill is ~4% slower. Startup spans 8.20 → 6.06 GiB, aligned artifacts 474/78.71 GiB → 259/74.16 GiB, and free memory after startup:

ctx	Q8 -0731	Q4 requant -0731
200000	9.8 GB	23.1 GB
300000	8.1 GB	20.3 GB

So on a 128 GB GB10 this variant now gives us --ctx 300000 with more headroom than --ctx 200000 had before, at +0.64% avg_nll and with faster decode. That is a straightforward win for long-context serving, and it is only possible because --source-gguf exists — thanks for adding it.

Happy to re-run any of this, or to try an imatrix-guided requantization if you think it is worth measuring the difference.

[GiorgioOppo]

Can you share th GGUF?

[adamlawi]

Final data point on the requantized -0731 model: a full 300k-token context window now fits comfortably on a 128 GB GB10, which was not possible with the Q8_0 attention projections.

Full-window load test, one session, prompt sized to fill the window, sampling free every 2s:

	Q8_0 baseline @ --ctx 200000	AProjQ4 requant @ --ctx 300000
filled	194,681 tok (97%)	293,390 tok (97.8%)
prefill time	265 s	449 s
min free memory at peak	2.5 GB	14.6 GB

So it is 1.5x more usable context with ~5.8x more headroom at the worst moment. The memory profile over the run: start 21.9 GB → min 14.6 GB → median 19.0 GB → end 17.4 GB. The minimum is a brief spike, not a plateau — we were within 1 GiB of it for only about 6 seconds. That matters here: on the Q8_0 model, --ctx 220000 left 0.94 GB for long enough that a cron job or a log rotation firing mid-prefill could push it over, so we never shipped it.

The model also answered correctly from the full 293k-token context (a sensible summary of the input), so this is not just an allocation test.

One caveat on the prefill number, since it looks lower than my earlier sweep: 293,390 tokens in 449 s is ~653 t/s effective, versus 840 t/s on the short bench frontiers. That is the attention cost growing with window length, not a regression — the Q8_0 baseline shows the same pattern (735 t/s when filling 200k, versus 878 t/s on the bench). Don't compare those two full-window numbers across the models, they cover windows of different sizes; the like-for-like prefill comparison is still the bench table (−4%).

Summary of where this lands for a 128 GB DGX Spark, all on the same -0731 checkpoint:

• +13.3 GB free memory at 200k, and 300k is now not just startable but fillable
• decode ~5% faster than Q8_0 (from the rest of 8b84e878)
• prefill −4%, quality +0.64% avg_nll / +1.87% ppl
• requantized locally from our own production GGUF via --source-gguf, no safetensors, no imatrix

For long-context serving on this hardware that is a clear win, and none of it would have been reachable without --source-gguf and the prefill fix. Thanks for turning both around so quickly.

Still happy to test the newer-checkpoint GGUF when you publish it, and to re-run the grouped-tok8 / QKV-pair comparison if you change those paths.