ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) - #148
Open
bri-prism wants to merge 14 commits into
Open
ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0)#148bri-prism wants to merge 14 commits into
bri-prism wants to merge 14 commits into
Conversation
Upstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a single fp16 scale per 256 weights. The ternary checkpoints this targets carry a scale per 128, so a 256-wide scale has to discard one of the two group scales it straddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern is preserved exactly, but the smaller-alpha group of each block is reconstructed at the larger alpha, giving 6.43% weight error that passes into activations at full strength because it is multiplicative on the weight rather than noise. PTQ1_0 keeps TQ1_0's trit packing and our group of 128: qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw against PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of information: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits. test-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute quantization error 0.008678 and dot product error 0.141111, versus tq1_0's 0.008681 and 0.141345. A round trip of real folded ternary weights is bit-identical to the PQ2_0 reconstruction, so the only error either format carries is the shared fp16 scale rounding. A distinct type id (143) rather than changing TQ1_0 in place, for the same reason PQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and reinterpreting its block width would silently misread every existing TQ1_0 gguf. TQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the stages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly its original behaviour. Scope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so llama-quantize can target it. The vec_dot decodes a block to element order before the dot, sharing the traversal with dequantize so the two cannot drift; it is generic C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels, and the repack gemv/gemm paths, are not implemented.
Three plumbing gaps the traits entry alone did not cover, each of which failed
end-to-end quantization in a different way:
ggml_quantize_chunk - without a dispatch case the returned byte count did not
match the row size and GGML_ASSERT(result == nrows *
row_size) aborted a worker thread mid-tensor, leaving a
truncated output file behind.
ggml_ftype -> wtype - the ftype could not resolve to a ggml type.
validate_row_data - "invalid type 143", so quantization refused the result
it had just produced.
Verified end to end on a 1.7B ternary model with the embedding and output tensors
pinned to f16 in both runs, so only the quantized tensors differ:
quantized tensors PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller
(1.75/2.125 bpw predicts 17.65%)
perplexity, 4 chunks at ctx 512, identical to every printed digit:
per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,
final 12.6944 +/- 1.12596 for both
The perplexity run goes through the new CPU vec_dot, so that path is exercised
rather than only the codec.
Adds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both
dequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations
for f32 and f16, get_rows, the simdgroup constants and both pipeline-selection
switches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently
skipped the new type and reported a clean pass.
Correct: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on
a 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed
digit, 12.6935 +/- 1.12567 for both.
Slow, and the reason is structural. Measured on M5 Pro:
PQ2_0 PTQ1_0
pp128 4505 tok/s 2590 tok/s 0.57x
tg32 182.5 tok/s 43.6 tok/s 0.24x
Base-3 packing makes element order non-positional, so a thread owning 16
contiguous weights cannot index bytes directly, and threads owning different trit
indices reload the same bytes. Hoisting the trit factor out of the inner loop for
spans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap
needs the thread mapping changed so one thread consumes all five trits of a byte
it loads once, instead of five threads each loading it for one trit.
This reproduces what the earlier TQ1_0 work already found on other backends:
base-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay
for itself at decode without that restructure. The 17.6% file-size reduction is
real and lossless; the speed is not there yet.
Four changes, each measured on Apple M5 Pro against a 1.7B ternary model, with
MUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:
1. byte-owning threads 29.8 -> 136.4 a thread now consumes all five
trits of a byte it loads once, instead of eight threads each reloading the
same qs bytes for one trit index; the block's 26 bytes are read once total
2. y staged in registers 136.4 -> 143.5 y was being re-read from device
memory once per row; it is now gathered once per block and reused, with the
ternary offset folded into a single sumy subtraction
3. N_R0 8 -> 4 143.5 -> 163.8 rows-per-thread was the real
limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163
4. balanced qh 163.8 -> 165.3 qh's 8 elements went to threads
0 and 1 only, so those did 19 elements while the rest did 15 and the whole
simdgroup waited; one element per thread makes it 16 each and drops yl to 16
Trit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with
v = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was
tried and is not better for decode: it adds a dependent load per byte, 26 extra
per block against PQ2_0's 32 total loads. It does help the mul_mm path, which
must produce 16 contiguous elements and cannot use the byte-owning mapping, so
dequantize keeps the table.
Final, PQ2_0 vs PTQ1_0 on M5 Pro:
pp128 4541 -> 3998 tok/s 0.88x
tg32 186 -> 165 tok/s 0.89x
size 357 -> 294 MiB 17.6% smaller, and lossless
Prefill's remaining gap is structural: mul_mm's dequantize contract is 16
contiguous elements, so five threads still touch the same 16 bytes for different
trit indices. Closing it needs mul_mm itself to stage a decoded block in
threadgroup memory, which is a larger change than this.
Adds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit accessor, the float2 dequantize entry point, the six dequant dispatch sites in convert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch sites, and both supports_op type lists. **This code has never been compiled or executed.** The machine it was written on is an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be built and run on a CUDA host before anyone relies on it. What *is* verified is the part where a bug would actually hide. The device accessor's element mapping is pure integer logic, so it was transcribed verbatim to the host and checked against the CPU codec's own traversal over 2,560,000 element positions across 20,000 random blocks: zero mismatches. That check is kept as tests/test-ptq1_0-element-map.cpp so it can be rerun. Trit extraction uses the same base-3 remainder recurrence as the Metal path, t = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and packs them into an int as signed int8 lanes so accumulation still goes through dp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk. MMQ is deliberately not implemented. Types without it fall back to dequantize plus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ tile loader blind against tile layouts I cannot test would be worse than leaving the fallback in place.
N_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,
so 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise
(+/- 3.4 on a single measurement).
Five further optimisations were tried and MEASURED, and four of them lost. Kept
here so nobody spends the same day twice:
256-entry trit lookup table 142.1 vs 143.5 recurrence no decode gain
(helps mul_mm, which keeps it)
independent pow3 extraction 159.8 vs 166.0 recurrence WORSE
contiguous byte ownership 162.0 vs 165.3 scattered WORSE
N_R0 = 3 / 5 / 6 129.8 / 156.7 / 149.6 all worse than 4
N_SG = 8 161.1 worse than 1-2
What that rules out. The lookup table shows decode is not bound by trit
arithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep
serial dependency either -- four rows per thread already supply enough
independent work, so fewer ops beat more ILP. Contiguous byte runs made it worse
because thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte
branch cost more than consolidating the loads saved.
Where the remaining 11% actually lives, per 128 weights:
PQ2_0 128 x (shift, mask, cvt, fma) = 512 ops, 34 B read
PTQ1_0 128 x (mul, shift, cvt, fma, mask-for-next) = 640 ops, 28 B read
ALU 1.25x, bytes 0.82x
A purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at
1.21x. Measured 0.89x sits between them, which is about where this instruction
and byte mix should land. Extracting a base-3 digit is inherently a multiply
where a 2-bit field is a shift, so this gap is the price of packing at the
1.585-bit information limit rather than on a 2-bit boundary, not an
implementation defect.
The one lever left with a real mechanism behind it is prefill, not decode. In
decode each row has its own weights so there is nothing to share, but mul_mm
reuses a weight tile across many output columns, and its dequantize contract
forces 16 contiguous elements, so five threads still touch the same 16 bytes.
Staging a cooperatively decoded block in threadgroup memory would remove that
redundancy outright. Prefill is at 0.87x (3946 vs 4492).
Perplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.
Self-review compared every switch that names PQ2_0 against those naming PTQ1_0 and found the new type absent from ten support enumerations: seven in ggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in ggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible with PQ2_0 there -- same block size, same to_float and from_float shape -- so it belongs in each. Closing those gaps then exposed a real defect. Adding the type to the Metal support lists enabled the mul_mv_ext path for wider n, which immediately failed with "kernel not found in any metal library" for kernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only r1_2 through r1_4 had been written, because the grep that enumerated them was truncated by head. Instantiating r1_5 fixes it, and the instantiation set is now derived from PQ2_0's rather than transcribed. Worth noting the earlier 1251/1251 passes were not wrong, they were narrow: with the type missing from the support lists the wider-n path was never selected, so the absent kernel could not be reached. The gap and the defect concealed each other. After: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical to PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.
The CUDA path still cannot be compiled here, but its arithmetic can be checked. This stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the 4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor verbatim, and compares them against a float reference built by dequantising the block and the q8_1 activations independently. Result over 5000 random weight blocks with random q8_1 activations: exact integer-accumulator checks 20000, mismatches 0 error / sum of term magnitudes 2.399e-08 The integer test is the one that matters: the kernel accumulates in int via dp4a before scaling, so if the trit extraction, the int8 lane packing or the chunk-to- q8_1 pairing were wrong, that accumulator would differ. It does not, on any of the 20000 chunks. A first pass reported 7.062e-05 "relative error" and looked like a failure. That was cancellation: the reference total is a signed sum that can land near zero, so dividing by it inflates the ratio. Measured against the sum of term magnitudes instead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32 rounding of the four per-chunk products. Together with tests/test-ptq1_0-element-map.cpp this leaves only compilation unproven for the CUDA path, not correctness of its math.
…DS NOW
First compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and
CMAKE_CUDA_ARCHITECTURES=120, failed with
vecdotq.cuh(822): error: identifier "ptq1_0_trit" is undefined
The accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.
PQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with
byte_perm -- so nothing in the file it was mirrored from exposed the dependency.
Both headers include common.cuh, so the helper belongs there.
Verified on the 5090 after the fix:
test-backend-ops -o MUL_MAT 1283/1283 passed, Backend CUDA0: OK
45 ptq1_0 cases OK, 0 failures
host element-map test 2,560,000 positions, 0 mismatches
host MMVQ dot test 20,000 integer checks, 0 mismatches
The two host tests were already green before this, which is the point worth
recording: they verified the arithmetic and could not have caught a missing
declaration. Compilation was the only gate that would find this, and the earlier
commit saying the CUDA path had never been built was accurate about the risk.
…mitted Adds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit accessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone dequant shader, the shader-generator registration, and 16 pipeline sites in ggml-vulkan.cpp mirrored from Q1_0's. Mirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan kernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still declares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a group-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0 would have inherited that. Q1_0 also has no MMVQ entry, so this follows the same dequantize-based path rather than inventing one. One divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block (qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C field order exactly. Getting that wrong misindexes every block silently instead of failing, which is the same class of error as the Q2_0 breakage above. Not wired, and it is a real gap rather than an oversight: on-device quantization (cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL, which is a separate piece of work and is not needed to run a model -- weights are quantized offline by llama-quantize. **This has not been compiled or run.** There was no Vulkan toolchain on the machine it was written on. It is pushed so it can be built and tested on a GPU box, and it should be treated as unverified until test-backend-ops says otherwise.
There was a problem hiding this comment.
🔵 Needs a closer look
The new wire format spans multiple backends, while CUDA remains uncompiled and contains a dispatch fallthrough.
Pull request overview
Adds the 1.75-bpw PTQ1_0 ternary format and integrates it across quantization, loading, CPU, Metal, CUDA, and Vulkan paths.
Changes:
- Defines PTQ1_0 encoding, quantization, dequantization, and CPU dot products.
- Adds Metal, CUDA, and Vulkan kernels and dispatch.
- Extends quantization and backend tests.
File summaries
| File | Description |
|---|---|
tools/quantize/quantize.cpp |
Exposes PTQ1_0 quantization. |
tests/test-quantize-fns.cpp |
Adds PTQ1_0 error thresholds. |
tests/test-ptq1_0-element-map.cpp |
Adds host-side mapping checks. |
tests/test-ptq1_0-cuda-dot.cpp |
Adds host-side CUDA dot checks. |
tests/test-backend-ops.cpp |
Includes PTQ1_0 in backend tests. |
src/llama-quant.cpp |
Adds type selection and fallback plumbing. |
src/llama-model-loader.cpp |
Recognizes PTQ1_0 models. |
include/llama.h |
Defines the llama file type. |
ggml/src/ggml.c |
Registers traits and quantization dispatch. |
ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp |
Generates PTQ1_0 Vulkan shaders. |
ggml/src/ggml-vulkan/vulkan-shaders/types.glsl |
Defines the Vulkan block layout. |
ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_funcs.glsl |
Adds Vulkan matrix-load decoding. |
ggml/src/ggml-vulkan/vulkan-shaders/dequant_ptq1_0.comp |
Implements Vulkan dequantization. |
ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl |
Adds shared Vulkan decode helpers. |
ggml/src/ggml-vulkan/ggml-vulkan.cpp |
Registers Vulkan pipelines. |
ggml/src/ggml-quants.h |
Declares codec functions. |
ggml/src/ggml-quants.c |
Implements the PTQ1_0 CPU codec. |
ggml/src/ggml-metal/kernels/quantize.metal |
Adds Metal row extraction. |
ggml/src/ggml-metal/kernels/mul_mv.metal |
Implements Metal matrix-vector kernels. |
ggml/src/ggml-metal/kernels/mul_mm.metal |
Instantiates Metal matrix kernels. |
ggml/src/ggml-metal/kernels/dequantize.h |
Adds Metal decoding helpers. |
ggml/src/ggml-metal/ggml-metal-ops.cpp |
Enables Metal matrix operations. |
ggml/src/ggml-metal/ggml-metal-impl.h |
Defines Metal tuning parameters. |
ggml/src/ggml-metal/ggml-metal-device.m |
Reports Metal operation support. |
ggml/src/ggml-metal/ggml-metal-device.cpp |
Selects Metal pipeline geometry. |
ggml/src/ggml-cuda/vecdotq.cuh |
Implements CUDA PTQ1_0 dot products. |
ggml/src/ggml-cuda/mmvq.cu |
Dispatches CUDA MMVQ. |
ggml/src/ggml-cuda/ggml-cuda.cu |
Enables CUDA operation support. |
ggml/src/ggml-cuda/getrows.cu |
Adds CUDA row extraction. |
ggml/src/ggml-cuda/dequantize.cuh |
Adds CUDA dequantization. |
ggml/src/ggml-cuda/convert.cu |
Registers CUDA conversion paths. |
ggml/src/ggml-cuda/common.cuh |
Defines CUDA traits and trit access. |
ggml/src/ggml-cpu/quants.h |
Declares CPU quantization operations. |
ggml/src/ggml-cpu/quants.c |
Implements generic CPU dot products. |
ggml/src/ggml-cpu/ops.cpp |
Enables PTQ1_0 CPU operations. |
ggml/src/ggml-cpu/ggml-cpu.c |
Registers CPU type traits. |
ggml/src/ggml-cpu/arch-fallback.h |
Selects generic CPU fallback. |
ggml/src/ggml-common.h |
Defines the PTQ1_0 block format. |
ggml/include/ggml.h |
Adds public type and file-type IDs. |
Review details
- Files reviewed: 39/39 changed files
- Comments generated: 5
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+436
to
+437
| GGML_TYPE_PTQ1_0 = 143, // Prism-private ternary, group 128 | ||
| GGML_TYPE_COUNT = 144, |
Comment on lines
+38
to
+40
| // ---- transcribed from ggml-cuda/vecdotq.cuh ------------------------------ | ||
| static inline float vec_dot_ptq1_0_q8_1(const void* vbq, const block_q8_1* bq8_1, | ||
| const int& kbx, const int& iqs) { |
Comment on lines
+12
to
+13
| // --- transcribed verbatim from ggml-cuda/dequantize.cuh --- | ||
| static int ptq1_0_trit(const block_ptq1_0 * x, const int e) { |
| "f32", | ||
| "f16", | ||
| "q1_0", | ||
| "ptq1_0", |
…ower
Benchmarked the CUDA path for the first time. It is correct and slow, and the
numbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against
pq2_0 on the same run.
n=1 n=2 n=3 n=8 n=512
q1_0 8.88 9.82 13.05 30.14 225.4
pq2_0 9.75 11.05 14.92 31.27 232.4
ptq1_0 46.8 64.6 99.1 287.3 525.6
The per-chunk version asked "which byte holds element e" per element, a branch plus
a variable-length recurrence, and iqs differs per lane so the branch diverged. Taking
the whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26
bytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.
So divergence was not the dominant cost, and the comment now says so. What remains is
load shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,
where pq2_0 issues eight dp4a calls over 4-byte words.
The fix is identified but deliberately not guessed at. Below element 80 the element
stride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES
at the same trit index. Four recurrences packed into one int would feed dp4a against a
4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this
session has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that
predictions about where GPU time goes are wrong about as often as they are right.
Correctness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090
and an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.
Note n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,
which is consistent with the problem being in the MMVQ load pattern specifically.
…rop cm2 pipelines The Vulkan commit did not build. Three errors, one root cause each, all found by a teammate on an Intel Arc B390 and independently by windows CI: 1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which falls back to the scalar and coopmat1 paths that are actually implemented. 2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour earlier, same helper, different backend**: defined in the dequantize chain, called from the matmul chain. Fixed the same way, with a shared header vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not duplicated: two copies would have to stay in step with the CPU codec in ggml-quants.c, and a divergence there appears as wrong matmul results, never as a build error. 3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2 pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a comment recording that the absence is deliberate. (1) and (3) had to be fixed together or one error just trades for the other. Verified locally this time, which is the other half of the lesson: glslc, molten-vk, vulkan-headers and vulkan-loader were already installed on the machine that wrote the original commit. It was pushed unverified on the false assumption that no toolchain was present, and a local build finds all three in minutes. -DGGML_VULKAN=ON builds clean SPIR-V generated 1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2 test-backend-ops MUL_MAT 1009/1009, Backend Vulkan0: OK, no regression That local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case "not supported", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an Apple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness comes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases actually executed and 0 failures, on KHR_coopmat. Still absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq), coopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.
The PTQ1_0 pipelines were created but the type was absent from every selection and support enumeration, so supports_op returned false, the scheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the generated shaders were unreachable. The build looked green while nothing executed on the backend. Add the type alongside GGML_TYPE_Q1_0 in the five pipeline getters and the two supports_op gates. Adding it there alone is not safe, because PTQ1_0 is the only type in those lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no decoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots are still allocated by vk_matmul_pipeline2's constructor, so the mm getter would return a non-null pipeline whose l/m/s are all null, and the caller's "mmp == nullptr" fallback would not fire; the mul_mat_id getter would instead trip GGML_ASSERT(support_fp32acc). Both getters now return nullptr when the cm2 slots are empty, which routes the caller to the dequant + f16 matmul fallback it already implements. That path needs ggml_vk_get_to_fp16, which is one of the gates opened above. The cpy, SET_ROWS and DUP gates are deliberately left alone: copy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on device is unimplemented and a refusal there is correct. Verified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78 declined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case for case, byte-identical across two runs. The coopmat2 guard is reasoned from the pipeline-creation structure and is not covered by that run, which is a coopmat1 device; it has no coopmat2 hardware behind it yet.
Author
|
The "Not included" section is stale and will mislead a reviewer. It says:
But the PR now carries 201 Vulkan additions across 8 files: The Vulkan commits landed after that section was written. Suggest updating it, since "no Vulkan support" understates what needs review by the largest single backend in the diff. Verified Vulkan state, for the recordTested on Intel Arc B390 (KHR_coopmat, proprietary Windows driver),
So the PTQ1_0 Vulkan kernel is numerically correct on Xe2. Two caveats a reviewer should carry:
|
Assisted-by: OpenAI Codex
Author
|
Edited the PR body — flagging since it isn't mine, and the author's session is offline. Three changes, all corrections of things that had gone stale as the branch moved; nothing else touched.
I kept the "untested" and "not included" claims that are still true, and did not soften anything. If any of it misreads your intent, revert freely — the substance is in the commits, not my summary of them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
PTQ1_0, a ternary weight type at group size 128: base-3 trit packing at fivetrits per byte, with one fp16 scale per 128 weights.
against
PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller.Why a new type rather than changing TQ1_0
Upstream
TQ1_0already packs ternary base-3, but carries one scale per 256weights. Ternary checkpoints with a scale per 128 cannot be represented by it: a
256-wide scale straddles two group scales and has to discard one. Measured on such a
checkpoint, every trit survives and the zero pattern is preserved exactly, but the
smaller-scale group of each block is reconstructed at the larger scale, giving 6.4%
weight error that passes into activations at full strength because it is multiplicative
on the weight rather than additive noise.
TQ1_0is also a wire format, so redefining its block width would silently misreadevery existing
TQ1_0gguf. A distinct type id coexists instead, the same reasoningthat keeps
PQ2_0separate from upstream's group-64Q2_0.The saving is real because
PQ2_0spends two bits storing a value that carrieslog2(3) = 1.585 bits of information. Base-3 packing recovers that waste; the group-128
scale costs 0.125 bits per weight.
Lossless, not approximately
test-quantize-fnsreportsptq1_0identical topq2_0on both metrics, absolutequantization error 0.008678 and dot product error 0.141111, where
tq1_0gives0.008681 and 0.141345.
A round trip of real folded ternary weights is bit-identical to the
PQ2_0reconstruction, so the only error either format carries is the shared fp16 scale
rounding.
End to end on a 1.7B ternary model, embedding and output pinned to f16 in both runs so
only the quantized tensors differ:
Identical to every printed digit, per chunk as well as final.
Backends
CPU — codec, ggml traits,
vec_dotagainst Q8_0, and the ftype plumbing sollama-quantizecan target it. Generic C on every architecture, no SIMD path yet.Decode measures 82.8 tok/s against
PQ2_0's 99.Metal — element accessor, both dequantize entry points, mat-vec,
mul_mmandmul_mm_idfor f32 and f16,get_rows, and both pipeline switches. 1251/1251MUL_MATtests pass on an M5 Pro.Decode started at 29.8 tok/s and reached 162-166 over four changes: byte-owning thread
ownership so a thread consumes all five trits of a byte it loads once; y staged in
registers per block with the ternary offset folded into one subtraction; rows-per-thread
4 rather than 8, which was the dominant knob; and balancing
qhone element per thread.Five further attempts lost and are recorded in the commit messages so they are not
repeated.
CUDA — type traits, device accessor, dequantize, the convert and
get_rowsdispatches, MMVQ with dp4a, and both
supports_oplists. Compiled and verified on hardware(RTX 5090, CUDA 12.8,
CMAKE_CUDA_ARCHITECTURES=120):test-backend-ops -o MUL_MAT1283/1283 with 45 ptq1_0 cases and 0 failures. This section previously said the CUDA path
had never been compiled; that was true when written and is no longer. The fix was moving
ptq1_0_trittocommon.cuhsovecdotq.cuhcan see it. Also verified is the deviceaccessor's element mapping, which is pure integer logic and was checked against the CPU
codec over 2,560,000 element positions across 20,000 random blocks with zero
mismatches, and the MMVQ dot product itself, checked by stubbing
__dp4a,__low2floatand the 4-byte int load and comparing against an independentlydequantized float reference: 20,000 exact integer-accumulator comparisons across
5000 random blocks, zero mismatches, residual error 2.4e-08 against the sum of
term magnitudes. Both ship as
tests/test-ptq1_0-element-map.cppandtests/test-ptq1_0-cuda-dot.cpp.So the CUDA arithmetic is verified even though the kernels are not built. What
remains unproven is compilation, not correctness. MMQ is not implemented, so
prefill falls back to dequantize plus cuBLAS.
Vulkan —
dequant_ptq1_0.comp,ptq1_0.glsl, and the dequant/mul_mm/get_rowsplumbing, plus the selection and
supports_openumerations. Verified on hardware(Intel Arc B390, KHR_coopmat, proprietary Windows driver):
test-backend-ops -b Vulkan0 -o MUL_MATgives 28 OK / 0 FAIL for ptq1_0, matching q1_0 and q2_0 exactly. 28 is thefull available set on this device — the other 50 are shape and permutation variants that
every quantized type declines here. Two runs, byte-identical.
Getting there took three build fixes (
599d0b876) and one reachability fix(
705d9ccda). Worth recording why the reachability bug was invisible: every pipeline wasbeing created and 54 SPIR-V shaders generated, but PTQ1_0 was absent from seven selection
and
supports_openumerations, sosupports_opreturned false and the scheduler neveroffered the op to Vulkan. The suite reported a clean
1009/1009while executing nothing —every ptq1_0 line said
not supported. Count OK results, not test-case lines, andkeep a known-good type in the same run as a control.
No coopmat2:
dequant_funcs_cm2.glslhas no PTQ1_0 decoder, so cm2 generation is skippedfor the type and its cm2 pipeline registrations are removed.
705d9ccdaguards the emptycm2 slots with
is_empty()rather than a null check, becausevk_matmul_pipeline2'sconstructor
make_shareds both accumulators — an unregistered family is a valid pointerwhose
l/m/sare null, sommp == nullptrnever fires. That guard is reasoned from thecreation structure and is untested: the B390 is KHR_coopmat and takes the other
branch, so exercising it needs a device with
VK_NV_cooperative_matrix2.mul_mat_vecq_funcs.glslis deliberately untouched —ptq1_0matches none of thegenerator's q8_1 predicates, so no MMVQ shader is emitted and nothing can select one.
Consistent absence, not an oversight.
Not included
No SIMD
vec_dot. No MMQ. No coopmat2 on Vulkan (see above). Metal prefill's remaining 12% needsmul_mmto stage a cooperatively decoded block in threadgroup memory, since itsdequantize contract forces 16 contiguous elements and five threads therefore still touch
the same 16 bytes.