Repository navigation
Verifying Q4_K scale decode: uint32 bit-op approach vs get_scale_min_k4 equivalence #1543
Replies: 1 comment
|
The memcpy does not read 16 bytes. It copies exactly 12 bytes into utmp[0], utmp[1], and utmp[2]. utmp[3] is uninitialized at that point, and the next line explicitly assigns it from utmp[2], utmp[1], and utmp[0]. No byte from qs is read. No out-of-bounds access happens. memcpy(utmp, x[i].scales, 12);
utmp[3] = ((utmp[2] >> 4) & kmask2) | (((utmp[1] >> 6) & kmask3) << 4);The 12 bytes cover the full scales field. As you noted, block_q4_K is d, dmin, scales[12], qs[128] — the copy must start at the scales field, not before it. The bit ops are little-endian only. Your utmp mapping is correct: utmp[0] bytes are sc[0..3] utmp[1] bytes are sc[4..7] utmp[2] bytes are mn[0..3] utmp[3] bytes are mn[4..7] This is exactly the get_scale_min_k4 logic. For the min correction, your pairing is also correct. |
Uh oh!
There was an error while loading. Please reload this page.
I'm writing a minimal C99 LLM inference engine and implementing a fused Q4_K matvec kernel. I have two open questions I want to verify before trusting the output.
Scale decode
My AVX2 path uses the uint32 bit-manipulation approach from
ggml_vec_dot_q4_K_q8_K:After this I treat the 16 bytes of
utmpas:utmp[0]bytes = 6-bit scales sc[0..3]utmp[1]bytes = 6-bit scales sc[4..7]utmp[2]bytes = 6-bit mins mn[0..3]utmp[3]bytes = 6-bit mins mn[4..7]Then
_mm256_cvtepu8_epi16loads all 16 as int16 in one register.My scalar path uses
get_scale_min_k4(thej < 4/j >= 4branch). Both produce values that look close to the F32 reference path but I want to confirm they are exactly equivalent before relying on the optimized path.Question 1: Is the
memcpy-as-4-uint32s approach safe on any little-endian target, or is there a subtle byte-ordering assumption specific to x86? The 12 bytes don't divide evenly into 3 uint32s — it loads 16 bytes (4 × uint32) from a field that is only 12 bytes wide. Is it relying on theqsfield immediately following in memory to supply the last 4 bytes, and those bits getting masked away?Min correction / bsums grouping
For the min correction I compute:
So
mn[j]is paired with activation elementsj*32 .. j*32+31.Question 2: Does min group
jin Q4_K correspond exactly to a contiguous window of 32 activation elements, and is that windowj*32 .. j*32+31? Or does the group structure follow the nibble interleaving (lo-nibbles of groupgfirst, then hi-nibbles), in which case the bsums pairing above would be wrong?Any correction or confirmation welcome — happy to share the full scalar and AVX2 paths if useful.
All reactions