Repository navigation
A 1.60 bpw (5 trits/byte) layout next to I2_S and TL2: feedback on the kernel design #686
Lukáš Velecký (velurex)
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi Eddie-Wang (@Eddie-Wang1120), Hansong Zhou (@potassiummmm), tsong-ms,
I would like feedback from the bitnet.cpp kernel authors on a ternary weight layout I have built, before I propose it anywhere else.
Layout. Five trits per byte, 1.60 bpw, one scale per tensor:
The lower three digits never exceed 26, so B = L + 28*U, and U = floor(B/28) needs only shifts and one rounding byte average (
vpavgb/vrhadd); on AVX-512 VBMI it is twovpermb. U (9 values) and L (27 values) index activation lookup tables directly, in 32-row tiles (AVX2, NEON) and 64-row tiles (AVX-512). Activations are split asx = 16*hi + lo, so the int8 dot product is exact.Parity. The baseline is a faithful Rust port of your I2_S MAD kernel. The runtime reproduces the 256 greedy tokens and the checked logits of bitnet.cpp on Falcon3-10B-Instruct-1.58bit bit for bit (x86-64 and AArch64).
Kernel results (Falcon3-10B projection shapes, weights from DRAM):
It is faster where I2_S is close to the memory ceiling and slower where it is compute-bound; the full table and the method are in the benchmarks document.
Questions:
Thanks for bitnet.cpp, and for making the reference implementation easy to verify against.
All reactions