Repository navigation
QFN WV TokenWing Plugin v1.2 (Linux x86_64 .so + Windows x86_64 .dll)
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, tokenwing.quant_plugin.types) runtime & converter plugins (--quant-plugin) for running WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs out-of-the-box on runmyai/tokenwing dev (>= 6a3e018f) across both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VBMI+VNNI runtime dispatch, on-load slot repacking WV_PLUGIN_CPU_REPACK_MODE=1 + --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).
ABI Compatibility Note:
wv-plugin-v1.2implements TokenWing ABI v2 (112..117) forWV-v5-Sub2.5bpwandWV-v5-3.5bpwonrunmyai/tokenwingdev. For the legacy ABI v1 (43..48)WV-v5-2.26bpwcheckpoint, usewv-plugin-v1.1.
Highlights in wv-plugin-v1.2 (ABI v2)
- Zero-Loss
EXT2(114)VQ4_G32 + SWZ2_G64Support (WV-v5-Sub2.5bpw,39.48 GiBShard 1): Custom[4, 256, 4]int8 VQ4 codebooks + 1,152 Rank-8 SVD records, on-load VNNI slot repacking (WV_PLUGIN_CPU_REPACK_MODE=1),0.2157 ms/matmul(8-thread AVX-512 VNNI), and23.10 tok/sunder<12 GBVRAM +6 GiBRAM Direct-I/O NVMe streaming. - Complete 4-Row
vec_dot_x4AcrossEXT0..EXT5&WV_OVERRIDE_EXT4n_g=20Override Fix (WV-v5-3.5bpw53.48 GiBShard 1): Eliminates~3.78 GBof duplicatePAIRED_Q3override bloat ondown_exps, delivering0.3814 ms/matmul(EXT0+EXT2) /0.3820 ms/matmul(EXT0+EXT1, 8-thread AVX-512 VNNI) and19.51 tok/sunder<12 GBVRAM Direct-I/O NVMe streaming. - Portability & Symbol Hygiene: Linux
.sobuilt onubuntu-22.04(GLIBC <= 2.34), Windows.dllbuilt withMSVC 19.44 + CUDA 12.8, staticcudart, singleggml_quant_plugin_get_apiC-ABI export.
SHA-256 Checksums
b2c3addeda808a635300f57b870615a99df690fa3f49d702d5c5a99668d73722 libqfn_wv_plugin.so
e820d38ad4736da54fc8fa3244c04a478336226d07652f5893a070154378f155 qfn_wv_plugin.dll