Skip to content

QFN WV TokenWing Plugin v1.2 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 15:59
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, tokenwing.quant_plugin.types) runtime & converter plugins (--quant-plugin) for running WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs out-of-the-box on runmyai/tokenwing dev (>= 6a3e018f) across both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VBMI+VNNI runtime dispatch, on-load slot repacking WV_PLUGIN_CPU_REPACK_MODE=1 + --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).

ABI Compatibility Note: wv-plugin-v1.2 implements TokenWing ABI v2 (112..117) for WV-v5-Sub2.5bpw and WV-v5-3.5bpw on runmyai/tokenwing dev. For the legacy ABI v1 (43..48) WV-v5-2.26bpw checkpoint, use wv-plugin-v1.1.

Highlights in wv-plugin-v1.2 (ABI v2)

  • Zero-Loss EXT2 (114) VQ4_G32 + SWZ2_G64 Support (WV-v5-Sub2.5bpw, 39.48 GiB Shard 1): Custom [4, 256, 4] int8 VQ4 codebooks + 1,152 Rank-8 SVD records, on-load VNNI slot repacking (WV_PLUGIN_CPU_REPACK_MODE=1), 0.2157 ms/matmul (8-thread AVX-512 VNNI), and 23.10 tok/s under <12 GB VRAM + 6 GiB RAM Direct-I/O NVMe streaming.
  • Complete 4-Row vec_dot_x4 Across EXT0..EXT5 & WV_OVERRIDE_EXT4 n_g=20 Override Fix (WV-v5-3.5bpw 53.48 GiB Shard 1): Eliminates ~3.78 GB of duplicate PAIRED_Q3 override bloat on down_exps, delivering 0.3814 ms/matmul (EXT0+EXT2) / 0.3820 ms/matmul (EXT0+EXT1, 8-thread AVX-512 VNNI) and 19.51 tok/s under <12 GB VRAM Direct-I/O NVMe streaming.
  • Portability & Symbol Hygiene: Linux .so built on ubuntu-22.04 (GLIBC <= 2.34), Windows .dll built with MSVC 19.44 + CUDA 12.8, static cudart, single ggml_quant_plugin_get_api C-ABI export.

SHA-256 Checksums

b2c3addeda808a635300f57b870615a99df690fa3f49d702d5c5a99668d73722  libqfn_wv_plugin.so
e820d38ad4736da54fc8fa3244c04a478336226d07652f5893a070154378f155  qfn_wv_plugin.dll