Skip to content

QFN WV TokenWing Plugin v1.1 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 04:55
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 1 runtime & converter plugins (--quant-plugin) for running WV GGUFs (WV-v5-2.26bpw and WV-v5-3.5bpw EXT0..EXT5) in TokenWing on both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VNNI runtime dispatch, on-load slot repacking --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).

Highlights in wv-plugin-v1.1

  • 6 External Quant Types (EXT0..EXT5): Adds wv_swz3_g16 (3.59 bpw), wv_vq4_u8 (2.26 bpw cold MoE + native down_exps), wv_swz4_r12 (4.50 bpw rotated spine), and wv_swz6_r12 (6.06 bpw rotated LM head / critical spine).
  • On-Load I/O-Thread VNNI Repack + vec_dot_x4: Background NVMe reader threads pre-swizzle streamed 2.26 bpw slices once on arrival (0.1418 ms/matmul 10-expert / 0.0999 ms/matmul 7-expert 8-thread AVX-512 VNNI).
  • Zero-Bank-Conflict SM120 CUDA Kernels & Speculative MTP: 71.42 tok/s full-VRAM GPU decode with --spec-type draft-mtp (55.74 tok/s B=1, 103.34 tok/s pp512), and 27.77 tok/s (tech) / 26.50 tok/s (code) under <11 GiB VRAM + 6 GiB RAM cold O_DIRECT NVMe streaming (82.35 MiB/tok with H=96 hybrid VRAM experts).
  • Portability & Symbol Hygiene: Linux .so built on ubuntu-22.04 (GLIBC <= 2.34), Windows .dll built with MSVC 19.44 + CUDA 12.8, static cudart, single ggml_quant_plugin_get_api C-ABI export.

SHA-256 Checksums

2285d62f00595e22de9d44e3e03ba0ef0bd07efd66ff90aab83119eae3fb30c9  libqfn_wv_plugin.so
6a62a3193019ba4e9e4e8c7cacc2d9d2e8f1dfca75f1985d6651d977ffcb0f80  qfn_wv_plugin.dll