Skip to content

QFN WV TokenWing Plugin v1.3 (Linux x86_64 .so + Windows x86_64 .dll)

Latest

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 20:39
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, plugin_version = 1.3.0) drop-in runtime & converter plugins (--quant-plugin) for WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs on runmyai/tokenwing dev across Linux x86_64 (GLIBC <= 2.34) and Windows x86_64 (MSVC 19.44 + CUDA 12.8, static cudart).

What's New in wv-plugin-v1.3 (1.3.0, ABI v2 Drop-In Upgrade)

  • Hardware F16C (_mm256_cvtph_ps / vcvtph2ps) Across All AVX2 & AVX-512 / VNNI Hot Loops: Replaces scalar bit-manipulation FP16->FP32 conversions inside EXT0..EXT5 vec_dot and 4-row vec_dot_x4 (including 8-lane batch scale decode in VNNI EXT2/EXT3), speeding up 8-thread Sub-2.5bpw VNNI vec_dot_x4 from 0.4387 -> 0.2615 ms/layer (0.1307 ms/matmul, 1.68x faster than v1.2.0 VNNI and 2.88x faster than un-repacked vec_dot).
  • SIMD-Vectorized 1-Row & 4-Row Rank-8 Residual SVD (AVX-512 + AVX2): Vectorizes U^T y_f32 (8-col dot + vcvtph2ps) and 4-row shared V[r0..r3] accumulation, cutting Rank-8 SVD layer time on Sub-2.5bpw (0.4358 -> 0.2817 ms/layer, 1.55x) and 3.5bpw (0.7876 -> 0.7085 ms/layer).
  • Simultaneous GPU Pool + CPU VNNI Repack + CUDA Overlap Capability Probe (repack_expert_slot(type, NULL, NULL, -1, 0) == 1): Paired with runmyai/tokenwing PR #9 (also attached here as tokenwing_repack_cuda_overlap.patch), eliminates the WV_PLUGIN_CPU_REPACK_MODE=1 workaround so GPU whole-MoE layers, multi-wave prefill (63.3 tok/s), CUDA overlap, and 120/120 VNNI-repacked CPU pools run together.
  • Measured < 12 GB VRAM Laptop Streaming (20 GiB RAM pool, -t 10, --spec-type ngram-mod --spec-ngram-mod-n-max 8): 37.34–37.75 tok/s (38.04 tok/s rolling peak) on WV-v5-Sub2.5bpw and 25.06–25.87 tok/s on WV-v5-3.5bpw (--moe-stream-cpu-x4 0), with 0 --moe-stream-out-check failures.

SHA-256 Checksums

43edc147a7fb9bd4071c75e9c61005539ae7938646239491c90ca60025b5ec31  libqfn_wv_plugin.so
c28190be00b449f9decdeca359ed0bc3fe7e2d330a0797f25e8c50dbc6b336ff  qfn_wv_plugin.dll
c528bc1efc77cfc1d76fc5bc9667600a42a819d8d3207ce540a1aed464155fb4  tokenwing_repack_cuda_overlap.patch