Repository navigation
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, plugin_version = 1.3.0) drop-in runtime & converter plugins (--quant-plugin) for WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs on runmyai/tokenwing dev across Linux x86_64 (GLIBC <= 2.34) and Windows x86_64 (MSVC 19.44 + CUDA 12.8, static cudart).
What's New in wv-plugin-v1.3 (1.3.0, ABI v2 Drop-In Upgrade)
- Hardware
F16C(_mm256_cvtph_ps/vcvtph2ps) Across AllAVX2&AVX-512/VNNIHot Loops: Replaces scalar bit-manipulation FP16->FP32 conversions insideEXT0..EXT5vec_dotand 4-rowvec_dot_x4(including 8-lane batch scale decode in VNNIEXT2/EXT3), speeding up 8-threadSub-2.5bpwVNNIvec_dot_x4from0.4387 -> 0.2615 ms/layer(0.1307 ms/matmul,1.68xfaster thanv1.2.0VNNI and2.88xfaster than un-repackedvec_dot). - SIMD-Vectorized 1-Row & 4-Row Rank-8 Residual SVD (
AVX-512+AVX2): VectorizesU^T y_f32(8-col dot +vcvtph2ps) and 4-row sharedV[r0..r3]accumulation, cutting Rank-8 SVD layer time onSub-2.5bpw(0.4358 -> 0.2817 ms/layer,1.55x) and3.5bpw(0.7876 -> 0.7085 ms/layer). - Simultaneous GPU Pool + CPU VNNI Repack + CUDA Overlap Capability Probe (
repack_expert_slot(type, NULL, NULL, -1, 0) == 1): Paired withrunmyai/tokenwingPR #9 (also attached here astokenwing_repack_cuda_overlap.patch), eliminates theWV_PLUGIN_CPU_REPACK_MODE=1workaround so GPU whole-MoE layers, multi-wave prefill (63.3 tok/s), CUDA overlap, and120/120VNNI-repacked CPU pools run together. - Measured
< 12 GBVRAM Laptop Streaming (20 GiBRAM pool,-t 10,--spec-type ngram-mod --spec-ngram-mod-n-max 8):37.34–37.75 tok/s(38.04 tok/srolling peak) onWV-v5-Sub2.5bpwand25.06–25.87 tok/sonWV-v5-3.5bpw(--moe-stream-cpu-x4 0), with0--moe-stream-out-checkfailures.
SHA-256 Checksums
43edc147a7fb9bd4071c75e9c61005539ae7938646239491c90ca60025b5ec31 libqfn_wv_plugin.so
c28190be00b449f9decdeca359ed0bc3fe7e2d330a0797f25e8c50dbc6b336ff qfn_wv_plugin.dll
c528bc1efc77cfc1d76fc5bc9667600a42a819d8d3207ce540a1aed464155fb4 tokenwing_repack_cuda_overlap.patch