Repository navigation
Releases
v1.1 - Qwen3.8-Flash-Next WV-v5 3.5bpw Companion Assets
TokenWing companion assets for Grawnch/Qwen3.8-Flash-Next-WV-v5-3.5bpw-GGUF (qwen4exp, 48 layers, 512 experts, top-10, 2560 hidden).
| File | Bytes | SHA-256 |
|---|---|---|
Qwen3.8-Flash-Next-WV-v5-3.5bpw.correction |
482,476,032 |
40f7b3c83bc6c9b8b12dc24bf74aa905f97ffa50052ef780d2559b1c8c94d102 |
Qwen3.8-Flash-Next-WV-v5-3.5bpw.vram-experts.json |
133,320 |
4c422100f12f3c73ae53d299ed89f048c84b3f3da3f1c38a22a100abf2b66590 |
Qwen3.8-Flash-Next-WV-v5-3.5bpw.experts.layer-order.json |
199 |
5fae95844715fbf91e9c37045275db086d0962da57243b07cee9eb4927aa0bde |
Qwen3.8-Flash-Next-WV-v5-3.5bpw.usage-seed.json |
116,636 |
d3bcb9469d00881379cec7cefafd8662b37ca4bab17bd96a8b1e52c0133c31d4 |
Also hosted on Hugging Face: https://huggingface.co/Grawnch/TokenWing-Model-Assets
TokenWing Model Assets (Ling 3.0 Flash & Flint 40B)
Pre-computed .correction, .vram-experts.json, and .experts.layer-order.json files for Ling-3.0-Flash-TokenWing and Flint-40B-TokenWing.
QFN WV TokenWing Plugin v1.3 (Linux x86_64 .so + Windows x86_64 .dll)
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, plugin_version = 1.3.0) drop-in runtime & converter plugins (--quant-plugin) for WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs on runmyai/tokenwing dev across Linux x86_64 (GLIBC <= 2.34) and Windows x86_64 (MSVC 19.44 + CUDA 12.8, static cudart).
What's New in wv-plugin-v1.3 (1.3.0, ABI v2 Drop-In Upgrade)
- Hardware
F16C(_mm256_cvtph_ps/vcvtph2ps) Across AllAVX2&AVX-512/VNNIHot Loops: Replaces scalar bit-manipulation FP16->FP32 conversions insideEXT0..EXT5vec_dotand 4-rowvec_dot_x4(including 8-lane batch scale decode in VNNIEXT2/EXT3), speeding up 8-threadSub-2.5bpwVNNIvec_dot_x4from0.4387 -> 0.2615 ms/layer(0.1307 ms/matmul,1.68xfaster thanv1.2.0VNNI and2.88xfaster than un-repackedvec_dot). - SIMD-Vectorized 1-Row & 4-Row Rank-8 Residual SVD (
AVX-512+AVX2): VectorizesU^T y_f32(8-col dot +vcvtph2ps) and 4-row sharedV[r0..r3]accumulation, cutting Rank-8 SVD layer time onSub-2.5bpw(0.4358 -> 0.2817 ms/layer,1.55x) and3.5bpw(0.7876 -> 0.7085 ms/layer). - Simultaneous GPU Pool + CPU VNNI Repack + CUDA Overlap Capability Probe (
repack_expert_slot(type, NULL, NULL, -1, 0) == 1): Paired withrunmyai/tokenwingPR #9 (also attached here astokenwing_repack_cuda_overlap.patch), eliminates theWV_PLUGIN_CPU_REPACK_MODE=1workaround so GPU whole-MoE layers, multi-wave prefill (63.3 tok/s), CUDA overlap, and120/120VNNI-repacked CPU pools run together. - Measured
< 12 GBVRAM Laptop Streaming (20 GiBRAM pool,-t 10,--spec-type ngram-mod --spec-ngram-mod-n-max 8):37.34–37.75 tok/s(38.04 tok/srolling peak) onWV-v5-Sub2.5bpwand25.06–25.87 tok/sonWV-v5-3.5bpw(--moe-stream-cpu-x4 0), with0--moe-stream-out-checkfailures.
SHA-256 Checksums
43edc147a7fb9bd4071c75e9c61005539ae7938646239491c90ca60025b5ec31 libqfn_wv_plugin.so
c28190be00b449f9decdeca359ed0bc3fe7e2d330a0797f25e8c50dbc6b336ff qfn_wv_plugin.dll
c528bc1efc77cfc1d76fc5bc9667600a42a819d8d3207ce540a1aed464155fb4 tokenwing_repack_cuda_overlap.patch
QFN WV TokenWing Plugin v1.2 (Linux x86_64 .so + Windows x86_64 .dll)
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, tokenwing.quant_plugin.types) runtime & converter plugins (--quant-plugin) for running WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs out-of-the-box on runmyai/tokenwing dev (>= 6a3e018f) across both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VBMI+VNNI runtime dispatch, on-load slot repacking WV_PLUGIN_CPU_REPACK_MODE=1 + --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).
ABI Compatibility Note:
wv-plugin-v1.2implements TokenWing ABI v2 (112..117) forWV-v5-Sub2.5bpwandWV-v5-3.5bpwonrunmyai/tokenwingdev. For the legacy ABI v1 (43..48)WV-v5-2.26bpwcheckpoint, usewv-plugin-v1.1.
Highlights in wv-plugin-v1.2 (ABI v2)
- Zero-Loss
EXT2(114)VQ4_G32 + SWZ2_G64Support (WV-v5-Sub2.5bpw,39.48 GiBShard 1): Custom[4, 256, 4]int8 VQ4 codebooks + 1,152 Rank-8 SVD records, on-load VNNI slot repacking (WV_PLUGIN_CPU_REPACK_MODE=1),0.2157 ms/matmul(8-thread AVX-512 VNNI), and23.10 tok/sunder<12 GBVRAM +6 GiBRAM Direct-I/O NVMe streaming. - Complete 4-Row
vec_dot_x4AcrossEXT0..EXT5&WV_OVERRIDE_EXT4n_g=20Override Fix (WV-v5-3.5bpw53.48 GiBShard 1): Eliminates~3.78 GBof duplicatePAIRED_Q3override bloat ondown_exps, delivering0.3814 ms/matmul(EXT0+EXT2) /0.3820 ms/matmul(EXT0+EXT1, 8-thread AVX-512 VNNI) and19.51 tok/sunder<12 GBVRAM Direct-I/O NVMe streaming. - Portability & Symbol Hygiene: Linux
.sobuilt onubuntu-22.04(GLIBC <= 2.34), Windows.dllbuilt withMSVC 19.44 + CUDA 12.8, staticcudart, singleggml_quant_plugin_get_apiC-ABI export.
SHA-256 Checksums
b2c3addeda808a635300f57b870615a99df690fa3f49d702d5c5a99668d73722 libqfn_wv_plugin.so
e820d38ad4736da54fc8fa3244c04a478336226d07652f5893a070154378f155 qfn_wv_plugin.dll
QFN WV TokenWing Plugin v1.1 (Linux x86_64 .so + Windows x86_64 .dll)
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 1 runtime & converter plugins (--quant-plugin) for running WV GGUFs (WV-v5-2.26bpw and WV-v5-3.5bpw EXT0..EXT5) in TokenWing on both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VNNI runtime dispatch, on-load slot repacking --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).
Highlights in wv-plugin-v1.1
- 6 External Quant Types (
EXT0..EXT5): Addswv_swz3_g16(3.59 bpw),wv_vq4_u8(2.26 bpwcold MoE + nativedown_exps),wv_swz4_r12(4.50 bpwrotated spine), andwv_swz6_r12(6.06 bpwrotated LM head / critical spine). - On-Load I/O-Thread VNNI Repack +
vec_dot_x4: Background NVMe reader threads pre-swizzle streamed2.26 bpwslices once on arrival (0.1418 ms/matmul10-expert /0.0999 ms/matmul7-expert 8-thread AVX-512 VNNI). - Zero-Bank-Conflict SM120 CUDA Kernels & Speculative MTP:
71.42 tok/sfull-VRAM GPU decode with--spec-type draft-mtp(55.74 tok/sB=1,103.34 tok/spp512), and27.77 tok/s(tech) /26.50 tok/s(code) under<11 GiBVRAM +6 GiBRAM coldO_DIRECTNVMe streaming (82.35 MiB/tokwithH=96hybrid VRAM experts). - Portability & Symbol Hygiene: Linux
.sobuilt onubuntu-22.04(GLIBC <= 2.34), Windows.dllbuilt withMSVC 19.44 + CUDA 12.8, staticcudart, singleggml_quant_plugin_get_apiC-ABI export.
SHA-256 Checksums
2285d62f00595e22de9d44e3e03ba0ef0bd07efd66ff90aab83119eae3fb30c9 libqfn_wv_plugin.so
6a62a3193019ba4e9e4e8c7cacc2d9d2e8f1dfca75f1985d6651d977ffcb0f80 qfn_wv_plugin.dll
QFN WV TokenWing Plugin v1.0 (Linux x86_64 .so + Windows x86_64 .dll)
Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 1 runtime & converter plugins (--quant-plugin) for running WV GGUFs (Qwen3.8-Flash-Next-WV-v5-3.5bpw / WV-3.5bpw) in TokenWing on both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120 + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL runtime dispatch).
SHA-256 Checksums
14ad2bbc856b8f3e6a84f2e30a66e0948455c4e6e80a9f43b852d1e094234568 libqfn_wv_plugin.so
a9749dff03f2ca1bc9811184a0681eaed7a673302560b6dbc6f4cc546774bbb0 qfn_wv_plugin.dll