Skip to content

Releases

v1.1 - Qwen3.8-Flash-Next WV-v5 3.5bpw Companion Assets

Choose a tag to compare

@grawnchtoader grawnchtoader released this 14 Nov 18:18
1843f5c

TokenWing companion assets for Grawnch/Qwen3.8-Flash-Next-WV-v5-3.5bpw-GGUF (qwen4exp, 48 layers, 512 experts, top-10, 2560 hidden).

File Bytes SHA-256
Qwen3.8-Flash-Next-WV-v5-3.5bpw.correction 482,476,032 40f7b3c83bc6c9b8b12dc24bf74aa905f97ffa50052ef780d2559b1c8c94d102
Qwen3.8-Flash-Next-WV-v5-3.5bpw.vram-experts.json 133,320 4c422100f12f3c73ae53d299ed89f048c84b3f3da3f1c38a22a100abf2b66590
Qwen3.8-Flash-Next-WV-v5-3.5bpw.experts.layer-order.json 199 5fae95844715fbf91e9c37045275db086d0962da57243b07cee9eb4927aa0bde
Qwen3.8-Flash-Next-WV-v5-3.5bpw.usage-seed.json 116,636 d3bcb9469d00881379cec7cefafd8662b37ca4bab17bd96a8b1e52c0133c31d4

Also hosted on Hugging Face: https://huggingface.co/Grawnch/TokenWing-Model-Assets

TokenWing Model Assets (Ling 3.0 Flash & Flint 40B)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 10 Nov 00:19
1843f5c

Pre-computed .correction, .vram-experts.json, and .experts.layer-order.json files for Ling-3.0-Flash-TokenWing and Flint-40B-TokenWing.

QFN WV TokenWing Plugin v1.3 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 20:39
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, plugin_version = 1.3.0) drop-in runtime & converter plugins (--quant-plugin) for WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs on runmyai/tokenwing dev across Linux x86_64 (GLIBC <= 2.34) and Windows x86_64 (MSVC 19.44 + CUDA 12.8, static cudart).

What's New in wv-plugin-v1.3 (1.3.0, ABI v2 Drop-In Upgrade)

  • Hardware F16C (_mm256_cvtph_ps / vcvtph2ps) Across All AVX2 & AVX-512 / VNNI Hot Loops: Replaces scalar bit-manipulation FP16->FP32 conversions inside EXT0..EXT5 vec_dot and 4-row vec_dot_x4 (including 8-lane batch scale decode in VNNI EXT2/EXT3), speeding up 8-thread Sub-2.5bpw VNNI vec_dot_x4 from 0.4387 -> 0.2615 ms/layer (0.1307 ms/matmul, 1.68x faster than v1.2.0 VNNI and 2.88x faster than un-repacked vec_dot).
  • SIMD-Vectorized 1-Row & 4-Row Rank-8 Residual SVD (AVX-512 + AVX2): Vectorizes U^T y_f32 (8-col dot + vcvtph2ps) and 4-row shared V[r0..r3] accumulation, cutting Rank-8 SVD layer time on Sub-2.5bpw (0.4358 -> 0.2817 ms/layer, 1.55x) and 3.5bpw (0.7876 -> 0.7085 ms/layer).
  • Simultaneous GPU Pool + CPU VNNI Repack + CUDA Overlap Capability Probe (repack_expert_slot(type, NULL, NULL, -1, 0) == 1): Paired with runmyai/tokenwing PR #9 (also attached here as tokenwing_repack_cuda_overlap.patch), eliminates the WV_PLUGIN_CPU_REPACK_MODE=1 workaround so GPU whole-MoE layers, multi-wave prefill (63.3 tok/s), CUDA overlap, and 120/120 VNNI-repacked CPU pools run together.
  • Measured < 12 GB VRAM Laptop Streaming (20 GiB RAM pool, -t 10, --spec-type ngram-mod --spec-ngram-mod-n-max 8): 37.34–37.75 tok/s (38.04 tok/s rolling peak) on WV-v5-Sub2.5bpw and 25.06–25.87 tok/s on WV-v5-3.5bpw (--moe-stream-cpu-x4 0), with 0 --moe-stream-out-check failures.

SHA-256 Checksums

43edc147a7fb9bd4071c75e9c61005539ae7938646239491c90ca60025b5ec31  libqfn_wv_plugin.so
c28190be00b449f9decdeca359ed0bc3fe7e2d330a0797f25e8c50dbc6b336ff  qfn_wv_plugin.dll
c528bc1efc77cfc1d76fc5bc9667600a42a819d8d3207ce540a1aed464155fb4  tokenwing_repack_cuda_overlap.patch

QFN WV TokenWing Plugin v1.2 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 15:59
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 2 (EXT0..EXT5 = type IDs 112..117, tokenwing.quant_plugin.types) runtime & converter plugins (--quant-plugin) for running WV-v5-Sub2.5bpw (2.46 bpw) and WV-v5-3.5bpw GGUFs out-of-the-box on runmyai/tokenwing dev (>= 6a3e018f) across both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VBMI+VNNI runtime dispatch, on-load slot repacking WV_PLUGIN_CPU_REPACK_MODE=1 + --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).

ABI Compatibility Note: wv-plugin-v1.2 implements TokenWing ABI v2 (112..117) for WV-v5-Sub2.5bpw and WV-v5-3.5bpw on runmyai/tokenwing dev. For the legacy ABI v1 (43..48) WV-v5-2.26bpw checkpoint, use wv-plugin-v1.1.

Highlights in wv-plugin-v1.2 (ABI v2)

  • Zero-Loss EXT2 (114) VQ4_G32 + SWZ2_G64 Support (WV-v5-Sub2.5bpw, 39.48 GiB Shard 1): Custom [4, 256, 4] int8 VQ4 codebooks + 1,152 Rank-8 SVD records, on-load VNNI slot repacking (WV_PLUGIN_CPU_REPACK_MODE=1), 0.2157 ms/matmul (8-thread AVX-512 VNNI), and 23.10 tok/s under <12 GB VRAM + 6 GiB RAM Direct-I/O NVMe streaming.
  • Complete 4-Row vec_dot_x4 Across EXT0..EXT5 & WV_OVERRIDE_EXT4 n_g=20 Override Fix (WV-v5-3.5bpw 53.48 GiB Shard 1): Eliminates ~3.78 GB of duplicate PAIRED_Q3 override bloat on down_exps, delivering 0.3814 ms/matmul (EXT0+EXT2) / 0.3820 ms/matmul (EXT0+EXT1, 8-thread AVX-512 VNNI) and 19.51 tok/s under <12 GB VRAM Direct-I/O NVMe streaming.
  • Portability & Symbol Hygiene: Linux .so built on ubuntu-22.04 (GLIBC <= 2.34), Windows .dll built with MSVC 19.44 + CUDA 12.8, static cudart, single ggml_quant_plugin_get_api C-ABI export.

SHA-256 Checksums

b2c3addeda808a635300f57b870615a99df690fa3f49d702d5c5a99668d73722  libqfn_wv_plugin.so
e820d38ad4736da54fc8fa3244c04a478336226d07652f5893a070154378f155  qfn_wv_plugin.dll

QFN WV TokenWing Plugin v1.1 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 15 Nov 04:55
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 1 runtime & converter plugins (--quant-plugin) for running WV GGUFs (WV-v5-2.26bpw and WV-v5-3.5bpw EXT0..EXT5) in TokenWing on both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120a + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL+VNNI runtime dispatch, on-load slot repacking --moe-stream-cpu-repack 1, and 4-row vec_dot_x4 --moe-stream-cpu-x4 2).

Highlights in wv-plugin-v1.1

  • 6 External Quant Types (EXT0..EXT5): Adds wv_swz3_g16 (3.59 bpw), wv_vq4_u8 (2.26 bpw cold MoE + native down_exps), wv_swz4_r12 (4.50 bpw rotated spine), and wv_swz6_r12 (6.06 bpw rotated LM head / critical spine).
  • On-Load I/O-Thread VNNI Repack + vec_dot_x4: Background NVMe reader threads pre-swizzle streamed 2.26 bpw slices once on arrival (0.1418 ms/matmul 10-expert / 0.0999 ms/matmul 7-expert 8-thread AVX-512 VNNI).
  • Zero-Bank-Conflict SM120 CUDA Kernels & Speculative MTP: 71.42 tok/s full-VRAM GPU decode with --spec-type draft-mtp (55.74 tok/s B=1, 103.34 tok/s pp512), and 27.77 tok/s (tech) / 26.50 tok/s (code) under <11 GiB VRAM + 6 GiB RAM cold O_DIRECT NVMe streaming (82.35 MiB/tok with H=96 hybrid VRAM experts).
  • Portability & Symbol Hygiene: Linux .so built on ubuntu-22.04 (GLIBC <= 2.34), Windows .dll built with MSVC 19.44 + CUDA 12.8, static cudart, single ggml_quant_plugin_get_api C-ABI export.

SHA-256 Checksums

2285d62f00595e22de9d44e3e03ba0ef0bd07efd66ff90aab83119eae3fb30c9  libqfn_wv_plugin.so
6a62a3193019ba4e9e4e8c7cacc2d9d2e8f1dfca75f1985d6651d977ffcb0f80  qfn_wv_plugin.dll

QFN WV TokenWing Plugin v1.0 (Linux x86_64 .so + Windows x86_64 .dll)

Choose a tag to compare

@grawnchtoader grawnchtoader released this 14 Nov 19:27
1843f5c

Prebuilt GGML_QUANT_PLUGIN_ABI_VERSION = 1 runtime & converter plugins (--quant-plugin) for running WV GGUFs (Qwen3.8-Flash-Next-WV-v5-3.5bpw / WV-3.5bpw) in TokenWing on both GPU (sm_75, sm_80, sm_86, sm_89, sm_90, sm_120 + compute_120 PTX, static cudart) and CPU (Scalar + AVX2+FMA + AVX-512BW+VL runtime dispatch).

SHA-256 Checksums

14ad2bbc856b8f3e6a84f2e30a66e0948455c4e6e80a9f43b852d1e094234568  libqfn_wv_plugin.so
a9749dff03f2ca1bc9811184a0681eaed7a673302560b6dbc6f4cc546774bbb0  qfn_wv_plugin.dll