Skip to content

v0.3.0: MTP speculative decoding for Flash-Next

Choose a tag to compare

View all tags
@star1872 star1872 released this 02 Oct 03:11
v0.3.0
862e9b9

NInfer-Offload v0.3.0 — MTP speculative decoding for Qwen3.8-Flash-Next

English | 中文

New

  • MTP speculative decoding for Flash-Next (--spec mtp --draft-tokens N on ninfer and ninfer-serve), with the model's own MTP predictor; its routed experts run on the CPU like the main model's.
  • ninfer-run turns MTP on with one draft when the model file contains it (--no-mtp, --draft-tokens N) and includes the MTP weights in its memory plan.
  • Converter --reuse FILE: add components (such as MTP) to an existing conversion without converting everything again.

Speed (RTX 4060 Ti 16 GB, Xeon E5-2673 v3, DDR3 ~21 GB/s, 64 GPU experts per layer, ninfer-serve, greedy, decode tok/s):

prompt no MTP 1 draft 2 drafts 3 drafts
essay 14.29 14.74 (+3%) 13.61 (-5%) 11.26 (-21%)
code edit 12.61 15.16 (+20%) 16.65 (+32%) 17.55 (+39%)
translation 13.30 15.77 (+19%) 15.51 (+17%) 14.97 (+13%)
explanation 13.69 16.65 (+22%) 17.49 (+28%) 16.87 (+23%)

One draft was faster on every prompt; more drafts help code and slow down free-form prose. Prefill is about 1-2% slower with MTP. MTP adds about 0.1 GiB of VRAM and 1.4 GB of RAM.

Notes

  • MTP needs a model file with the MTP component. The ModelScope conversion mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload is being updated to include it (110 GB); older binaries keep working with the new file (without MTP).
  • Greedy output with MTP is not bit-identical to greedy output without it (verifying several tokens at once changes rounding); checked against an FP32 reference, MTP output agrees with the reference as often as non-MTP output.
  • Tested only on the machine above.

中文

新功能

  • Flash-Next 支持 MTP 投机解码（ninfer 和 ninfer-serve 加 --spec mtp --draft-tokens N），使用模型自带的 MTP 预测层，它的路由专家和主模型一样在 CPU 上计算。
  • ninfer-run：模型文件带 MTP 时自动开启 1 个草稿（--no-mtp 关闭，--draft-tokens N 调整），内存规划包含 MTP 权重。
  • 转换器 --reuse FILE：给已有转换补上 MTP 等组件，不用全部重新转换。

速度：见上表。每轮 1 个草稿时所有提示词都更快（+3%～+22%）；2～3 个草稿对代码更快，但写散文会变慢。开启 MTP 后预填充约慢 1%～2%，额外占用约 0.1 GiB 显存和 1.4 GB 内存。

说明

  • MTP 需要包含 MTP 组件的模型文件。ModelScope 上的转换正在更新为含 MTP 的版本（110 GB）；旧版程序也能读取新文件（只是不开 MTP）。
  • 开启 MTP 后的贪心输出与不开时不完全一致（一次验证多个 token 会改变舍入）；与 FP32 参考实现对比，两者和参考实现的一致程度相当。
  • 只在上面这台机器上测试过。
Assets 4
Loading