3.6.2

[wangzhaode]

MNN 3.6.2 版本发布说明

发布日期: 2026年10月

📦 安装: pip install mnn==3.6.2 ｜ PyPI

📌 版本概述

MNN 3.6.2 聚焦 端侧异构推理性能与投机解码加速。本版本以 Metal、Hexagon 和 DFlash 为重点，持续提升 Qwen3 / Qwen3.5 在 CPU、GPU 与 NPU 上的推理效率。

典型性能：Qwen3.5-2B 在 iPad Pro M5 的 GPU 上通过 Metal 后端运行，Prefill 达到 3542.91 tok/s；启用 DFlash 后，Decode 从 111.89 提升至 233.98 tok/s，达到常规解码的 2.09 倍。

🚀 版本亮点

• Metal / Apple GPU Neural Accelerators 优化：通过 Metal Tensor API 利用 Apple GPU 内的神经网络加速单元，提升 Qwen3 / Qwen3.5 Dense 推理性能
• Hexagon 推理优化：提升 NPU 上的 Prefill 效率，减少 Decode 阶段的计算与调度开销
• DFlash 投机解码加速：加速草稿生成与验证，复用目标模型输出层权重，提高生成吞吐并减少内存占用
• CPU / OpenCL 性能优化：改进 OpenCL Attention 与融合解码，完善 CPU 的 SME2 检测与 SME2 / NEON 混合执行
• LinearAttention 与 MoE 多后端优化：提升 Qwen3.5 LinearAttention 在 Metal、Hexagon DSP、x86 与 RISC-V 上的执行效率，优化 MoE 专家计算
• RISC-V / 进迭时空 K3 LLM 适配：完善 RVV 与 IME2 支持，优化低比特 LLM 推理和动态输入处理

📊 性能基准

图：左侧为 iPad Pro M5 的 CPU / Metal GPU 性能，Qwen3.5-2B 额外展示 DFlash 投机解码；右侧为 OnePlus 13 的 CPU / OpenCL GPU / Hexagon NPU 性能。上排 Prefill，下排 Decode。

✨ 新功能与性能优化

Metal 后端

• Apple GPU Neural Accelerators：围绕 Metal Tensor API 优化矩阵计算、Attention 和 LinearAttention 的分块与数据访问，发挥 M5 等受支持 GPU 内神经网络加速单元的计算能力
• Metal 低比特计算：新增 W2 / W3 解码 GEMV 与 conv1x1 支持，优化 Q4 权重读取、分块计算和投机解码块处理
• Metal Attention 与融合计算：优化 prefill / decode Attention、RMSNorm、RoPE、融合 QKV / SwiGLU 投影及命令提交，减少中间计算与同步开销
• Metal LinearAttention：优化 gated delta rule 的融合计算，为 Tensor API 和 SIMD-group 矩阵计算提供分块 prefill 路径，并优化短输入和 decode 命令回放
• Metal 上下文释放：新增 context release API

Hexagon / NPU 后端

• Hexagon 融合解码：新增融合 INT4 GEMV、GatedRMSNorm 与 norm + RoPE 路径，减少解码阶段的算子调度
• Hexagon DSP LinearAttention：支持 Qwen3.5 gated delta rule 与 LFM2 short convolution，并引入分块 prefill 与 HMX 计算
• Hexagon prefill 优化：优化 INT4 反量化、HMX GEMM、分组因果 Attention、DMA 双缓冲和 VTCM 工作区，复用 prefill / decode 间的 KV 页面与动态内存池
• Hexagon 模型与量化覆盖：新增 Qwen3-VL 支持，完善对称 / 非对称分块量化、INT8 及 FP16 激活计算
• Rockchip RKNN 后端：新增 RKNN NPU 后端的初步支持
• QNN 工具与算子：新增 Deconvolution、中间 Tensor dump 和 VTCM 大小配置，支持 LLM / visual 模型导出至同一 QNN 目录

LLM / 推理与导出

• MoE 专家计算优化：将 Token 按专家归组，批量执行专家投影，融合 gate / up 与 SwiGLU；为 prefill 和 decode 提供对应的计算路径，覆盖 CPU / Metal / OpenCL / CUDA / Vulkan
• Transformer 融合优化：导出时融合投影层与 gated norm，并保留 split-LoRA 导出中的 FusedLinear 融合
• DFlash 投机解码优化：将特征投影、拼接与 argmax 放到设备端，增量维护草稿 KVCache，减少主机往返与重复计算；Metal 使用适合短验证块的 Q4 分块矩阵计算与 K 维切分
• DFlash 权重复用：草稿模型共享目标模型的 lm_head，减少磁盘与运行时的重复权重

升级提示：旧版 DFlash 草稿模型需重新导出。

• QAT checkpoint 导出：新增 --qat，读取 qat_config.json 与 safetensors，保留 scale / zero_point，支持 2 / 3 / 4 / 8-bit 权重
• 可选文本 Token 压缩：新增 MNN_LLM_TOKEN_COMPRESS 编译选项和 token_compress 配置，按 Token 预算选取文本句子；默认关闭，支持纯文本请求及多模态模型的纯文本输入
• OpenCL 前缀 KVCache 文件：新增 setPrefixCacheFile 支持
• Attention mask 统一：统一 mask 构造及各后端的处理语义

CPU 后端

• SME2 计算单元检测：识别可用计算单元及共享分组，为任务划分提供准确的硬件信息
• SME2 / NEON 混合优化：依据计算单元数量分配矩阵任务，优化 Attention、INT8 和 KleidiAI 混合执行，减少线程调度开销
• Arm Attention / RoPE / LayerNorm：优化 FlashAttention prefill / decode，新增 Arm82 FP16 RoPE，优化 packed FP16 C4 LayerNorm
• RISC-V RVV 算子扩展：新增或优化 LinearAttention、Attention mask、KVCache 量化、GEMV、矩阵计算、卷积 / 反卷积、采样、Scale / PReLU 和 INT8 算子
• 进迭时空 K3 / IME2：优化低比特矩阵计算与激活打包，在动态 prefill 中保留执行对象并复用权重预打包结果，减少输入长度变化时的重复准备开销。采用 8 线程、W4B64 配置，Qwen3.5-2B 实测 Prefill（PP512）达到 85.60 tok/s，Decode（TG128）达到 17.68 tok/s
• 卷积与图像算子：优化反卷积输出复用、2×2 反卷积后处理及双线性采样

其他 GPU 后端

• OpenCL FlashAttention 与调优：重构 prefill / decode kernel，完善 mask、尾块、KV 布局和长输入处理；优化异步准备、运行时调优与 kernel 程序拆分
• OpenCL 融合解码：融合 C4 gated / binary RMSNorm 和 FusedLinear GEMV，优化 decode RoPE，减少 QCOM record replay 同步开销
• Vulkan cooperative matrix：优化 INT4 矩阵计算与 Attention 矩阵计算路径

LLM / 多模态模型

• MiniCPM-V 4.7：新增模型导出与推理支持
• MobileMoE：新增模型映射、导出与推理支持
• HunyuanOCR 1.5：新增模型支持
• Qwen3-ASR / Qwen3-VL-Embedding：新增导出与运行时支持
• Qwen3-TTS-12Hz-0.6B-Base：新增语音合成支持，优化 talker / code-predictor 的 KVCache 解码
• Qwen 系列视频输入：新增 Qwen3.5 视频输入，以及 Qwen2.5-VL / Omni 的逐帧 mask、window index 和时间维 mRoPE 处理

工具

• NeedleBench：新增长文本检索评测工具、压缩检查模式和 iOS mnn-llm demo 运行入口
• iOS LLM 一键性能测试：新增设备端 benchmark 流程
• llm_bench -pg：支持共享 KVCache 的 Prefill / Decode 联合测试

🐛 缺陷修复

LLM / VLM / 音频

• DFlash / Eagle 投机解码：修复 DFlash 跨轮状态、Metal fused-attention 崩溃、FP16 路径及验证模块池大小问题；修复 Eagle draft RoPE 配置和 OpenCL KVCache 预留逻辑
• KVCache 与 mask：修复 Metal 混合 LinearAttention 模型的前缀缓存，按后端与精度隔离缓存文件，并修复 Omni mask 路由
• FP16 RoPE：修复位置索引在 FP16 后端上的碰撞问题
• 多模态导出与预处理：修复 Qwen-VL、InternVL、MiniCPM-V 与 Gemma4 的导出、预处理及推理问题
• Transformers 兼容性：修复 Transformers 5.x 下 BGE / BERT embedding 和 Qwen2-VL / Qwen2.5-VL 导出，完善 Qwen3.5 text-only 与 MoE 导出兼容性
• 语音处理：修复 Qwen3-ASR 音频长度、Whisper 周期 Hann 窗、Paraformer CIF 权重读取，以及 Qwen3-TTS 音频输出问题
• LoRA / AWQ：修复 LoRA 权重解析与融合导出、视觉模型 create_lora 崩溃，以及 AWQ KVCache 清理和位置索引问题
• Tokenizer / 采样：修复流式 Unicode、重复 BOS、词表越界、初始化失败及 Windows 文件读取问题，修正混合采样中的重复惩罚
• 多轮与并发推理：修复 Omni 连续请求的 PLE 状态和 LlmContext 跨线程释放后访问问题
• 评测统计：修复 EOS 提前停止导致的 benchmark 统计偏差，以及 NeedleBench 输入构造与纯文本压缩处理

GPU / NPU 后端

• Metal：修复 Attention 并发访问、mask 与 KVCache 扩容问题，以及部分算子计算和资源管理错误；GPU 命令失败时终止生成
• OpenCL：修复整数算子计算与布局、Omni 音频推理和 Mali 兼容性问题，完善低内存模式、调优与动态内存统计
• Vulkan：修复大图 descriptor pool 耗尽崩溃和 multi-image softmax 路径
• CUDA：修复大输入 im2col 打包中的 INT32 溢出，以及 BinaryBlit 的 INT32 REALDIV 分派
• Hexagon：修复 Attention tile 数据路径和主机写入与排队命令的冲突，增加 VTCM 工作区边界检查
• QNN / 多后端 Pooling：修复 QNN Interp / Flatten / LeakyReLU、导出路径与错误处理，以及多后端显式 pooling pads 处理

CPU / 框架

• 输入与 Resize 缓存：修复重复 forward 未刷新输入副本、布局变化后复用旧副本，以及 CPUResizeCache 引用过期内存的问题
• 内存与生命周期：修复 KleidiAI depthwise 的释放后访问、模型加载泄漏、Express loadMap / readMap 缓冲区问题及大权重偏移溢出
• 数值正确性：修复动态量化、FP16 residual、矩阵运算和 RVV 尾部处理中的错误
• RISC-V 运行时分派：使用 riscv_hwprobe 检测能力，保留标量回退，避免在不支持 RVV 的设备上错误分派
• 跨平台构建：修复 Linux、Harmony、iOS 和 Windows 构建，以及关闭部分可选功能时的编译问题

转换器 / 安全 / 其他

• ONNX 转换：修复算子属性解析、默认值处理、形状与类型识别问题
• TFLite 与非法模型：完善 Tensor / buffer / operand 索引校验，修复畸形卷积、池化模型的崩溃与除零
• 文件与命令安全：限制 ONNX external_data 的读取目录，修复 QNN 导出命令注入风险，校验外置卷积权重长度和 LLM HTTP 请求 JSON
• 内存安全：修复多算子边界问题，更新 stb_image 并修复空 IDAT 处理
• 转换失败回报：正确传播优化与转换失败，报告模型文件和外置权重写入失败
• PyMNN / Diffusion / App：恢复 expr.const 整数地址输入，修复 Diffusion tokenizer / ONNX 导出、Android 下载服务竞争崩溃和流式 Markdown 表格闪烁

📚 其他改进

• 多模态处理结构：按模型家族共享处理逻辑，减少重复实现
• 可选 NPU 插件：整理可配置的插件接口、头文件和构建依赖，保留运行时回退
• 测试覆盖与校验：补充 Metal Attention / LinearAttention、多实例并发、视觉模型批量推理测试及多 LoRA 示例，完善异常数值检查和测试失败上报
• iOS 集成：补齐 framework 中的 LLM 引擎头文件，修复 Metal Tensor API 兼容性
• 文档与示例：完善 C++ demo、PyMNN、LLM Python 示例及后端开发指南
• 构建与打包：完善 PyMNN CI 依赖、Harmony 动态库名称与 CocoaPods 版本配置

🙏 致谢

衷心感谢所有贡献者对本版本的宝贵贡献，以下按 GitHub login 列出部分代表性工作：

• @wangzhaode — MiniCPM-V 4.7、MobileMoE、IndexedLinear MoE、文本 Token 压缩、SpacemiT IME2 与 Vulkan INT4 优化
• @v0jiuqi — Metal / Arm 推理优化、Transformer 融合、Qwen2.5-VL / Omni 视频、NeedleBench iOS 工具及多项稳定性修复
• @Qxinyu — Hexagon 融合解码与 DSP LinearAttention、OpenCL FlashAttention / 融合计算、Vulkan 与 QNN 修复
• @ValiantDiligent — DFlash 优化与修复、Gemma4、Eagle RoPE、Android 下载服务和界面修复 ([LLM:Bugfix] Honor Eagle draft RoPE configuration #4942, [App:Bugfix] Fix download foreground service crash when a stop races the start #4969, [App:Bugfix] Fix chat list flicker during streaming #4881)
• @Tangesion — QAT checkpoint 导出、Metal 优化与资源管理、转换工具安全修复
• @huangzhengxiang — Qwen3-ASR / Qwen3-VL-Embedding ([LLM:Feature] Add Qwen3-ASR and Qwen3-VL-Embedding export and runtime support. #4478)、Qwen3-TTS ([LLM:Feature] Support Qwen3-TTS-12Hz-0.6B-Base #4589)、RKNN ([NPU:rknn] Add initial rknn support for Rockchip NPUs, supporting traditional models. #4524)、SME2 Attention ([CPU:Perf] Add SME2 attention SME-NEON mixed optimization #4720)
• @typer-J — RVV LinearAttention、Attention mask、KVCache 量化、GEMV 及多项 CPU 算子优化
• @hulkbig — ONNX 转换语义、文件写入错误处理、DynamicQuant 与 CV 矩阵修复
• @vklonghml — 输入副本刷新 ([Core:Bugfix] 修复循环子模块输入复制缓存未刷新导致的 CIF 错误 #5014)、FP16 residual ([CPU:Bugfix] 修复 packed FP16 residual 输出缺少饱和转换的问题 #4989)、Qwen3-ASR / Whisper / Paraformer 修复
• @yydhYYDH — Hexagon Qwen3-VL ([Hexagon:Feature] Support Qwen3-VL with symmetric and asymmetric block-wise quantization #4774)、Qwen-VL 预处理 ([LLM:Bugfix] Align Qwen3-VL runtime preprocessing #4595)、输入缓存布局修复 ([CPU:Bugfix] Invalidate input copy cache on layout changes #5006)
• @HenryDen — KleidiAI 集成与 SME2 / NEON 混合执行 ([CPU:Refact] Refactor the KleidiAI integration code #4602, [CPU:Feature] Add KleidiAI SME2/NEON hybrid execution #4900)
• @fshuang8299 — TFLite 校验与 Express loadMap 修复 ([Converter:Bugfix] Fix MNNConvert crash on malformed tflite models (#4796) #4823, [Converter:Bugfix] Validate the remaining tflite operand and buffer lookups in ConvolutionTflite #4899, [Converter:Bugfix] Validate the raw tflite operand and buffer lookups in DepthwiseConv2DTflite #4914, [Express:Bugfix] Fix loadMap input tensor losing host buffer #4733, [Express:Bugfix] Fix loadMap/load output readMap returning NULL #4751)
• @ZedingZhang — Omni 多轮状态和 mask、AWQ position IDs、QNN embedding 导出 ([LLM:Bugfix] Refresh PLE for successive Omni text requests #4894, [LLM:Bugfix] Route Omni attention masks by mode #4888, [LLM:Bugfix] Pass calibration input IDs to AWQ position IDs #4806, [LLM:Bugfix] Fix embedding QNN export paths #4555)
• @LudovicoYIN — x86 LinearAttention、AWQ KVCache 与 QNN 导出修复 ([CPU:Perf] Optimize LinearAttention x86 kernels #4682, [LLM:Bugfix] Clear AWQ attention KV cache #4646, [QNN:Bugfix] Fix NPU export and error logging #4752)
• @Fatemanx — HunyuanOCR 1.5、Qwen3.5 视频与 InternVL 修复 ([LLM:Feature] Add HunyuanOCR 1.5 support #4625, [LLM:Feature] Add Qwen3.5 video input #4577, [LLM] Fix InternVL OpenCL vision inference #4689)
• @luke-opus — RVV INT8 尾部、测试数值校验和构建流程修复 ([CPU:Bugfix] Preserve RVV INT8 sum accumulator tails #4869, [Infra:Bugfix] Reject non-finite mismatches in test comparisons #4965, [Infra:Bugfix] Reject invalid numeric dumps in value checker #5055, [Infra:Bugfix] Propagate test stage generator failures #5056)
• @ZhuoenX — C++ / PyMNN / Python LLM 示例和贡献指南 ([Doc] Document C++ demo arguments and prerequisites #4953, [Doc] Document runnable pymnn examples and prerequisites #4955, [Python] Use manual decoding in the LLM example and sync documentation #4956, [Doc:Bugfix] Fix stale fork wording and branch targets in CONTRIBUTING #4941)
• @jxt1234 — Hexagon Attention、INT8 与 INT4 计算优化 ([Hexagon:Perf] Sync hexagon optimization for attention, unary, raster… #4715)
• @Hosi121 — 反卷积输出复用与后处理融合 ([CPU:Perf] Share output for deconvolution without overlap #4897, [CPU:Perf] Fuse output processing for 2x2 deconvolution #4912)
• @ilyas-mallah — HuggingFace tokenizer 编码性能优化 ([LLM:Perf] Stop copying the rest of the prompt for every regex match in HuggingfaceTokenizer::encode #5028)
• @kerry0520 — tokenizer 初始化失败与 benchmark 文件句柄修复 ([LLM:Bugfix] Stop loading when tokenizer creation fails #5027, [LLM:Bugfix] Close the weight size probe handle in llm_bench #5025)

同时感谢 @BillyONeal、@JingliangGao、@PanicTitan、@TheodoreGan、@Xlawy、@baicai-1145、@bhu619、@blueskycoco、@eharris128、@eyesofish、@hebangwen、@huluhuluu、@ihb2032、@jxgxxx、@kabishou11、@mistletoe1024、@oftrongs、@patrickkabwe、@primorLee、@pruthvikar、@srpatcha、@vera-rubin，以及参与开发、测试和问题反馈的所有贡献者。

---

MNN 3.6.2 Release Notes

Release Date: October 2026

📦 Install: pip install mnn==3.6.2 ｜ PyPI

📌 Overview

MNN 3.6.2 focuses on on-device heterogeneous inference performance and speculative decoding acceleration. With Metal, Hexagon, and DFlash as key priorities, this release improves Qwen3 / Qwen3.5 inference across CPUs, GPUs, and NPUs.

Representative result: Qwen3.5-2B running on the iPad Pro M5 GPU through Metal achieves 3542.91 tok/s prefill. DFlash increases decoding throughput from 111.89 to 233.98 tok/s, reaching 2.09× the throughput of standard decoding.

🚀 Highlights

• Metal / Apple GPU Neural Accelerators: Uses the Metal Tensor API to accelerate Qwen3 / Qwen3.5 dense-model inference with Neural Accelerators in Apple GPUs
• Hexagon Inference Optimization: Improves NPU prefill performance and reduces computation and dispatch overhead during decoding
• DFlash Speculative Decoding: Accelerates draft generation and verification, and shares target-model output weights to improve throughput and reduce memory use
• CPU / OpenCL Performance: Improves OpenCL Attention and fused decoding, plus CPU SME2 detection and SME2 / NEON hybrid execution
• LinearAttention and MoE Across Backends: Improves Qwen3.5 LinearAttention on Metal, Hexagon DSP, x86, and RISC-V, along with MoE expert computation
• RISC-V / SpacemiT K3 LLM Support: Improves RVV and IME2 support for low-bit LLM inference and dynamic inputs

📊 Performance Benchmarks

Figure: Left: CPU / Metal GPU on iPad Pro M5, with DFlash decoding also shown for Qwen3.5-2B. Right: CPU / OpenCL GPU / Hexagon NPU on OnePlus 13. Prefill is on the top row and decode on the bottom row.

✨ New Features and Performance Improvements

Metal Backend

• Apple GPU Neural Accelerators: Optimizes tiling and data access for matrix computation, Attention, and LinearAttention through the Metal Tensor API to use the Neural Accelerators in supported GPUs such as M5
• Metal Low-Bit Computation: Added W2 / W3 decoding GEMV and conv1x1 support; optimized Q4 weight loading, tiling, and speculative decoding blocks
• Metal Attention and Fusion: Optimized prefill / decode Attention, RMSNorm, RoPE, fused QKV / SwiGLU projections, and command submission to reduce intermediate computation and synchronization
• Metal LinearAttention: Improved fused gated delta rule, chunked prefill using Tensor API and SIMD-group matrix computation, short-input execution, and decode command replay
• Metal Context Release: Added a context release API

Hexagon / NPU Backends

• Hexagon Fused Decoding: Added fused INT4 GEMV, GatedRMSNorm, and norm + RoPE paths to reduce decoding dispatches
• Hexagon DSP LinearAttention: Added Qwen3.5 gated delta rule and LFM2 short convolution, with chunked prefill and HMX computation
• Hexagon Prefill Optimization: Improved INT4 dequantization, HMX GEMM, grouped causal Attention, DMA double buffering, and VTCM workspace usage; reused KV pages and dynamic memory pools across prefill / decode
• Hexagon Model and Quantization Coverage: Added Qwen3-VL support and improved symmetric / asymmetric block quantization, INT8, and FP16 activation computation
• Rockchip RKNN Backend: Added initial support for the RKNN NPU backend
• QNN Tools and Operators: Added Deconvolution, intermediate tensor dumps, configurable VTCM size, and LLM / visual export into the same QNN directory

LLM / Inference and Export

• MoE Expert Execution: Groups tokens by expert for batched projections, fuses gate / up with SwiGLU, and provides computation paths for prefill and decoding across CPU / Metal / OpenCL / CUDA / Vulkan
• Transformer Fusion: Fused projections and gated norms during export; preserved FusedLinear fusion in split-LoRA export
• DFlash Speculative Decoding: On-device feature projections, concatenation, and argmax, plus incremental draft KVCache, reduce host round trips and repeated computation; Metal adds Q4 tiling and K splitting for short verification blocks
• DFlash Weight Sharing: The draft shares the target model's lm_head to avoid duplicate weights on disk and in memory

Upgrade note: Older DFlash draft models must be re-exported.

• QAT Checkpoint Export: Added --qat to read qat_config.json and safetensors, preserving scale / zero_point for 2 / 3 / 4 / 8-bit weights
• Optional Text Token Compression: Added the MNN_LLM_TOKEN_COMPRESS build option and token_compress setting to select text sentences within a token budget; disabled by default; supports text-only requests, including text-only inputs to multimodal models
• OpenCL Prefix KVCache Files: Added setPrefixCacheFile support
• Unified Attention Masks: Aligned mask construction and processing semantics across backends

CPU Backend

• SME2 Compute-Unit Detection: Identifies available compute units and shared groups to guide workload partitioning
• SME2 / NEON Hybrid Optimization: Partitions matrix work according to compute-unit counts, improves Attention, INT8, and KleidiAI hybrid execution, and reduces thread-scheduling overhead
• Arm Attention / RoPE / LayerNorm: Optimized FlashAttention prefill / decode, added Arm82 FP16 RoPE, and improved packed FP16 C4 LayerNorm
• RISC-V RVV Kernels: Added or optimized LinearAttention, Attention masks, KVCache quantization, GEMV, matrix operations, convolution / deconvolution, sampling, Scale / PReLU, and INT8 operators
• SpacemiT K3 / IME2: Optimized low-bit matrix computation and activation packing; dynamic prefill preserves execution objects and reuses prepacked weights to reduce repeated preparation when input lengths change. With 8 threads and W4B64 quantization, Qwen3.5-2B achieves 85.60 tok/s prefill (PP512) and 17.68 tok/s decode (TG128) in recorded benchmarks
• Convolution and Image Operators: Improved deconvolution output reuse, 2×2 deconvolution post-processing, and bilinear sampling

Other GPU Backends

• OpenCL FlashAttention and Tuning: Reworked prefill / decode kernels, including masks, tails, KV layouts, and long inputs; improved asynchronous preparation, runtime tuning, and program splitting
• OpenCL Fused Decoding: Fused C4 gated / binary RMSNorm and FusedLinear GEMV; optimized decode RoPE and reduced QCOM record replay synchronization
• Vulkan Cooperative Matrix: Optimized INT4 matrix computation and Attention matrix operations

LLM / Multimodal Models

• MiniCPM-V 4.7: Added model export and inference support
• MobileMoE: Added model mapping, export, and inference support
• HunyuanOCR 1.5: Added model support
• Qwen3-ASR / Qwen3-VL-Embedding: Added export and runtime support
• Qwen3-TTS-12Hz-0.6B-Base: Added speech synthesis support and optimized talker / code-predictor decoding with KVCache
• Qwen Video Inputs: Added Qwen3.5 video inputs and per-frame masks, window indices, and temporal mRoPE handling for Qwen2.5-VL / Omni

Tools

• NeedleBench: Added long-text retrieval evaluation, compression inspection, and an iOS mnn-llm demo runner
• One-Command iOS LLM Benchmarking: Added an on-device benchmark workflow
• llm_bench -pg: Supports combined prefill / decode benchmarking with a shared KVCache

🐛 Bug Fixes

LLM / VLM / Audio

• DFlash / Eagle Speculative Decoding: Fixed DFlash cross-turn state, Metal fused-attention crashes, FP16 paths, and verification module pool sizing; fixed Eagle draft RoPE configuration and OpenCL KVCache reservation
• KVCache and Masks: Fixed Metal prefix caching for hybrid LinearAttention models, separated cache files by backend and precision, and corrected Omni mask routing
• FP16 RoPE: Fixed position-index collisions on FP16 backends
• Multimodal Export and Preprocessing: Fixed export, preprocessing, and inference issues for Qwen-VL, InternVL, MiniCPM-V, and Gemma4
• Transformers Compatibility: Fixed BGE / BERT embedding and Qwen2-VL / Qwen2.5-VL export with Transformers 5.x; improved Qwen3.5 text-only and MoE export compatibility
• Audio Processing: Fixed Qwen3-ASR audio lengths, Whisper periodic Hann windows, Paraformer CIF weight access, and Qwen3-TTS audio output
• LoRA / AWQ: Fixed LoRA weight parsing and fused export, create_lora crashes for visual models, and AWQ KVCache cleanup / position handling
• Tokenizer / Sampling: Fixed streaming Unicode, duplicate BOS tokens, vocabulary bounds, initialization failures, and Windows file reads; corrected repetition penalties in mixed sampling
• Multi-Turn and Concurrent Inference: Fixed Omni PLE state across consecutive requests and cross-thread LlmContext use-after-free
• Evaluation Statistics: Fixed benchmark bias from early EOS termination, NeedleBench input construction, and text-only compression handling

GPU / NPU Backends

• Metal: Fixed Attention races, masks, KVCache expansion, and operator / resource-management errors; aborts generation on GPU command failures
• OpenCL: Fixed integer operators and layouts, Omni audio inference, and Mali compatibility; improved low-memory execution, tuning, and dynamic-memory accounting
• Vulkan: Fixed descriptor-pool exhaustion on large graphs and the multi-image softmax path
• CUDA: Fixed INT32 overflow in im2col packing for large inputs and INT32 REALDIV dispatch in BinaryBlit
• Hexagon: Fixed Attention tile data paths and conflicts between host writes and queued commands; added VTCM workspace boundary checks
• QNN / Cross-Backend Pooling: Fixed QNN Interp / Flatten / LeakyReLU, export paths, and error handling, plus explicit pooling pads across multiple backends

CPU / Core

• Input and Resize Caches: Fixed stale input copies across forward calls and layout changes, and CPUResizeCache references to expired memory
• Memory and Lifetimes: Fixed KleidiAI depthwise use-after-free, model-loading leaks, Express loadMap / readMap buffer issues, and large-weight offset overflow
• Numerical Correctness: Fixed dynamic quantization, FP16 residuals, matrix operations, and RVV tail handling
• RISC-V Runtime Dispatch: Uses riscv_hwprobe for capability detection and preserves scalar fallbacks on devices without RVV
• Cross-Platform Builds: Fixed Linux, Harmony, iOS, and Windows builds, including configurations with optional features disabled

Converter / Security / Misc

• ONNX Conversion: Fixed operator attributes, default values, shapes, and type handling
• TFLite and Malformed Models: Improved tensor / buffer / operand index validation; fixed crashes and division by zero in malformed convolution and pooling models
• File and Command Safety: Confined ONNX external_data reads to the model directory, fixed QNN export command-injection risks, and validated external convolution weights and LLM HTTP request JSON
• Memory Safety: Fixed operator boundary issues, updated stb_image, and corrected empty IDAT handling
• Conversion Error Reporting: Propagates optimization / conversion failures and reports model / external-weight write failures
• PyMNN / Diffusion / Apps: Restored integer-address inputs in expr.const; fixed Diffusion tokenizer / ONNX export, Android download-service races, and streaming Markdown table flicker

📚 Other Improvements

• Multimodal Processing: Shared processing logic by model family
• Optional NPU Plugins: Reorganized configurable plugin interfaces, headers, and build dependencies while preserving runtime fallbacks
• Test Coverage and Validation: Added Metal Attention / LinearAttention, multi-instance concurrency, and batch visual inference tests, plus multi-LoRA examples; improved invalid-value checks and test-failure reporting
• iOS Integration: Added LLM engine headers to the framework and fixed Metal Tensor API compatibility
• Documentation and Examples: Improved C++ demos, PyMNN, Python LLM examples, and backend development guides
• Build and Packaging: Improved PyMNN CI dependencies, Harmony library names, and CocoaPods version configuration

🙏 Acknowledgements

We sincerely thank all contributors. Selected contributions are listed below by GitHub login:

• @wangzhaode — MiniCPM-V 4.7, MobileMoE, IndexedLinear MoE, text token compression, SpacemiT IME2, and Vulkan INT4 optimization
• @v0jiuqi — Metal / Arm inference optimization, Transformer fusion, Qwen2.5-VL / Omni video, iOS NeedleBench tools, and stability fixes
• @Qxinyu — Hexagon fused decoding and DSP LinearAttention, OpenCL FlashAttention / fusion, and Vulkan / QNN fixes
• @ValiantDiligent — DFlash optimization and fixes, Gemma4, Eagle RoPE, and Android download-service / UI fixes ([LLM:Bugfix] Honor Eagle draft RoPE configuration #4942, [App:Bugfix] Fix download foreground service crash when a stop races the start #4969, [App:Bugfix] Fix chat list flicker during streaming #4881)
• @Tangesion — QAT checkpoint export, Metal optimization and resource management, and converter security fixes
• @huangzhengxiang — Qwen3-ASR / Qwen3-VL-Embedding ([LLM:Feature] Add Qwen3-ASR and Qwen3-VL-Embedding export and runtime support. #4478), Qwen3-TTS ([LLM:Feature] Support Qwen3-TTS-12Hz-0.6B-Base #4589), RKNN ([NPU:rknn] Add initial rknn support for Rockchip NPUs, supporting traditional models. #4524), and SME2 Attention ([CPU:Perf] Add SME2 attention SME-NEON mixed optimization #4720)
• @typer-J — RVV LinearAttention, Attention masks, KVCache quantization, GEMV, and CPU operator optimization
• @hulkbig — ONNX semantics, write-error reporting, DynamicQuant, and CV matrix fixes
• @vklonghml — Input-copy refresh ([Core:Bugfix] 修复循环子模块输入复制缓存未刷新导致的 CIF 错误 #5014), FP16 residuals ([CPU:Bugfix] 修复 packed FP16 residual 输出缺少饱和转换的问题 #4989), and Qwen3-ASR / Whisper / Paraformer fixes
• @yydhYYDH — Hexagon Qwen3-VL ([Hexagon:Feature] Support Qwen3-VL with symmetric and asymmetric block-wise quantization #4774), Qwen-VL preprocessing ([LLM:Bugfix] Align Qwen3-VL runtime preprocessing #4595), and input-cache layout fixes ([CPU:Bugfix] Invalidate input copy cache on layout changes #5006)
• @HenryDen — KleidiAI integration and SME2 / NEON hybrid execution ([CPU:Refact] Refactor the KleidiAI integration code #4602, [CPU:Feature] Add KleidiAI SME2/NEON hybrid execution #4900)
• @fshuang8299 — TFLite validation and Express loadMap fixes ([Converter:Bugfix] Fix MNNConvert crash on malformed tflite models (#4796) #4823, [Converter:Bugfix] Validate the remaining tflite operand and buffer lookups in ConvolutionTflite #4899, [Converter:Bugfix] Validate the raw tflite operand and buffer lookups in DepthwiseConv2DTflite #4914, [Express:Bugfix] Fix loadMap input tensor losing host buffer #4733, [Express:Bugfix] Fix loadMap/load output readMap returning NULL #4751)
• @ZedingZhang — Omni state / masks, AWQ position IDs, and QNN embedding export ([LLM:Bugfix] Refresh PLE for successive Omni text requests #4894, [LLM:Bugfix] Route Omni attention masks by mode #4888, [LLM:Bugfix] Pass calibration input IDs to AWQ position IDs #4806, [LLM:Bugfix] Fix embedding QNN export paths #4555)
• @LudovicoYIN — x86 LinearAttention, AWQ KVCache, and QNN export fixes ([CPU:Perf] Optimize LinearAttention x86 kernels #4682, [LLM:Bugfix] Clear AWQ attention KV cache #4646, [QNN:Bugfix] Fix NPU export and error logging #4752)
• @Fatemanx — HunyuanOCR 1.5, Qwen3.5 video, and InternVL fixes ([LLM:Feature] Add HunyuanOCR 1.5 support #4625, [LLM:Feature] Add Qwen3.5 video input #4577, [LLM] Fix InternVL OpenCL vision inference #4689)
• @luke-opus — RVV INT8 tails, numeric validation, and build-flow fixes ([CPU:Bugfix] Preserve RVV INT8 sum accumulator tails #4869, [Infra:Bugfix] Reject non-finite mismatches in test comparisons #4965, [Infra:Bugfix] Reject invalid numeric dumps in value checker #5055, [Infra:Bugfix] Propagate test stage generator failures #5056)
• @ZhuoenX — C++ / PyMNN / Python LLM examples and contribution guidelines ([Doc] Document C++ demo arguments and prerequisites #4953, [Doc] Document runnable pymnn examples and prerequisites #4955, [Python] Use manual decoding in the LLM example and sync documentation #4956, [Doc:Bugfix] Fix stale fork wording and branch targets in CONTRIBUTING #4941)
• @jxt1234 — Hexagon Attention, INT8, and INT4 optimization ([Hexagon:Perf] Sync hexagon optimization for attention, unary, raster… #4715)
• @Hosi121 — Deconvolution output reuse and post-processing fusion ([CPU:Perf] Share output for deconvolution without overlap #4897, [CPU:Perf] Fuse output processing for 2x2 deconvolution #4912)
• @ilyas-mallah — HuggingFace tokenizer encoding optimization ([LLM:Perf] Stop copying the rest of the prompt for every regex match in HuggingfaceTokenizer::encode #5028)
• @kerry0520 — Tokenizer initialization and benchmark file-handle fixes ([LLM:Bugfix] Stop loading when tokenizer creation fails #5027, [LLM:Bugfix] Close the weight size probe handle in llm_bench #5025)

Additional thanks to @BillyONeal, @JingliangGao, @PanicTitan, @TheodoreGan, @Xlawy, @baicai-1145, @bhu619, @blueskycoco, @eharris128, @eyesofish, @hebangwen, @huluhuluu, @ihb2032, @jxgxxx, @kabishou11, @mistletoe1024, @oftrongs, @patrickkabwe, @primorLee, @pruthvikar, @srpatcha, and @vera-rubin, and everyone who contributed development, testing, and feedback.

Full Changelog: 3.6.1...3.6.2

---

This discussion was created from the release 3.6.2.