v0.23.0

Overview

Version 0.23.0 adds new operators and backend scheduling APIs for sparse attention, asynchronous execution, and allocation-dependency tracking. It includes broad correctness and performance work across CPU, GPU, and accelerator backends, plus an expanded RPC stack with Apple RDMA support.

API changes

• Added GGML_GLU_OP_SWIGLU_CLAMP and ggml_swiglu_clamp() (llama/27930).
• Added ggml_flash_attn_ext_set_n_kv_max() for sparse K/V masks (llama/27970).
• RPC protocol bumped to 6.0.0 for event/async APIs (llama/18626).

Core changes

• Added SWIGLU_CLAMP and sparse-attention n_kv_max support (llama/27930, llama/27970).
• Added allocation-size expansion tracking and allocation-dependency support in backend graph optimization (llama/27960, llama/27301).
• Fixed multi-batch conv_transpose_2d and allowed SET_ROWS in backward passes (llama/26132).
• Backend search-path enumeration tolerates filesystem errors; meta buffers propagate usage (llama/28271, llama/27586).
• Replaced compile-definition versioning with generated ggml-version.h (llama/28364).
• RPC gains event/async APIs, Apple RDMA transport, and avoids serializing remote buffers (llama/18626, llama/26421, llama/26500).

Backend changes

CPU

• AVX2 IQ prompt speedups for large batches (llama/27402).
• Fixed s390x Q5_1 uninitialized accumulator and added conditional SpacemiT IME kernels (llama/28332, llama/27961).
• Reworked KleidiAI build/integration and avoided buffer-type init on dispatch (llama/26077, llama/27891).

CUDA / ROCm

• Added sparse FA, XOR-swizzled FA K/V smem, radix top-k, and extended MoE fusions (llama/27970, llama/25635, llama/27466, llama/25952).
• Added concurrent streams per split, MMQ/MMVQ tunings, and MoE MMQ fixes (llama/28198, llama/28285, llama/26264).
• ROCm/hip: tuned RDNA3 MMQ configs and optimized Q2_0 dot products (llama/26284, llama/26753).

Metal

• Added sparse FA, radix top-k, quantized concat, and chunked SSD MMA (llama/28098, llama/28073, llama/28116, llama/26647).
• Added fa-vec tunings across many Apple GPUs and Metal 4.0 tensor API support (llama/28373, llama/27461).
• Added metallib xcframework support and fixed memory, leak, and dispatch issues (llama/28163, llama/27758, llama/28306).

Vulkan

• Added cross-entropy loss, radix top-k, Lightning Indexer op, and IQ3_S batch mat-vec improvements (llama/27216, llama/28032, llama/27453, llama/27449).
• Fixed FA dequant, mul_mat_id padding, view-alias deps, and warp-size handling (llama/28190, llama/27925, llama/27812, llama/27726).
• Added shader hoisting for row IDs/expert counts and bfloat16 extension gating (llama/26686, llama/28155).

SYCL

• Added residual fusions, peer-to-peer copy, and host-pinned memory limits (llama/27610, llama/27550, llama/27559).
• Added GPU free-memory query, split TOP_K long rows, TILE quantized KV decode, and oneDNN SDPA f16 KV binding (llama/27968, llama/27847, llama/26689, llama/27468).
• Reduced redundant Q4_K MMVQ work and improved --fit handling (llama/27062, llama/27629).

OpenCL

• Added quantized lm_head/decode GEMV and medium-batch GEMM optimizations (llama/26477).
• Added Adreno fixes and matmul path improvements, Intel Xe-LP quant tuning, and MoE binary kernels (llama/27640, llama/26438, llama/27768).

Hexagon

• Added F16 unary ops, ABS/LOG, MUL_MAT/MUL_MAT_ID fusion, and CPY fence fixes (llama/28228, llama/27786, llama/28202, llama/28033).
• Added multi-NPU device support, fully async backend, and on-demand device discovery (llama/26501, llama/27785).

OpenVINO

• Updated OpenVINO, added new op support, whisper.cpp support, and a fuse-to-conv pass (llama/27843).

RPC

• Implemented event and async backend APIs and Apple RDMA transport (llama/18626, llama/26421).
• Avoided serializing remote buffers and fixed pre-RDMA macOS teardown/version issues (llama/26500, llama/27815, llama/27908).

WebGPU

• Fixed misaligned offset crash in tensor get and marked MUL_MAT for allocation expansion (llama/28045, llama/28071).

More info

• Releases and versioning of ggml-org projects

Changelog since v0.22.0

e91ded1 ggml : bump version to 0.23.0 (#1618)
d8e6687 sync : llama.cpp
7523a09 metal : add remaining fa-vec tunings for M3 Max (llama/28373)
72d22bd ggml : replace compile definitions with version.h.in (llama/28364)
7aced59 ggml : rename and make private ggml_op_alloc_size_may_expand() (#0)
24b5dee sync : llama.cpp
b849c94 ggml : don't crash when backend search path can't be read (llama/28271)
da8baab ggml : remove GGML_CUDA_PEER_MAX_BATCH_SIZE (llama/28177)
3573afc ggml-cpu(s390x) : fix q5_1 uninitialized v_acc (llama/28332)
b70ff92 sycl: fuse rms_norm+mul+add and add+add residual chains (llama/27610)
db62b97 SYCL: Refactor GGML_SYCL_ENABLE_MKL_FA to global var (llama/26863)
6921674 opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP) (llama/26477)
90c4733 tune MMVQ to MMQ crossover for SM87 (llama/28285)
1145b17 metal : add sparse FA (llama/28098)
8ed43cf metal : fix glu dispatch with ne00 = 1 (llama/28306)
eb62444 CUDA: Allow concurrent streams per split for multi-GPU (llama/28198)
355bd37 vulkan: fix FA dequant path engagement (llama/28190)
04c25ad sycl : enhance the api to support peer-to-peer copy (llama/27550)
db157a7 sycl: reduce redundant work in Q4_K multi-column MMVQ (llama/27062)
ee9842c finetune: fix no KV cache (llama/27199)
8d6211c ggml-hexagon: add F16 support for unary ops (llama/28228)
1e7de5d metal : add fa-vec tunings for M3 (llama/28236)
f78f88b metal : fix memory query under low-memory conditions (llama/27701)
b40e3c4 ggml-cuda : remove unused vars (llama/28235)
589cf31 CUDA + ggml: add sparse-fa for DSV4/GLM (llama/27970)
b9ba0ae ggml: avoid KleidiAI buffer type init on dispatch (llama/27891)
d9d16fd hexagon: MUL_MAT and MUL_MAT_ID fusion and fixes (llama/28202)
a4325bb vulkan: handle larger batch sizes (>4) efficiently for IQ3_S mat-vec (llama/27449)
bc19469 vulkan : only request VK_KHR_shader_bfloat16 extension if supported (llama/28155)
ac6b4c6 ggml-cpu : conditionally add SpacemiT IME kernel sources (llama/27961)
902e018 opencl: fix out‐of‐bound reads in the Adreno image kernels (#27632)
b201750 hexagon: add missing FARF logs for cpy/get_rows/set_rows/gdn ops (llama/28217)
4a53a5b metal : add metallib build support for xcframework (llama/28163)
ee18d24 cuda: fuse MoE weighted expert reduction (llama/25952)
4af3ab1 Revert "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184)
bed3f69 sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 (llama/28016)
4ca9b70 metal : add fa-vec tuning for M2 Pro (llama/28122)
d6aeb74 metal : add fa-vec tunings for A18 Pro (MacBook Neo) (llama/28152)
0aa6c57 metal : fix more leaks due to missing autoreleasepools (llama/27883)
095419b metal : add fa-vec tuning for M2 Max (llama/28015)
7641be2 sycl : support limit max alloc memory within 2GB for host-pinned memory (llama/27559)
4ac4c29 metal: enable Metal 4.0 tensor API on M5+/A19+ (llama/27461)
27438ee metal : add fa-vec tunings for M1 Ultra (llama/28088)
fda7330 CUDA: XOR swizzle flash attn K,V smem fp16 tiles (llama/25635)
f0251eb metal : add concat support for quantized types (llama/28116)
d2df1bc AVX2: Speed up large batch size prompt processing of IQ models (llama/27402)
5eee39f metal : add top-k radix implementation (llama/28073)
4ed94fa opencl: tune the quant paths for Intel Xe-LP GPUs to improve its TG and PP performance (llama/26438)
5478775 webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (llama/28045)
ef26aae ROCm: add radix TOP_K for long rows (llama/27466)
209bc90 metal : add fa-vec tunings for M1 (llama/28078)
4590461 CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token (llama/27621)
0a97fdf sycl : Enhance to get the free memory of Intel GPU (llama/27968)
b953b35 vulkan: tune mat-vec rows for batched inference on Strix Halo (llama/27909)
5bdb63f ggml : add MUL_MAT to the list of ops that may need additional memory (for WebGPU) (llama/28071)
8521c3b vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next (llama/28032)
f90ed9f hexagon: fix CPY fence bug (llama/28033)
2682b56 metal : add remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 (llama/28017)
d471637 sync : llama.cpp
9da56ef rpc: avoid serializing buffers from other servers (llama/26500)
6b44b36 ggml : fix ggml_backend_buft_get_alloc_size() guard (llama/28038)
7c06f01 ggml: add SWIGLU_CLAMP (llama/27930)
f29745e CUDA: use the fast mm_ids_helper path for any n_expert_used (llama/27978)
985f244 hip: tune rdna 3 mmq config (llama/26284)
6b9f44d hip : optimize Q2_0 dot-product path for gfx1201 (llama/26753)
7989099 sync : llama.cpp
57ce217 ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (llama/27960)
43c9531 rpc: fix apple rdma error spew on teardown (llama/27908)
d6fa510 metal: add fa-vec tunings for M3 Ultra (llama/27999)
b2fc6c4 metal : Add fa-vec tuning for M3 Pro (llama/27963)
bb5b5eb rpc : fix pre-rdma macOS versions (llama/27815)
3a2711c hexagon: support for device discovery and create sessions on demand (llama/27785)
db62fb3 sycl: split long rows in TOP_K instead of one work-group per row (llama/27847)
5033e53 metal : fix null-pipeline crash for F16 src1 mul_mat/mul_mat_id (llama/25648)
5600df5 ggml: allow passing alloc dependencies in graph_optimize (llama/27301)
f494535 metal : add fa-vec tunings for M2 (llama/27940)
8fd7c64 opencl: use a better matmul path on two Adreno GPU generations (llama/27640)
7e3e8ad metal : assert shared memory padding (llama/27951)
309f6c7 metal : add remaining fa-vec tunings for M4 Pro (llama/27915)
1a82d66 sycl: make --fit respect --fit-target better (llama/27629)
503e641 vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs (llama/27526)
3c1172d metal : add fa-vec tunings for M1 Max (llama/27932)
8e6d7db vulkan: Change mul_mat_id to pad K rather than N (llama/27925)
8ec11f0 vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize (llama/27812)
a573f76 ggml : fix conv_transpose_2d for multiple batches (llama/26132)
6555125 Vulkan: add hoisting support for row IDs and expert count in shaders (llama/26686)
cfb6123 metal : add fa-vec tunings for M4 (llama/27875)
bf88bab OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (llama/27843)
11a951b sycl: use TILE for quantized KV decode on BMG (llama/26689)
31cd2ca sycl: bind the f16 KV cache in place for the oneDNN SDPA path (llama/27468)
f1354dc metal : add fa-vec tunings for M3 Max, M5 and M5 Pro (llama/27863)
15bfbd2 metal : add fa-vec tunings for M4 Pro (llama/27824)
28e9bae ggml-hexagon: add HTP unary ops for ABS and LOG (llama/27786)
96e5258 hex-unary: fix RMS_NORM_MUL weight-offset bugs for grouped/broadcast norms (llama/27798)
202d077 spec : add DFlash2 support (local convolution + candidate selector) (#27342) (llama/27816)
9f32169 opencl: add bin kernels kernel_gemm_moe_q4_0_q8_1_dp4a_bin, kernel_gemm_moe_mxfp4_q8_1_dp4a_bin (llama/27768)
161c829 Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend (llama/27453)
09eaacf metal : fix memory leaks due to missing autoreleasepools (llama/27758)
fee87ab hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (llama/26501)
acb2e0d vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps (llama/27726)
f49b86e Implemented vulkan cross_entropy_loss and cross_entropy_loss_back (llama/27216)
82d091a rpc : implement event and async backend APIs (llama/18626)
41a2a15 sync : llama.cpp
098d355 cuda: unblock mmq for MoE on sm_60 (llama/26264)
68fca9e sync : llama.cpp
ecaf424 ggml-metal: add chunked SSD MMA for Mamba-2 prefill optimization (llama/26647)
d845b71 ggml-meta: propagate buffer usage and call init on the new tensors (llama/27586)
359a1df kleidiai: Rework KleidiAI Build System/Integration (llama/26077)
9655792 rpc: support apple RDMA as an RPC transport (llama/26421)
e9c3b15 metal : null-check buffer alloc to fix OOM crash (llama/25371)
36da571 scripts : add a prompt for generting release summary