Repository navigation
b12563
Pre-releaseCUDA: keep the MUL_MAT_ID precision in the per-expert fallback (#31345)
- tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range
IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.
- CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback
On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.
Diagnosis and fix by denizyuret in #30951.
- CUDA: keep the MUL_MAT_ID precision in the per-expert fallback
The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.
Co-authored-by: JonSnow1807 JonSnow1807@users.noreply.github.com
Website:
Attestations:
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu 24.04 s390x (CPU)
- Ubuntu 26.04 s390x (CPU)
- Ubuntu 26.04 s390x (zDNN)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
- Ubuntu x64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu arm64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu x64 (ROCm 10.0)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
- Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.5 DLLs
- Windows arm64 (CUDA 13) - CUDA 13.5 DLLs
- Windows x64 (Vulkan)
- Windows arm64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (ROCm 10.0)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: