Compile bug: CUDA 13.2 breaks some quants (IQ3_S)

[erazortt]

Git commit

git rev-parse HEAD
8845816

This is the current master, however I have tried also tags one month back.

Operating systems

Windows

GGML backends

CUDA

Problem description & steps to reproduce

When building from source I have one specific quant of a model which completly breaks down, when building the same tags as the official builds. The really weird thing is that I have that same model in another quant and this does not break. The difference in the two quants am using is that one has the ffn_down quantized as IQ3_XXS (this one works) and the other has them quantized as IQ3_S (this one breaks). Both models have been quantized identically, apart from this difference.

I have started off using just the default flags:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

but then I moved to what appears to me is used in the official build in https://github.com/ggml-org/llama.cpp/tree/master/.github/workflows/build.yml:

-DLLAMA_BUILD_SERVER=ON ^
            -DLLAMA_BUILD_BORINGSSL=ON ^
            -DGGML_NATIVE=OFF ^
            -DGGML_BACKEND_DL=ON ^
            -DGGML_CPU_ALL_VARIANTS=ON ^
            -DGGML_CUDA=ON ^
            -DGGML_RPC=ON ^
            -DGGML_CUDA_CUB_3DOT2=ON

This however has not changed anything. So to sum up, CUDA 13.2 somehow breaks IQ3_S.

First Bad Commit

No response

Compile command

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

Relevant log output

I have nothing usefull to share here

[erazortt]

In the meantime I have found a publicly available quantization which shows the same issue:
https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/blob/main/Qwen3.5-35B-A3B-UD-IQ3_S.gguf

While this does work correctly:
https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/main/Qwen3.5-35B-A3B-UD-IQ3_XXS.gguf

Also here, the difference between the two quantizations is solely in the ffn_down tensor: one is IQ3_S and the other is IQ3_XXS.

[erazortt]

I am compiling with VS 2022 and CUDA 13.2 and the native build in my case is for a Blackwell.

[pwilkin]

@am17an @JohannesGaessler

[xyehya]

@erazortt might be same issue here (#21371 (comment)) as @theo77186 tagged.

workaround that worked for me, try it and report, this might help in root cause identification:

1. Compile with below flags
   GGML_CUDA=ON
   CMAKE_CUDA_ARCHITECTURES=120f (replace with your gpu architecture and compute capability, use: nvidia-smi --query-gpu=name,compute_cap )
   GGML_CUDA_FORCE_CUBLAS=ON
   GGML_NATIVE=OFF
   GGML_LTO=OFF
   GGML_CUDA_GRAPHS=OFF
   GGML_CUDA_FA_ALL_QUANTS=OFF

2. Load model with -fa off

try this (adjust based on your compiler env variable)

rmdir /s /q build
cmake -B build -G "Visual Studio 17 2022" -A x64 ^
-DGGML_CUDA=ON ^
-DGGML_CUDA_FORCE_CUBLAS=ON ^
-DGGML_NATIVE=OFF ^
-DGGML_LTO=OFF ^
-DGGML_CUDA_GRAPHS=OFF ^
-DGGML_CUDA_FA_ALL_QUANTS=OFF ^
-DGGML_CUDA_F16=ON
cmake --build build --config Release

[xyehya]

The quants are 100% fine. The issue is with CUDA 13.2 . I recompiled latest b8648 on a cuda12.8 container running on the same machine and tested it and it works flawlessly.

CUDA issue confirmed.

[am17an]

cc: @ggml-org/nvidia

[danielhanchen]

I'll ping NVIDIA as well

[gaugarg-nv]

Thanks for the report!

We will repro and report to the CUDA compiler team. I'll keep this updated as I get more information.

CC: @prewaskar-nv

[danielhanchen]

@gaugarg-nv @xyehya Yes I can reproduce - asked CC to test combos.

TLDR: CUDA 13.2 seems problematic for all quants using IQ3_S - full gibberish will be seen.
CUDA 12.8 works fine - CUDA 13.0 is also fine.

@ngxson @ggerganov @pwilkin @CISC @JohannesGaessler for visibility (sorry for tagging you guys, but if people complain about issues, this will be relevant)

More updated details at unslothai/unsloth#4849

Reproduced on an RTX PRO 6000 Blackwell Server Edition (compute 12.0, 96GB VRAM).

Setup:

• Driver: 580.82.07
• CUDA 12.8: nvcc V12.8.93 (baseline, works correctly)
• CUDA 13.2: nvcc V13.2.51 + cuda-compat-13-2 (broken)
• llama.cpp: commit 650bf14 (latest main)
• All builds targeting CMAKE_CUDA_ARCHITECTURES=120

Results -- Gemma-4-31B-it (unsloth/gemma-4-31b-it-GGUF):

Build	Quant	Output	Coherent?
CUDA 12.8	IQ3_XXS	[Start thinking] * User wants a short joke... Why don't scientists trust atoms?	Yes
CUDA 13.2	IQ3_XXS	deHH laesLH laHse지원KLHLeHsenH坐es الأخرىHal laHLLteLH...	No
CUDA 13.2 + FORCE_CUBLAS	IQ3_XXS	CHistTLBalal/THHS singlealalistHHalingLH/alLHLalHen...	No
CUDA 12.8	IQ2_M	[Start thinking] * User wants a short joke... Why don't scientists trust atoms?	Yes
CUDA 13.2	IQ2_M	laistySS own own own laBoge// own own own own de que que la la...	No
CUDA 13.2	IQ4_XS	[Start thinking] * User wants a "short joke"... Why don't scientists trust atoms?	Yes
CUDA 13.2	IQ4_NL	[Start thinking] * User wants a short joke... Why don't scientists trust atoms?	Yes
CUDA 13.2	Q4_K_M	[Start thinking] * User wants a short joke... Why don'...	Yes

Results -- Qwen3.5-35B-A3B (unsloth/Qwen3.5-35B-A3B-GGUF):

Build	Quant	Output	Coherent?
CUDA 12.8	IQ3_S	[Start thinking] Thinking Process: 1. Analyze the Request...	Yes
CUDA 13.2	IQ3_S	I\ns,\nHello!	No
CUDA 13.2 + FORCE_CUBLAS	IQ3_S	(2 p: wjg')'. 1 and 1 0 * + abort/core dump	No + crash

All tests used: -n 64 --temp 0.0 --top-k 1 --ctx-size 512 --no-mmap -ngl 999

Additional findings beyond the original report:

1. IQ2_M is also affected -- not just IQ3_S/IQ3_XXS. The bug boundary appears to be IQ3 and below (IQ2_M, IQ3_XXS, IQ3_S broken) vs IQ4 and above (IQ4_XS, IQ4_NL, Q4_K_M all fine).
2. GGML_CUDA_FORCE_CUBLAS=ON does not help on this configuration -- still produces gibberish, and for Qwen IQ3_S it actually crashes with an abort/core dump. This differs from Eval bug: gemma-4-31B-it-UD-IQ3_XXS.gguf (unsloth) produces gibberish on blackwell (RTX 5080 tested) with GGML_CUDA=ON compilation VS normal correct output with BLAS/CPU compilations #21371 where cuBLAS was reported as a workaround.
3. Recompiling with CUDA 12.8 fixes everything, consistent with other reports.