Misc. bug: ROCm backend leaks VRAM with quantized KV cache

[Edremon]

Name and Version

I had a local unrelated server commit applied on top, it was built on 6ed481e.

version: 9414 (4614480ab)
built with GNU 16.1.1 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

libllama (core library)

Command line

# Leaking case (rocm-kv-q8_0-vram-leak.log)
./llama-server \
    --no-mmap --no-mmproj-offload --ctx-size 65536 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --fit-target 3072 \
    --model gemma-4-31B-it-UD-Q4_K_XL.gguf \
    --mmproj mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 --reasoning off \
    --verbosity 4

# Baseline (rocm-kv-f16-no-leak.log)
./llama-server \
    --no-mmap --no-mmproj-offload --ctx-size 65536 \
    --fit-target 3072 \
    --model gemma-4-31B-it-UD-Q4_K_XL.gguf \
    --mmproj mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 --reasoning off \
    --verbosity 4

# Vulkan (vulkan-kv-q8_0-no-leak.log)
./llama-server \
    --no-mmap --no-mmproj-offload --ctx-size 65536 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --fit-target 3072 \
    --model gemma-4-31B-it-UD-Q4_K_XL.gguf \
    --mmproj mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 --reasoning off \
    --verbosity 4

# V only crash (rocm-kv-v-q8_0-crash.log)
./llama-server \
    --no-mmap --no-mmproj-offload --ctx-size 65536 \
    --cache-type-v q8_0 --fit off --gpu-layers 18 \
    --model gemma-4-31B-it-UD-Q4_K_XL.gguf \
    --mmproj mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 --reasoning off \
    --verbosity 4

Problem description & steps to reproduce

When using KV cache quantization (--cache-type-k q8_0, --cache-type-v q8_0) with the ROCm/HIP backend, VRAM continuously grows during prompt processing. This makes --fit completely wrong, for a simple ~20k tokens prompt, VRAM is growing by more than 2 GiB, at this time, KV cache quantization is unusable, if it's a know issue, then KV cache quantization for the ROCm/HIP backend should simply be disabled in release build or with several warning attached. I did try if it was only K or V related, I reproduced it with K only, but didn't manage to run server with only V quantized, it crashed even when manually setting gpu-layers with --fit off. Vulkan backend doesn't have this problem, fit correctly estimate VRAM needed and it doesn't grow during prompt processing. It does seem to happen with every models that I have tried, forcing me to use Vulkan backend which is slower on my system, but at least doesn't have that issue.

I'm using a RDNA 2 GPU with ROCm 7.2.3.

I did see some potentially related issue, notably #19979 which is closed as completed. But I do still see a similar problem.

First Bad Commit

No response

Relevant log output

rocm-kv-q8_0-vram-leak.log
rocm-kv-f16-no-leak.log
vulkan-kv-q8_0-no-leak.log
rocm-kv-v-q8_0-crash.log

[OPS-NeoRetro]

Can you try the same commands on your problem PC but with -fa off?

[OPS-NeoRetro]

The RX 6700 XT is unfortunately not fully supported by ROCm, especially version 7.2.3/7.2.4 because AMD simply neglected maintenance for older GPUs, and they would rather support the big AI server GPUs instead

[OPS-NeoRetro]

Your GPU (gfx1031) is not shown in the official compatibility matrix, so llama.cpp must use their fork of ROCm, or you should switch to Vulkan inference instead if you really need flash attention

[OPS-NeoRetro]

In #19979, some people there recommended the person who opened the issue to switch to the Vulkan version

[OPS-NeoRetro]

You should read #19979 more, and since ROCm is unstable on mid-range cards and low-end ones don't even support it, you should just use the Vulkan version instead

[OPS-NeoRetro]

@Edremon, just use the RADV driver if you can, you can then use almost any AMD GPU that can run Gemma 4 31B with the Vulkan version

[Edremon]

Can you try the same commands on your problem PC but with -fa off?

I don't think this is something interesting to do, the VRAM usage would be way too much to even attempt to do similar tests.

The RX 6700 XT is unfortunately not fully supported by ROCm, especially version 7.2.3/7.2.4 because AMD simply neglected maintenance for older GPUs, and they would rather support the big AI server GPUs instead

That's not exactly true, but not important for the purpose of this issue anyway.

Your GPU (gfx1031) is not shown in the official compatibility matrix, so llama.cpp must use their fork of ROCm, or you should switch to Vulkan inference instead if you really need flash attention

I think you don't know what you are talking about. gfx1031 (and gfx1032/gfx1033/gfx1034/gfx35) can be considered as gfx1030, they share the same ISA.

In #19979, some people there recommended the person who opened the issue to switch to the Vulkan version
You should read #19979 more, and since ROCm is unstable on mid-range cards and low-end ones don't even support it, you should just use the Vulkan version instead
@Edremon, just use the RADV driver if you can, you can then use almost any AMD GPU that can run Gemma 4 31B with the Vulkan version

How is that relevant? I'm reporting a bug in the ROCm backend that doesn't exist in the Vulkan version. I would like the ROCm backend to work correctly since it's faster and usually more maintained since it's getting most of the CUDA code. The ROCm backend is quite a fair bit faster then Vulkan. I'm only forced to use Vulkan right now because I'm frequently using high context and KV quantization is currently broken, with F16 KV cache, ROCm is noticeably faster, and it's also faster than Vulkan with q8 cache, just using way more VRAM, even more than F16.

Please stop spamming my issue if you have no idea what you are talking about.

[boyedarat]

@Edremon do you also receive the following warning in the logs:
0.14.529.829 W llama_sampler_backend_support: device 'ROCm0' does not have support for op TOP_K needed for sampler 'top-k'

Just curious as I get this in my logs.

[Edremon]

This is now fixed. I'm not exactly sure which commit fixed it, but I don't see my VRAM growing anymore when using a quantized KV cache. The estimation is still quite off, I have to adjust it per model to keep ~512 MiB free. The --fit-target I use range from 2560 to -512, but that's obviously a separate issue. The fix could also have come from upgrading ROCm, as I'm now on 7.2.4. In any case, the issue is solved.