Performance VULKAN vs CUDA

[cumal]

Hi every one,
After installing llamacpp on my server, I decided to test it into an old pc and see how it runs.
Specs:
Intel Core2 Quad Desktop Q6600
4gb DDR2 Ram
NVIDIA GT 1030 2gb
SSD 256gb
I installed ubuntu server and start testing (The model is from HF):

Download linux x64 binary from github releases

load_backend: loaded RPC backend from /llamacpp/llama-b9102/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GT 1030 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from /llamacpp/llama-b9102/libggml-vulkan.so
load_backend: loaded CPU backend from /llamacpp/llama-b9102/libggml-cpu-x64.so

model	size	params	backend	ngl	n_batch	type_k	type_v	fa	test	t/s
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	Vulkan	99	1024	q8_0	q8_0	1	pp512	494.23 ± 0.55
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	Vulkan	99	1024	q8_0	q8_0	1	tg128	64.77 ± 0.22

Compiled for Vulkan:
cmake -B build -DGGML_VULKAN=1

_0 -ctv q8_0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GT 1030 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	n_batch	type_k	type_v	fa	test	t/s
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	Vulkan	99	1024	q8_0	q8_0	1	pp512	242.19 ± 0.40
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	Vulkan	99	1024	q8_0	q8_0	1	tg128	64.36 ± 0.01

Compiled for CUDA:
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 1991 MiB):
Device 0: NVIDIA GeForce GT 1030, compute capability 6.1, VMM: yes, VRAM: 1991 MiB

model	size	params	backend	ngl	n_batch	type_k	type_v	fa	test	t/s
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	CUDA	99	1024	q8_0	q8_0	1	pp512	834.55 ± 2.12
qwen3 0.6B Q4_K - Medium	372.65 MiB	596.05 M	CUDA	99	1024	q8_0	q8_0	1	tg128	40.72 ± 0.02

Anyone can explain me why using Vulkan tg increases? And why using Cuda the pp increases?
Is there any way to get the mix of pp ~800t/s and tg ~60t/s?

Thanks!!

[aivisionslab-studios]

Tested this directly on a different card (AMD RX 580, Polaris/Vulkan — warp size 64, int dot: 0) using the exact same model file as cumal: Qwen3 0.6B Q4_K_M, 372.65 MiB. Base config: -ngl 99 -b 1024 -fa 1.

Baseline (q8_0/q8_0, averaged across 3 runs):

pp512: ~990 t/s — well ahead of both your CUDA (834) and Vulkan (242-494) numbers, makes sense given the RX 580 is a bigger/newer card.
tg128: ~30.8 t/s — actually below your GT 1030 on both backends, despite the stronger card. That's the part worth digging into.

KV cache type matters more than backend choice:

K type | V type | pp512 | tg128 -- | -- | -- | -- f16 | f16 | 940.38 ± 6.17 | 37.14 ± 0.17 q8_0 | q8_0 | 974.39 ± 30.98 | 30.51 ± 0.45 f16 | q8_0 (mixed) | 346.79 ± 7.60 | 12.89 ± 0.28 q8_0 | f16 (mixed) | 524.27 ± 5.66 | 12.68 ± 0.31

Mixing K and V types is the real trap — it tanks both pp and tg hard, likely falling back to a slower path when types differ. Pure f16 beats pure q8_0 on tg by ~22% here (probably no per-token dequant overhead, since Polaris has no native int8 path), at a small pp cost (~3.6%). Also worth noting: this card requires -fa 1 just to use a quantized KV cache at all — tried -ctk q8_0 -ctv q8_0 without -fa and context creation failed outright.

Side note on -ngl 99: also re-ran the f16/f16 case without specifying -ngl at all, to check if the explicit flag was overriding some smarter default placement. It wasn't — statistically identical (pp512: 935.47 ± 9.86, tg128: 37.32 ± 0.22). For a model this small relative to 8GB VRAM there's no real placement decision to make, so default and "offload everything" land in the same place. Only matters once the model doesn't fit in VRAM.

Bottom line on "can I get both pp and tg high": backend choice isn't the main lever here — matching K/V cache types is. f16/f16 if tg matters more (chat), q8_0/q8_0 if pp matters more (long prompts), never mix them.

Reproducible with: llama-bench -m <model> --device Vulkan0 -ngl 99 -b 1024 -fa 1 -ctk f16,q8_0 -ctv f16,q8_0 -p 512 -n 128

If you're curious how far this kind of legacy hardware revival goes — Vulkan's been supported since 2016/2017, so this isn't AMD-specific. I've documented the full stack (llama.cpp + stable-diffusion.cpp, no CUDA/ROCm) on an RX 580, including an interactive step-by-step "Cookbook" for setting up local AI on old Nvidia/AMD/Intel GPUs: https://github.com/aivisionslab-studios/rx580-local-ai-guide

Hardware doesn't die — it just waits for the right software.