CUDA/HIP: Flash Attention tuning (gfx1201) by pwilkin · Pull Request #28102 · ggml-org/llama.cpp

pwilkin · 2026-08-31T16:54:33Z

Overview

So, got my new R9700 PRO. It's nice and shiny and has 32GB VRAM, so I decided to try out Qwen3.8 27B. That was a mistake. The prefill performance at longer contexts was abysmal, so I decided to do something about it. Also managed to fix a HS=256 bug in the general CUDA FA code which was preventing the selection of 256 kernels before.

Additional information

Before:

model	size	params	backend	threads	dev	test	t/s
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512	1101.41 ± 221.33
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128	29.81 ± 0.19
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d40000	426.36 ± 37.60
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d40000	26.75 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d150000	164.42 ± 5.79
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d150000	19.70 ± 0.20

After:

model	size	params	backend	threads	dev	test	t/s
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512	1103.66 ± 254.83
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128	29.87 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d40000	639.46 ± 185.36
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d40000	26.94 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d150000	399.01 ± 30.87
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d150000	19.67 ± 0.39

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes

Geramy · 2026-08-31T17:03:11Z

This looks really good! Such a simple change providing extra performance, good job on finding it!

pwilkin · 2026-09-01T07:34:24Z

I'm not done yet!

model	size	params	backend	threads	dev	test	t/s
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512	1138.55 ± 224.34
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128	29.48 ± 0.83
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d40000	704.92 ± 239.12
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d40000	27.00 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	pp512 @ d150000	456.51 ± 38.04
qwen35 27B IQ4_XS - 4.25 bpw	13.26 GiB	27.32 B	BLAS,CUDA,ROCm,Vulkan	8	ROCm0	tg128 @ d150000	19.67 ± 0.42

IMbackK

nice, looks good

Details

GPU	Model	Microbatch size	Test	t/s master	t/s fattn-wmma-rdna4-256	Speedup
AI PRO R9700	gemma4 26B.A4B Q6_K	8	pp2048@d32768	327.19	308.26	0.94
AI PRO R9700	gemma4 26B.A4B Q6_K	64	pp2048@d32768	662.22	668.99	1.01
AI PRO R9700	gemma4 26B.A4B Q6_K	512	pp2048@d32768	1263.12	1256.57	0.99
AI PRO R9700	gemma4 26B.A4B Q6_K	1024	pp2048@d32768	1366.90	1495.99	1.09
AI PRO R9700	gpt-oss 20B MXFP4 MoE	8	pp2048@d32768	523.46	525.46	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	64	pp2048@d32768	1408.82	1364.30	0.97
AI PRO R9700	gpt-oss 20B MXFP4 MoE	512	pp2048@d32768	2588.42	2838.27	1.10
AI PRO R9700	gpt-oss 20B MXFP4 MoE	1024	pp2048@d32768	2869.95	3281.25	1.14
AI PRO R9700	lfm2moe 8B.A1B Q8_0	8	pp2048@d32768	817.76	817.97	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	64	pp2048@d32768	2947.91	2947.64	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	512	pp2048@d32768	7976.07	7986.46	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	1024	pp2048@d32768	9433.99	8928.84	0.95
AI PRO R9700	llama 8B Q8_0	8	pp2048@d32768	312.91	312.85	1.00
AI PRO R9700	llama 8B Q8_0	64	pp2048@d32768	1209.11	1181.70	0.98
AI PRO R9700	llama 8B Q8_0	512	pp2048@d32768	1145.38	1705.73	1.49
AI PRO R9700	llama 8B Q8_0	1024	pp2048@d32768	1064.61	1587.85	1.49
AI PRO R9700	qwen35 27B Q5_K_M	8	pp2048@d32768	75.64	72.25	0.96
AI PRO R9700	qwen35 27B Q5_K_M	64	pp2048@d32768	372.91	479.10	1.28
AI PRO R9700	qwen35 27B Q5_K_M	512	pp2048@d32768	468.45	707.39	1.51
AI PRO R9700	qwen35 27B Q5_K_M	1024	pp2048@d32768	475.40	708.34	1.49

<\details>

IMbackK · 2026-09-03T07:46:58Z

+        bool use_stream_k = cc >= GGML_CUDA_CC_ADA_LOVELACE || amd_wmma_available(cc) || tiles_efficiency_percent < 75;
+        if (amd_wmma_available(cc) && ntiles_dst >= 2*max_blocks && tiles_efficiency_percent >= 75) {
+            use_stream_k = false;
+        }


we might want to refactor this into a helper function with table at this point.

sbstnh · 2026-09-03T23:03:48Z

can confirm the performance improvements on my R9700. Hope this PR will be merged soon.

pwilkin · 2026-09-05T16:43:10Z

@IMbackK added the helper.

JohannesGaessler

Please avoid piling on many unrelated changes into a single PR like this.

pwilkin · 2026-09-05T17:09:13Z

Yeah, I just noticed the GDA changes got swept in as well, I'll move them to a separate branch.

pwilkin · 2026-09-05T17:37:35Z

@JohannesGaessler aight, cleaned it up to just the FATTN changes.

JohannesGaessler · 2026-09-05T17:53:45Z

For this PR, please reduce it to just the configuration changes.
Make a follow-up with only the changes for the stream-k logic.
The other changes I'm not sure are logically correct so I would like to treat them separately.

It is a lot easier for me to check the performance with separate PRs. I'm available to work on this this weekend so I think we can get it done if you are as well.

pwilkin · 2026-09-05T22:48:24Z

Aight, will try that.

JohannesGaessler · 2026-09-06T06:58:28Z

Actually, looking at the order the commits were done in it's not clear to me that the config change and the change to stream-k can be separated. But Claude is definitely wrong about the supposed race condition it fixed.

IMbackK · 2026-09-06T07:28:13Z

imo the state of this pr was fine at 2d55b5f

IMbackK · 2026-09-06T07:49:06Z

nvm, your right d6feff3 is not necessary its just reading its own rows in that case, got bamboozled by that one. up to 2d55b5f minus d6feff3 is good.

pwilkin · 2026-09-06T08:38:27Z

Yeah, shouldn't have trusted Claude on that one. Reverting that.

IMbackK · 2026-09-06T08:48:23Z

mind doing a git rebase -i instead? gets kinda confusing to review like this.

JohannesGaessler · 2026-09-06T09:13:48Z

-    // swizzle the tile stride for K and V based on the batch size.
    constexpr int stride_tile_K = ggml_cuda_fattn_smem_swizzle::tile_stride(nbatch_K2);
+#if defined(AMD_WMMA_AVAILABLE)
+    constexpr int stride_tile_V = V_is_K_view ? stride_tile_K : nbatch_V2 + 6;
+#else
    constexpr int stride_tile_V = V_is_K_view ? stride_tile_K : ggml_cuda_fattn_smem_swizzle::tile_stride(nbatch_V2);
+#endif // defined(AMD_WMMA_AVAILABLE)


Again, please submit changes to the actual device code beyond changes to the config and the host-side orchestration separately. I would be surprised if this is actually the correct padding to minimize LDS bank conflicts.

Assisted-by: Claude Assisted-by: Codex

pwilkin · 2026-09-06T13:20:28Z

Rebased, squashed, hopefully it's good now.

JohannesGaessler · 2026-09-06T13:42:14Z

+    if (GGML_CUDA_CC_IS_RDNA4(cc)) {
+        if (use_gqa_opt && gqa_ratio % 8 == 0) {
+            ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 8>(ctx, dst);
+            return;
+        }
+
+        if (use_gqa_opt && gqa_ratio % 4 == 0) {
+            ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 4>(ctx, dst);
+            return;
+        }
+
+        if (use_gqa_opt && gqa_ratio % 2 == 0) {
+            ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 2>(ctx, dst);
+            return;
+        }
+    }
+


I don't think these extra code branches are needed.

Looking at the exact parameters for Qwen 3.5, what these seem to be doing is launch different kernels for GQA ratios that are not a power of 2, Qwen 3.5 has 6.

Yeah, is that wrong?

I don't know yet.

JohannesGaessler · 2026-09-06T20:24:57Z

I observed some performance regressions on my RX 9060 XT with this PR. I pushed revised logic for whether or not to use stream-k, can you check how that performs on your system? This is the data I got:

PW vs. master

GPU	Model	Microbatch size	Test	t/s master	t/s `a3b28f9`	Speedup
RX 9060 XT	gemma 2B Q4_0	1	pp1024	199.77	199.51	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d1024	195.02	194.96	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d2048	193.32	192.71	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d4096	185.70	185.87	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d8192	176.77	176.34	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d16384	159.14	158.58	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d32768	133.21	133.01	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024	251.97	244.16	0.97
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d1024	250.26	241.04	0.96
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d2048	247.19	235.81	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d4096	247.23	234.11	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d8192	239.84	227.26	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d16384	225.61	214.44	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d32768	201.53	191.32	0.95
RX 9060 XT	gemma 2B Q4_0	4	pp1024	497.70	490.16	0.98
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d1024	480.05	473.93	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d2048	474.65	469.08	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d4096	462.00	455.97	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d8192	436.76	436.99	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d16384	398.36	406.74	1.02
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d32768	343.28	369.11	1.08
RX 9060 XT	gemma 2B Q4_0	8	pp1024	813.96	807.04	0.99
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d1024	793.26	796.88	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d2048	776.17	789.05	1.02
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d4096	739.38	770.05	1.04
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d8192	677.58	745.12	1.10
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d16384	582.69	697.48	1.20
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d32768	447.92	618.61	1.38
RX 9060 XT	gemma 2B Q4_0	16	pp1024	1534.08	1541.23	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d1024	1470.10	1514.17	1.03
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d2048	1404.48	1488.73	1.06
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d4096	1298.53	1431.61	1.10
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d8192	1113.28	1350.10	1.21
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d16384	882.45	1225.79	1.39
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d32768	608.97	1007.76	1.65
RX 9060 XT	gemma 2B Q4_0	32	pp1024	2495.77	2598.26	1.04
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d1024	2308.08	2526.66	1.09
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d2048	2138.33	2463.82	1.15
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d4096	1891.24	2299.89	1.22
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d8192	1523.24	2092.87	1.37
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d16384	1099.01	1779.51	1.62
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d32768	708.51	1353.04	1.91
RX 9060 XT	gemma 2B Q4_0	64	pp1024	3681.43	3768.68	1.02
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d1024	3286.00	3628.76	1.10
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d2048	2951.91	3462.08	1.17
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d4096	2493.95	3212.83	1.29
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d8192	1899.01	2844.27	1.50
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d16384	1291.77	2286.47	1.77
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d32768	784.82	1639.84	2.09
RX 9060 XT	gemma 2B Q4_0	128	pp1024	4925.28	5037.19	1.02
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d1024	4321.66	4701.78	1.09
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d2048	3884.52	4465.72	1.15
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d4096	3231.15	4080.08	1.26
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d8192	2429.50	3465.77	1.43
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d16384	1637.36	2669.78	1.63
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d32768	966.82	1828.07	1.89
RX 9060 XT	gemma 2B Q4_0	256	pp1024	7290.98	7670.59	1.05
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d1024	6013.49	7042.40	1.17
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d2048	5185.53	6697.28	1.29
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d4096	4099.34	6031.23	1.47
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d8192	2910.29	5102.77	1.75
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d16384	1862.82	3937.68	2.11
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d32768	1071.32	2686.63	2.51
RX 9060 XT	gemma 2B Q4_0	512	pp1024	7538.15	8163.80	1.08
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d1024	6284.13	7601.69	1.21
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d2048	5439.49	7168.91	1.32
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d4096	4267.23	6480.79	1.52
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d8192	3024.31	5388.91	1.78
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d16384	1922.31	4049.56	2.11
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d32768	1099.73	2713.10	2.47
RX 9060 XT	llama 1B Q4_0	1	pp1024	326.87	329.53	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d1024	313.94	315.89	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d2048	303.64	303.91	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d4096	285.31	287.63	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d8192	256.54	258.10	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d16384	211.00	211.25	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d32768	155.10	155.04	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024	411.48	412.22	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d1024	397.65	398.61	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d2048	388.50	389.11	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d4096	368.65	369.63	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d8192	333.23	333.75	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d16384	279.96	280.08	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d32768	210.77	210.57	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024	604.69	605.93	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d1024	590.53	594.39	1.01
RX 9060 XT	llama 1B Q4_0	4	pp1024@d2048	582.25	583.94	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d4096	562.41	562.68	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d8192	526.75	527.05	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d16384	465.37	465.15	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d32768	376.68	376.38	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024	1208.72	1215.98	1.01
RX 9060 XT	llama 1B Q4_0	8	pp1024@d1024	1183.24	1189.28	1.01
RX 9060 XT	llama 1B Q4_0	8	pp1024@d2048	1158.59	1159.71	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d4096	1116.13	1117.16	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d8192	1040.70	1040.65	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d16384	915.93	915.79	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d32768	745.76	744.42	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024	2603.47	2595.57	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d1024	2517.27	2527.75	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d2048	2465.07	2466.36	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d4096	2375.96	2373.25	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d8192	2201.16	2201.90	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d16384	1911.19	1907.22	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d32768	1514.74	1512.98	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024	3843.24	3859.03	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d1024	3737.16	3779.37	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d2048	3663.59	3667.18	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d4096	3525.29	3508.37	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d8192	3306.39	3245.74	0.98
RX 9060 XT	llama 1B Q4_0	32	pp1024@d16384	2777.85	2768.55	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d32768	2121.97	2115.00	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024	5335.02	5354.61	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d1024	5219.23	5219.14	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d2048	4979.65	4979.69	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d4096	4681.70	4685.37	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d8192	4132.44	4159.06	1.01
RX 9060 XT	llama 1B Q4_0	64	pp1024@d16384	3372.08	3365.11	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d32768	2413.12	2403.37	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024	8225.01	8084.94	0.98
RX 9060 XT	llama 1B Q4_0	128	pp1024@d1024	7756.62	7739.91	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d2048	7189.73	7152.98	0.99
RX 9060 XT	llama 1B Q4_0	128	pp1024@d4096	6542.08	6534.04	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d8192	5507.57	5518.92	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d16384	4186.53	4201.58	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d32768	2789.97	2783.58	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024	12965.20	13001.92	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d1024	11452.63	11462.87	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d2048	10539.07	10416.98	0.99
RX 9060 XT	llama 1B Q4_0	256	pp1024@d4096	9192.08	9153.15	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d8192	7259.93	7211.04	0.99
RX 9060 XT	llama 1B Q4_0	256	pp1024@d16384	5078.09	5086.47	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d32768	3162.75	3153.39	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024	12388.58	12692.93	1.02
RX 9060 XT	llama 1B Q4_0	512	pp1024@d1024	11988.93	12100.80	1.01
RX 9060 XT	llama 1B Q4_0	512	pp1024@d2048	11252.73	11171.83	0.99
RX 9060 XT	llama 1B Q4_0	512	pp1024@d4096	9744.95	9590.98	0.98
RX 9060 XT	llama 1B Q4_0	512	pp1024@d8192	7656.92	7280.01	0.95
RX 9060 XT	llama 1B Q4_0	512	pp1024@d16384	5247.30	4949.73	0.94
RX 9060 XT	llama 1B Q4_0	512	pp1024@d32768	3195.56	2993.52	0.94
RX 9060 XT	llama 8B Q4_0	1	pp1024	68.21	68.24	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d1024	66.15	66.18	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d2048	64.34	64.37	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d4096	61.09	61.06	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d8192	55.44	55.45	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d16384	46.69	46.69	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d32768	35.56	35.56	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024	100.66	100.58	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d1024	98.28	98.23	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d2048	96.34	96.27	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d4096	92.76	92.62	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d8192	86.07	85.87	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d16384	74.97	75.00	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d32768	59.84	59.81	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024	182.37	182.12	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d1024	177.02	176.78	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d2048	173.13	173.07	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d4096	165.30	165.23	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d8192	151.11	151.10	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d16384	130.26	130.25	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d32768	100.95	100.85	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024	199.43	198.86	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d1024	196.56	196.26	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d2048	194.44	194.77	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d4096	190.47	190.18	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d8192	182.34	182.38	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d16384	169.30	169.17	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d32768	147.88	147.82	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024	660.73	661.43	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d1024	641.12	638.73	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d2048	624.75	624.71	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d4096	595.31	594.47	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d8192	542.94	542.96	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d16384	462.64	462.56	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d32768	357.99	357.78	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024	990.51	991.28	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d1024	953.80	948.60	0.99
RX 9060 XT	llama 8B Q4_0	32	pp1024@d2048	921.67	920.75	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d4096	866.31	864.65	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d8192	769.47	769.72	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d16384	628.25	628.00	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d32768	464.45	461.98	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024	1610.44	1604.77	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d1024	1507.52	1504.59	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d2048	1447.53	1444.09	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d4096	1320.89	1318.74	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d8192	1087.31	1087.43	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d16384	832.08	832.58	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d32768	569.23	566.67	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024	2359.64	2340.84	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d1024	2142.74	2142.86	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d2048	2005.35	2005.38	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d4096	1771.69	1775.94	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d8192	1374.73	1377.59	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d16384	1003.99	1000.71	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d32768	648.49	646.06	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024	2612.44	2595.07	0.99
RX 9060 XT	llama 8B Q4_0	256	pp1024@d1024	2347.26	2372.22	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d2048	2185.63	2206.63	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d4096	1934.63	1967.04	1.02
RX 9060 XT	llama 8B Q4_0	256	pp1024@d8192	1477.02	1607.39	1.09
RX 9060 XT	llama 8B Q4_0	256	pp1024@d16384	1056.31	1186.13	1.12
RX 9060 XT	llama 8B Q4_0	256	pp1024@d32768	674.72	785.23	1.16
RX 9060 XT	llama 8B Q4_0	512	pp1024	2700.30	2701.59	1.00
RX 9060 XT	llama 8B Q4_0	512	pp1024@d1024	2458.57	2490.01	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d2048	2287.08	2301.61	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d4096	2009.40	2031.68	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d8192	1502.39	1641.36	1.09
RX 9060 XT	llama 8B Q4_0	512	pp1024@d16384	1056.29	1196.08	1.13
RX 9060 XT	llama 8B Q4_0	512	pp1024@d32768	674.64	787.39	1.17

JG vs. master

GPU	Model	Microbatch size	Test	t/s master	t/s `0ce2938`	Speedup
RX 9060 XT	gemma 2B Q4_0	1	pp1024	199.77	198.92	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d1024	195.02	195.75	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d2048	193.32	192.97	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d4096	185.70	185.69	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d8192	176.77	176.69	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d16384	159.14	158.96	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d32768	133.21	133.54	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024	251.97	245.62	0.97
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d1024	250.26	237.92	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d2048	247.19	231.55	0.94
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d4096	247.23	234.05	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d8192	239.84	227.32	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d16384	225.61	214.02	0.95
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d32768	201.53	191.30	0.95
RX 9060 XT	gemma 2B Q4_0	4	pp1024	497.70	488.44	0.98
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d1024	480.05	473.32	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d2048	474.65	469.23	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d4096	462.00	455.94	0.99
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d8192	436.76	437.16	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d16384	398.36	405.73	1.02
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d32768	343.28	368.71	1.07
RX 9060 XT	gemma 2B Q4_0	8	pp1024	813.96	804.03	0.99
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d1024	793.26	799.46	1.01
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d2048	776.17	788.50	1.02
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d4096	739.38	771.84	1.04
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d8192	677.58	742.04	1.10
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d16384	582.69	695.22	1.19
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d32768	447.92	621.36	1.39
RX 9060 XT	gemma 2B Q4_0	16	pp1024	1534.08	1517.28	0.99
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d1024	1470.10	1513.59	1.03
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d2048	1404.48	1491.95	1.06
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d4096	1298.53	1435.81	1.11
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d8192	1113.28	1352.22	1.21
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d16384	882.45	1221.14	1.38
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d32768	608.97	1013.05	1.66
RX 9060 XT	gemma 2B Q4_0	32	pp1024	2495.77	2556.41	1.02
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d1024	2308.08	2513.54	1.09
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d2048	2138.33	2447.43	1.14
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d4096	1891.24	2313.25	1.22
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d8192	1523.24	2094.07	1.37
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d16384	1099.01	1770.91	1.61
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d32768	708.51	1353.49	1.91
RX 9060 XT	gemma 2B Q4_0	64	pp1024	3681.43	3722.82	1.01
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d1024	3286.00	3605.65	1.10
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d2048	2951.91	3477.13	1.18
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d4096	2493.95	3217.50	1.29
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d8192	1899.01	2837.30	1.49
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d16384	1291.77	2286.57	1.77
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d32768	784.82	1646.99	2.10
RX 9060 XT	gemma 2B Q4_0	128	pp1024	4925.28	4979.60	1.01
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d1024	4321.66	4739.79	1.10
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d2048	3884.52	4467.60	1.15
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d4096	3231.15	4077.59	1.26
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d8192	2429.50	3451.47	1.42
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d16384	1637.36	2663.55	1.63
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d32768	966.82	1834.82	1.90
RX 9060 XT	gemma 2B Q4_0	256	pp1024	7290.98	7674.55	1.05
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d1024	6013.49	7099.76	1.18
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d2048	5185.53	6663.30	1.28
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d4096	4099.34	6042.98	1.47
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d8192	2910.29	5091.22	1.75
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d16384	1862.82	3942.74	2.12
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d32768	1071.32	2708.68	2.53
RX 9060 XT	gemma 2B Q4_0	512	pp1024	7538.15	8075.33	1.07
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d1024	6284.13	7581.18	1.21
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d2048	5439.49	7169.73	1.32
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d4096	4267.23	6458.02	1.51
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d8192	3024.31	5373.58	1.78
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d16384	1922.31	4017.62	2.09
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d32768	1099.73	2729.53	2.48
RX 9060 XT	llama 1B Q4_0	1	pp1024	326.87	328.13	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d1024	313.94	316.89	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d2048	303.64	306.31	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d4096	285.31	285.68	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d8192	256.54	256.90	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d16384	211.00	211.71	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d32768	155.10	155.09	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024	411.48	413.35	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d1024	397.65	397.89	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d2048	388.50	389.31	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d4096	368.65	369.20	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d8192	333.23	334.24	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d16384	279.96	280.07	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d32768	210.77	210.79	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024	604.69	607.44	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d1024	590.53	592.13	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d2048	582.25	583.17	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d4096	562.41	563.12	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d8192	526.75	527.40	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d16384	465.37	465.15	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d32768	376.68	376.29	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024	1208.72	1215.55	1.01
RX 9060 XT	llama 1B Q4_0	8	pp1024@d1024	1183.24	1185.52	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d2048	1158.59	1162.43	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d4096	1116.13	1116.34	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d8192	1040.70	1042.10	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d16384	915.93	915.60	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d32768	745.76	753.23	1.01
RX 9060 XT	llama 1B Q4_0	16	pp1024	2603.47	2609.29	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d1024	2517.27	2522.66	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d2048	2465.07	2478.37	1.01
RX 9060 XT	llama 1B Q4_0	16	pp1024@d4096	2375.96	2370.15	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d8192	2201.16	2200.86	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d16384	1911.19	1905.40	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d32768	1514.74	1511.23	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024	3843.24	3892.46	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d1024	3737.16	3757.39	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d2048	3663.59	3690.24	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d4096	3525.29	3509.35	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d8192	3306.39	3241.91	0.98
RX 9060 XT	llama 1B Q4_0	32	pp1024@d16384	2777.85	2773.07	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d32768	2121.97	2119.66	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024	5335.02	5431.01	1.02
RX 9060 XT	llama 1B Q4_0	64	pp1024@d1024	5219.23	5179.40	0.99
RX 9060 XT	llama 1B Q4_0	64	pp1024@d2048	4979.65	5029.23	1.01
RX 9060 XT	llama 1B Q4_0	64	pp1024@d4096	4681.70	4680.36	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d8192	4132.44	4162.66	1.01
RX 9060 XT	llama 1B Q4_0	64	pp1024@d16384	3372.08	3364.45	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d32768	2413.12	2416.83	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024	8225.01	8210.65	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d1024	7756.62	7654.44	0.99
RX 9060 XT	llama 1B Q4_0	128	pp1024@d2048	7189.73	7166.29	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d4096	6542.08	6511.14	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d8192	5507.57	5528.07	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d16384	4186.53	4174.03	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d32768	2789.97	2798.74	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024	12965.20	13003.61	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d1024	11452.63	11463.06	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d2048	10539.07	10549.90	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d4096	9192.08	9144.06	0.99
RX 9060 XT	llama 1B Q4_0	256	pp1024@d8192	7259.93	7274.18	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d16384	5078.09	5060.10	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d32768	3162.75	3156.67	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024	12388.58	12430.03	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024@d1024	11988.93	11833.29	0.99
RX 9060 XT	llama 1B Q4_0	512	pp1024@d2048	11252.73	11289.88	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024@d4096	9744.95	9743.92	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024@d8192	7656.92	7671.60	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024@d16384	5247.30	5200.23	0.99
RX 9060 XT	llama 1B Q4_0	512	pp1024@d32768	3195.56	3195.23	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024	68.21	68.24	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d1024	66.15	66.19	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d2048	64.34	64.41	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d4096	61.09	61.01	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d8192	55.44	55.53	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d16384	46.69	46.72	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d32768	35.56	35.56	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024	100.66	100.52	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d1024	98.28	98.20	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d2048	96.34	96.35	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d4096	92.76	92.67	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d8192	86.07	86.04	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d16384	74.97	75.06	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d32768	59.84	59.80	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024	182.37	181.86	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d1024	177.02	176.88	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d2048	173.13	172.84	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d4096	165.30	165.19	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d8192	151.11	151.20	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d16384	130.26	130.37	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d32768	100.95	100.74	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024	199.43	199.17	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d1024	196.56	196.15	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d2048	194.44	194.35	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d4096	190.47	189.79	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d8192	182.34	182.97	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d16384	169.30	169.50	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d32768	147.88	147.57	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024	660.73	660.13	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d1024	641.12	638.67	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d2048	624.75	624.18	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d4096	595.31	593.88	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d8192	542.94	543.14	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d16384	462.64	463.15	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d32768	357.99	357.17	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024	990.51	989.06	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d1024	953.80	949.24	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d2048	921.67	919.02	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d4096	866.31	863.64	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d8192	769.47	769.91	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d16384	628.25	628.19	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d32768	464.45	462.46	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024	1610.44	1609.12	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d1024	1507.52	1492.75	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024@d2048	1447.53	1435.53	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024@d4096	1320.89	1314.26	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024@d8192	1087.31	1089.54	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d16384	832.08	831.96	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d32768	569.23	567.28	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024	2359.64	2355.97	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d1024	2142.74	2125.51	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d2048	2005.35	1992.39	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d4096	1771.69	1759.99	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d8192	1374.73	1380.62	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d16384	1003.99	1003.00	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d32768	648.49	646.66	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024	2612.44	2617.89	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024@d1024	2347.26	2361.80	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d2048	2185.63	2204.24	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d4096	1934.63	1953.74	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d8192	1477.02	1614.86	1.09
RX 9060 XT	llama 8B Q4_0	256	pp1024@d16384	1056.31	1187.05	1.12
RX 9060 XT	llama 8B Q4_0	256	pp1024@d32768	674.72	780.75	1.16
RX 9060 XT	llama 8B Q4_0	512	pp1024	2700.30	2722.46	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d1024	2458.57	2483.76	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d2048	2287.08	2308.23	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d4096	2009.40	2019.50	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d8192	1502.39	1651.06	1.10
RX 9060 XT	llama 8B Q4_0	512	pp1024@d16384	1056.29	1197.76	1.13
RX 9060 XT	llama 8B Q4_0	512	pp1024@d32768	674.64	783.05	1.16

JG vs. PW

GPU	Model	Microbatch size	Test	t/s `a3b28f9`	t/s `0ce2938`	Speedup
RX 9060 XT	gemma 2B Q4_0	1	pp1024	199.51	198.92	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d1024	194.96	195.75	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d2048	192.71	192.97	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d4096	185.87	185.69	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d8192	176.34	176.69	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d16384	158.58	158.96	1.00
RX 9060 XT	gemma 2B Q4_0	1	pp1024@d32768	133.01	133.54	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024	244.16	245.62	1.01
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d1024	241.04	237.92	0.99
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d2048	235.81	231.55	0.98
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d4096	234.11	234.05	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d8192	227.26	227.32	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d16384	214.44	214.02	1.00
RX 9060 XT	gemma 2B Q4_0	2	pp1024@d32768	191.32	191.30	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024	490.16	488.44	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d1024	473.93	473.32	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d2048	469.08	469.23	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d4096	455.97	455.94	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d8192	436.99	437.16	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d16384	406.74	405.73	1.00
RX 9060 XT	gemma 2B Q4_0	4	pp1024@d32768	369.11	368.71	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024	807.04	804.03	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d1024	796.88	799.46	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d2048	789.05	788.50	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d4096	770.05	771.84	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d8192	745.12	742.04	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d16384	697.48	695.22	1.00
RX 9060 XT	gemma 2B Q4_0	8	pp1024@d32768	618.61	621.36	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024	1541.23	1517.28	0.98
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d1024	1514.17	1513.59	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d2048	1488.73	1491.95	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d4096	1431.61	1435.81	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d8192	1350.10	1352.22	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d16384	1225.79	1221.14	1.00
RX 9060 XT	gemma 2B Q4_0	16	pp1024@d32768	1007.76	1013.05	1.01
RX 9060 XT	gemma 2B Q4_0	32	pp1024	2598.26	2556.41	0.98
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d1024	2526.66	2513.54	0.99
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d2048	2463.82	2447.43	0.99
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d4096	2299.89	2313.25	1.01
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d8192	2092.87	2094.07	1.00
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d16384	1779.51	1770.91	1.00
RX 9060 XT	gemma 2B Q4_0	32	pp1024@d32768	1353.04	1353.49	1.00
RX 9060 XT	gemma 2B Q4_0	64	pp1024	3768.68	3722.82	0.99
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d1024	3628.76	3605.65	0.99
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d2048	3462.08	3477.13	1.00
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d4096	3212.83	3217.50	1.00
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d8192	2844.27	2837.30	1.00
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d16384	2286.47	2286.57	1.00
RX 9060 XT	gemma 2B Q4_0	64	pp1024@d32768	1639.84	1646.99	1.00
RX 9060 XT	gemma 2B Q4_0	128	pp1024	5037.19	4979.60	0.99
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d1024	4701.78	4739.79	1.01
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d2048	4465.72	4467.60	1.00
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d4096	4080.08	4077.59	1.00
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d8192	3465.77	3451.47	1.00
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d16384	2669.78	2663.55	1.00
RX 9060 XT	gemma 2B Q4_0	128	pp1024@d32768	1828.07	1834.82	1.00
RX 9060 XT	gemma 2B Q4_0	256	pp1024	7670.59	7674.55	1.00
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d1024	7042.40	7099.76	1.01
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d2048	6697.28	6663.30	0.99
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d4096	6031.23	6042.98	1.00
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d8192	5102.77	5091.22	1.00
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d16384	3937.68	3942.74	1.00
RX 9060 XT	gemma 2B Q4_0	256	pp1024@d32768	2686.63	2708.68	1.01
RX 9060 XT	gemma 2B Q4_0	512	pp1024	8163.80	8075.33	0.99
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d1024	7601.69	7581.18	1.00
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d2048	7168.91	7169.73	1.00
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d4096	6480.79	6458.02	1.00
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d8192	5388.91	5373.58	1.00
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d16384	4049.56	4017.62	0.99
RX 9060 XT	gemma 2B Q4_0	512	pp1024@d32768	2713.10	2729.53	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024	329.53	328.13	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d1024	315.89	316.89	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d2048	303.91	306.31	1.01
RX 9060 XT	llama 1B Q4_0	1	pp1024@d4096	287.63	285.68	0.99
RX 9060 XT	llama 1B Q4_0	1	pp1024@d8192	258.10	256.90	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d16384	211.25	211.71	1.00
RX 9060 XT	llama 1B Q4_0	1	pp1024@d32768	155.04	155.09	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024	412.22	413.35	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d1024	398.61	397.89	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d2048	389.11	389.31	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d4096	369.63	369.20	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d8192	333.75	334.24	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d16384	280.08	280.07	1.00
RX 9060 XT	llama 1B Q4_0	2	pp1024@d32768	210.57	210.79	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024	605.93	607.44	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d1024	594.39	592.13	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d2048	583.94	583.17	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d4096	562.68	563.12	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d8192	527.05	527.40	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d16384	465.15	465.15	1.00
RX 9060 XT	llama 1B Q4_0	4	pp1024@d32768	376.38	376.29	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024	1215.98	1215.55	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d1024	1189.28	1185.52	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d2048	1159.71	1162.43	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d4096	1117.16	1116.34	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d8192	1040.65	1042.10	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d16384	915.79	915.60	1.00
RX 9060 XT	llama 1B Q4_0	8	pp1024@d32768	744.42	753.23	1.01
RX 9060 XT	llama 1B Q4_0	16	pp1024	2595.57	2609.29	1.01
RX 9060 XT	llama 1B Q4_0	16	pp1024@d1024	2527.75	2522.66	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d2048	2466.36	2478.37	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d4096	2373.25	2370.15	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d8192	2201.90	2200.86	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d16384	1907.22	1905.40	1.00
RX 9060 XT	llama 1B Q4_0	16	pp1024@d32768	1512.98	1511.23	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024	3859.03	3892.46	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d1024	3779.37	3757.39	0.99
RX 9060 XT	llama 1B Q4_0	32	pp1024@d2048	3667.18	3690.24	1.01
RX 9060 XT	llama 1B Q4_0	32	pp1024@d4096	3508.37	3509.35	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d8192	3245.74	3241.91	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d16384	2768.55	2773.07	1.00
RX 9060 XT	llama 1B Q4_0	32	pp1024@d32768	2115.00	2119.66	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024	5354.61	5431.01	1.01
RX 9060 XT	llama 1B Q4_0	64	pp1024@d1024	5219.14	5179.40	0.99
RX 9060 XT	llama 1B Q4_0	64	pp1024@d2048	4979.69	5029.23	1.01
RX 9060 XT	llama 1B Q4_0	64	pp1024@d4096	4685.37	4680.36	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d8192	4159.06	4162.66	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d16384	3365.11	3364.45	1.00
RX 9060 XT	llama 1B Q4_0	64	pp1024@d32768	2403.37	2416.83	1.01
RX 9060 XT	llama 1B Q4_0	128	pp1024	8084.94	8210.65	1.02
RX 9060 XT	llama 1B Q4_0	128	pp1024@d1024	7739.91	7654.44	0.99
RX 9060 XT	llama 1B Q4_0	128	pp1024@d2048	7152.98	7166.29	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d4096	6534.04	6511.14	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d8192	5518.92	5528.07	1.00
RX 9060 XT	llama 1B Q4_0	128	pp1024@d16384	4201.58	4174.03	0.99
RX 9060 XT	llama 1B Q4_0	128	pp1024@d32768	2783.58	2798.74	1.01
RX 9060 XT	llama 1B Q4_0	256	pp1024	13001.92	13003.61	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d1024	11462.87	11463.06	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d2048	10416.98	10549.90	1.01
RX 9060 XT	llama 1B Q4_0	256	pp1024@d4096	9153.15	9144.06	1.00
RX 9060 XT	llama 1B Q4_0	256	pp1024@d8192	7211.04	7274.18	1.01
RX 9060 XT	llama 1B Q4_0	256	pp1024@d16384	5086.47	5060.10	0.99
RX 9060 XT	llama 1B Q4_0	256	pp1024@d32768	3153.39	3156.67	1.00
RX 9060 XT	llama 1B Q4_0	512	pp1024	12692.93	12430.03	0.98
RX 9060 XT	llama 1B Q4_0	512	pp1024@d1024	12100.80	11833.29	0.98
RX 9060 XT	llama 1B Q4_0	512	pp1024@d2048	11171.83	11289.88	1.01
RX 9060 XT	llama 1B Q4_0	512	pp1024@d4096	9590.98	9743.92	1.02
RX 9060 XT	llama 1B Q4_0	512	pp1024@d8192	7280.01	7671.60	1.05
RX 9060 XT	llama 1B Q4_0	512	pp1024@d16384	4949.73	5200.23	1.05
RX 9060 XT	llama 1B Q4_0	512	pp1024@d32768	2993.52	3195.23	1.07
RX 9060 XT	llama 8B Q4_0	1	pp1024	68.24	68.24	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d1024	66.18	66.19	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d2048	64.37	64.41	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d4096	61.06	61.01	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d8192	55.45	55.53	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d16384	46.69	46.72	1.00
RX 9060 XT	llama 8B Q4_0	1	pp1024@d32768	35.56	35.56	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024	100.58	100.52	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d1024	98.23	98.20	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d2048	96.27	96.35	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d4096	92.62	92.67	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d8192	85.87	86.04	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d16384	75.00	75.06	1.00
RX 9060 XT	llama 8B Q4_0	2	pp1024@d32768	59.81	59.80	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024	182.12	181.86	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d1024	176.78	176.88	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d2048	173.07	172.84	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d4096	165.23	165.19	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d8192	151.10	151.20	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d16384	130.25	130.37	1.00
RX 9060 XT	llama 8B Q4_0	4	pp1024@d32768	100.85	100.74	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024	198.86	199.17	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d1024	196.26	196.15	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d2048	194.77	194.35	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d4096	190.18	189.79	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d8192	182.38	182.97	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d16384	169.17	169.50	1.00
RX 9060 XT	llama 8B Q4_0	8	pp1024@d32768	147.82	147.57	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024	661.43	660.13	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d1024	638.73	638.67	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d2048	624.71	624.18	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d4096	594.47	593.88	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d8192	542.96	543.14	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d16384	462.56	463.15	1.00
RX 9060 XT	llama 8B Q4_0	16	pp1024@d32768	357.78	357.17	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024	991.28	989.06	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d1024	948.60	949.24	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d2048	920.75	919.02	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d4096	864.65	863.64	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d8192	769.72	769.91	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d16384	628.00	628.19	1.00
RX 9060 XT	llama 8B Q4_0	32	pp1024@d32768	461.98	462.46	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024	1604.77	1609.12	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d1024	1504.59	1492.75	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024@d2048	1444.09	1435.53	0.99
RX 9060 XT	llama 8B Q4_0	64	pp1024@d4096	1318.74	1314.26	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d8192	1087.43	1089.54	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d16384	832.58	831.96	1.00
RX 9060 XT	llama 8B Q4_0	64	pp1024@d32768	566.67	567.28	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024	2340.84	2355.97	1.01
RX 9060 XT	llama 8B Q4_0	128	pp1024@d1024	2142.86	2125.51	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d2048	2005.38	1992.39	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d4096	1775.94	1759.99	0.99
RX 9060 XT	llama 8B Q4_0	128	pp1024@d8192	1377.59	1380.62	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d16384	1000.71	1003.00	1.00
RX 9060 XT	llama 8B Q4_0	128	pp1024@d32768	646.06	646.66	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024	2595.07	2617.89	1.01
RX 9060 XT	llama 8B Q4_0	256	pp1024@d1024	2372.22	2361.80	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024@d2048	2206.63	2204.24	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024@d4096	1967.04	1953.74	0.99
RX 9060 XT	llama 8B Q4_0	256	pp1024@d8192	1607.39	1614.86	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024@d16384	1186.13	1187.05	1.00
RX 9060 XT	llama 8B Q4_0	256	pp1024@d32768	785.23	780.75	0.99
RX 9060 XT	llama 8B Q4_0	512	pp1024	2701.59	2722.46	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d1024	2490.01	2483.76	1.00
RX 9060 XT	llama 8B Q4_0	512	pp1024@d2048	2301.61	2308.23	1.00
RX 9060 XT	llama 8B Q4_0	512	pp1024@d4096	2031.68	2019.50	0.99
RX 9060 XT	llama 8B Q4_0	512	pp1024@d8192	1641.36	1651.06	1.01
RX 9060 XT	llama 8B Q4_0	512	pp1024@d16384	1196.08	1197.76	1.00
RX 9060 XT	llama 8B Q4_0	512	pp1024@d32768	787.39	783.05	0.99

JohannesGaessler · 2026-09-06T20:28:09Z

Sorry, the columns were swapped in the "PW vs. master" table, this should be the correct data.

eugene-su · 2026-09-07T03:10:49Z

Just leaving this here.

Without the patch.

Device 0: AMD Radeon RX 9070 XT, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB

model	size	test	t/s
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048	1282.84 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64	29.82 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d16384	352.02 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d16384	27.52 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d32768	262.59 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d32768	19.83 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d65535	357.37 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d65535	12.25 ± 0.00

After the patch (a huge boost!)
Device 0: AMD Radeon RX 9070 XT, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB

Device 0: AMD Radeon RX 9070 XT, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB

model	size	test	t/s
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048	1332.62 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64	29.86 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d16384	428.81 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d16384	23.86 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d32768	936.58 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d32768	19.90 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	pp2048 @ d65535	727.25 ± 0.00
qwen35 27B IQ4_XS - 4.25 bpw	15.50 GiB	tg64 @ d65535	12.16 ± 0.00

But I'm noticing a strange slowdown on pp2048 @ d16384.

pwilkin requested review from a team and ggerganov as code owners August 31, 2026 16:54

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 31, 2026

IMbackK self-assigned this Aug 31, 2026

pwilkin force-pushed the fattn-wmma-rdna4-256 branch from 6b24ee9 to d68f876 Compare September 1, 2026 07:54

IMbackK approved these changes Sep 3, 2026

View reviewed changes

JohannesGaessler requested changes Sep 5, 2026

View reviewed changes

JohannesGaessler self-assigned this Sep 5, 2026

JohannesGaessler reviewed Sep 6, 2026

View reviewed changes

pwilkin added 2 commits September 6, 2026 14:48

HIP: enable mma FA for head size 256 on RDNA4, tune configs

d856932

Assisted-by: Claude Assisted-by: Codex

HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

a3b28f9

Assisted-by: Claude Assisted-by: Codex

pwilkin force-pushed the fattn-wmma-rdna4-256 branch from f6264d8 to a3b28f9 Compare September 6, 2026 13:08

JohannesGaessler reviewed Sep 6, 2026

View reviewed changes

DanoPTT mentioned this pull request Sep 6, 2026

Eval bug: with -np 2 and --kv-unified, prompt processing drops 42-54% from the second long request onward (single GPU, sequential requests, no spill, no speculation) #28495

Open

revise stream_k logic

1873e6e

Uh oh!

Conversation

pwilkin commented Aug 31, 2026

Overview

Additional information

Requirements

Uh oh!

Geramy commented Aug 31, 2026

Uh oh!

pwilkin commented Sep 1, 2026

Uh oh!

IMbackK left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IMbackK Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

sbstnh commented Sep 3, 2026

Uh oh!

pwilkin commented Sep 5, 2026

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Sep 5, 2026

Uh oh!

pwilkin commented Sep 5, 2026

Uh oh!

JohannesGaessler commented Sep 5, 2026

Uh oh!

pwilkin commented Sep 5, 2026

Uh oh!

JohannesGaessler commented Sep 6, 2026

Uh oh!

IMbackK commented Sep 6, 2026

Uh oh!

IMbackK commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Sep 6, 2026

Uh oh!

IMbackK commented Sep 6, 2026

Uh oh!

JohannesGaessler Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Sep 6, 2026

Uh oh!

JohannesGaessler Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

pwilkin Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented Sep 6, 2026

Uh oh!

eugene-su commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

IMbackK left a comment •

edited

Loading

IMbackK commented Sep 6, 2026 •

edited

Loading

JohannesGaessler commented Sep 6, 2026 •

edited

Loading