HIP: tune rdna 3 mmq configs by itterative · Pull Request #26284 · ggml-org/llama.cpp

itterative · 2026-07-29T15:55:00Z

Overview

Builds on #26199 to tune further the mmq configs for RDNA 3 and RDNA 4.

Additional information

The results are from branching from e1af89a, not including the Q2_K and Q6_K from #25940 .

In terms of results, it seems like most of these configurations are more or less equivalent, I have many changes mainly because the script would recreate the mmq config file during the sweep. For RDNA 4, I can drop the tuned parts which give no changes (like occupancy 4 vs occupancy 2)

RDNA 3: 7900 XTX

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	366.64	370.16	1.01
llama 8B IQ1_M - 1.75 bpw	32	pp2048	659.70	675.28	1.02
llama 8B IQ1_M - 1.75 bpw	64	pp2048	1005.46	1015.52	1.01
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1563.64	1583.76	1.01
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2988.66	3067.91	1.03
llama 8B IQ1_M - 1.75 bpw	512	pp2048	3110.02	3102.60	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	3146.04	3161.69	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	3352.84	3355.57	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1006.70	1047.20	1.04
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1427.10	1506.89	1.06
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1422.22	1482.54	1.04
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	1879.44	2029.09	1.08
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	3097.16	3368.97	1.09
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	3533.58	3551.07	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	3632.33	3691.73	1.02
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	3697.61	3676.34	0.99
llama 8B IQ2_M - 2.7 bpw	16	pp2048	700.83	703.85	1.00
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1117.51	1160.73	1.04
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1439.19	1438.43	1.00
llama 8B IQ2_M - 2.7 bpw	128	pp2048	1715.58	1865.17	1.09
llama 8B IQ2_M - 2.7 bpw	256	pp2048	2902.22	3002.41	1.03
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3014.57	3009.82	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3043.63	3068.94	1.01
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3221.52	3215.72	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	686.66	688.61	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1097.71	1138.75	1.04
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1369.84	1476.52	1.08
llama 8B IQ2_S - 2.5 bpw	128	pp2048	1715.20	1907.87	1.11
llama 8B IQ2_S - 2.5 bpw	256	pp2048	2923.40	3031.06	1.04
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3026.75	3024.00	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3032.76	3084.81	1.02
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3209.26	3222.97	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	680.61	691.73	1.02
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1097.97	1124.59	1.02
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1143.83	1215.88	1.06
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	1528.16	1644.65	1.08
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2982.42	3021.38	1.01
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	3085.78	3088.63	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	3105.41	3108.05	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	3320.76	3217.46	0.97
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	817.02	796.15	0.97
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1159.92	1300.94	1.12
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1375.63	1404.67	1.02
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	1800.47	1900.22	1.06
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	2939.02	3105.12	1.06
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	3281.97	3228.94	0.98
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	3454.18	3452.24	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	3530.24	3451.58	0.98
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	763.57	763.83	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	911.56	1099.86	1.21
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1670.72	1718.43	1.03
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2001.65	2210.22	1.10
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	2863.78	3093.30	1.08
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3132.29	3206.29	1.02
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3291.37	3364.74	1.02
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3315.97	3298.56	0.99
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	793.97	791.33	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1004.03	1151.35	1.15
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1675.52	1747.42	1.04
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2017.40	2238.93	1.11
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	2885.41	3124.86	1.08
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3165.52	3230.72	1.02
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3318.80	3397.83	1.02
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3345.00	3350.70	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	764.48	763.50	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	921.33	1062.10	1.15
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1615.68	1823.73	1.13
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	1994.60	2202.84	1.10
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	2875.04	3102.71	1.08
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3158.15	3210.03	1.02
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3306.67	3368.25	1.02
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3332.50	3307.40	0.99
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	747.50	748.80	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	975.37	996.00	1.02
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1530.57	1756.72	1.15
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	1917.39	2164.34	1.13
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	2932.25	3075.15	1.05
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3184.94	3208.60	1.01
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3267.60	3293.91	1.01
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3313.73	3285.84	0.99
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1125.82	1125.22	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1257.80	1265.67	1.01
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	1921.71	1977.32	1.03
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	2287.67	2520.30	1.10
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	3221.45	3514.65	1.09
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	3749.75	3767.24	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	3775.78	3821.07	1.01
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	3807.40	3745.89	0.98
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1123.52	1121.33	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1305.42	1300.30	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	1926.17	1993.75	1.04
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	2267.63	2486.32	1.10
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	3204.51	3476.84	1.08
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	3718.60	3731.83	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	3768.09	3810.04	1.01
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	3827.89	3761.17	0.98
llama 8B Q2_K_M	16	pp2048	604.41	869.56	1.44
llama 8B Q2_K_M	32	pp2048	659.83	694.47	1.05
llama 8B Q2_K_M	64	pp2048	412.39	418.36	1.01
llama 8B Q2_K_M	128	pp2048	549.53	552.55	1.01
llama 8B Q2_K_M	256	pp2048	2734.39	2971.95	1.09
llama 8B Q2_K_M	512	pp2048	2885.21	2867.88	0.99
llama 8B Q2_K_M	1024	pp2048	2982.34	3022.03	1.01
llama 8B Q2_K_M	2048	pp2048	3298.40	3278.32	0.99
llama 8B Q2_K_S	16	pp2048	543.28	908.36	1.67
llama 8B Q2_K_S	32	pp2048	509.33	581.89	1.14
llama 8B Q2_K_S	64	pp2048	270.07	271.58	1.01
llama 8B Q2_K_S	128	pp2048	400.99	403.62	1.01
llama 8B Q2_K_S	256	pp2048	2940.30	3014.71	1.03
llama 8B Q2_K_S	512	pp2048	3062.64	3082.45	1.01
llama 8B Q2_K_S	1024	pp2048	3094.33	3114.79	1.01
llama 8B Q2_K_S	2048	pp2048	3334.42	3341.10	1.00
llama 8B Q3_K_L	16	pp2048	945.24	939.47	0.99
llama 8B Q3_K_L	32	pp2048	1424.38	1417.76	1.00
llama 8B Q3_K_L	64	pp2048	1640.99	1714.70	1.04
llama 8B Q3_K_L	128	pp2048	2000.99	2188.02	1.09
llama 8B Q3_K_L	256	pp2048	2753.49	2959.59	1.07
llama 8B Q3_K_L	512	pp2048	3029.80	3022.81	1.00
llama 8B Q3_K_L	1024	pp2048	3240.41	3291.85	1.02
llama 8B Q3_K_L	2048	pp2048	3341.89	3300.54	0.99
llama 8B Q3_K_M	16	pp2048	950.04	947.57	1.00
llama 8B Q3_K_M	32	pp2048	1436.51	1435.66	1.00
llama 8B Q3_K_M	64	pp2048	1627.22	1714.93	1.05
llama 8B Q3_K_M	128	pp2048	1988.00	2200.86	1.11
llama 8B Q3_K_M	256	pp2048	2761.43	2992.52	1.08
llama 8B Q3_K_M	512	pp2048	3037.18	3042.78	1.00
llama 8B Q3_K_M	1024	pp2048	3240.26	3310.50	1.02
llama 8B Q3_K_M	2048	pp2048	3346.45	3320.20	0.99
llama 8B Q3_K_S	16	pp2048	865.74	863.04	1.00
llama 8B Q3_K_S	32	pp2048	1324.17	1318.81	1.00
llama 8B Q3_K_S	64	pp2048	1532.06	1671.71	1.09
llama 8B Q3_K_S	128	pp2048	1888.49	2067.92	1.10
llama 8B Q3_K_S	256	pp2048	2569.77	2792.83	1.09
llama 8B Q3_K_S	512	pp2048	2849.40	2846.02	1.00
llama 8B Q3_K_S	1024	pp2048	3039.19	3103.66	1.02
llama 8B Q3_K_S	2048	pp2048	3149.15	3111.21	0.99
llama 8B Q4_0	16	pp2048	1069.88	1067.63	1.00
llama 8B Q4_0	32	pp2048	1120.43	1179.60	1.05
llama 8B Q4_0	64	pp2048	1841.14	1914.67	1.04
llama 8B Q4_0	128	pp2048	2243.49	2446.16	1.09
llama 8B Q4_0	256	pp2048	3140.75	3412.69	1.09
llama 8B Q4_0	512	pp2048	3627.08	3599.72	0.99
llama 8B Q4_0	1024	pp2048	3704.16	3781.15	1.02
llama 8B Q4_0	2048	pp2048	3833.02	3785.63	0.99
llama 8B Q4_1	16	pp2048	1053.36	1051.17	1.00
llama 8B Q4_1	32	pp2048	1618.03	1622.50	1.00
llama 8B Q4_1	64	pp2048	1828.27	1929.04	1.06
llama 8B Q4_1	128	pp2048	2214.12	2413.13	1.09
llama 8B Q4_1	256	pp2048	3127.33	3395.32	1.09
llama 8B Q4_1	512	pp2048	3620.70	3565.29	0.98
llama 8B Q4_1	1024	pp2048	3686.29	3742.87	1.02
llama 8B Q4_1	2048	pp2048	3802.52	3770.98	0.99
llama 8B Q4_K_M	16	pp2048	990.62	988.52	1.00
llama 8B Q4_K_M	32	pp2048	1510.59	1502.35	0.99
llama 8B Q4_K_M	64	pp2048	1679.82	1969.99	1.17
llama 8B Q4_K_M	128	pp2048	2033.37	2297.92	1.13
llama 8B Q4_K_M	256	pp2048	3077.06	3267.82	1.06
llama 8B Q4_K_M	512	pp2048	3533.31	3551.24	1.01
llama 8B Q4_K_M	1024	pp2048	3635.87	3684.76	1.01
llama 8B Q4_K_M	2048	pp2048	3696.83	3678.01	0.99
llama 8B Q4_K_S	16	pp2048	1063.59	1061.69	1.00
llama 8B Q4_K_S	32	pp2048	1631.02	1626.95	1.00
llama 8B Q4_K_S	64	pp2048	1786.75	2051.12	1.15
llama 8B Q4_K_S	128	pp2048	2190.71	2404.56	1.10
llama 8B Q4_K_S	256	pp2048	3115.88	3382.54	1.09
llama 8B Q4_K_S	512	pp2048	3558.73	3533.20	0.99
llama 8B Q4_K_S	1024	pp2048	3655.34	3718.21	1.02
llama 8B Q4_K_S	2048	pp2048	3773.07	3739.06	0.99
llama 8B Q5_0	16	pp2048	951.18	949.34	1.00
llama 8B Q5_0	32	pp2048	887.43	1028.35	1.16
llama 8B Q5_0	64	pp2048	1755.91	1962.40	1.12
llama 8B Q5_0	128	pp2048	1731.15	2371.97	1.37
llama 8B Q5_0	256	pp2048	2495.20	3308.49	1.33
llama 8B Q5_0	512	pp2048	2751.90	3498.66	1.27
llama 8B Q5_0	1024	pp2048	2987.16	3665.09	1.23
llama 8B Q5_0	2048	pp2048	3067.53	3644.53	1.19
llama 8B Q5_1	16	pp2048	742.45	743.39	1.00
llama 8B Q5_1	32	pp2048	1253.51	1253.25	1.00
llama 8B Q5_1	64	pp2048	1564.27	1670.20	1.07
llama 8B Q5_1	128	pp2048	1531.77	2243.43	1.46
llama 8B Q5_1	256	pp2048	2321.14	3214.94	1.39
llama 8B Q5_1	512	pp2048	2630.26	3424.52	1.30
llama 8B Q5_1	1024	pp2048	2889.42	3610.11	1.25
llama 8B Q5_1	2048	pp2048	2972.88	3610.76	1.21
llama 8B Q5_K_M	16	pp2048	1008.85	1008.24	1.00
llama 8B Q5_K_M	32	pp2048	1495.13	1491.67	1.00
llama 8B Q5_K_M	64	pp2048	1693.22	1798.67	1.06
llama 8B Q5_K_M	128	pp2048	2034.43	2279.22	1.12
llama 8B Q5_K_M	256	pp2048	3052.55	3212.88	1.05
llama 8B Q5_K_M	512	pp2048	3508.29	3508.94	1.00
llama 8B Q5_K_M	1024	pp2048	3614.16	3639.62	1.01
llama 8B Q5_K_M	2048	pp2048	3679.87	3630.89	0.99
llama 8B Q5_K_S	16	pp2048	1085.95	1087.16	1.00
llama 8B Q5_K_S	32	pp2048	1608.72	1610.66	1.00
llama 8B Q5_K_S	64	pp2048	1813.65	1888.98	1.04
llama 8B Q5_K_S	128	pp2048	2198.38	2382.46	1.08
llama 8B Q5_K_S	256	pp2048	3076.73	3321.10	1.08
llama 8B Q5_K_S	512	pp2048	3503.00	3492.65	1.00
llama 8B Q5_K_S	1024	pp2048	3617.45	3672.03	1.02
llama 8B Q5_K_S	2048	pp2048	3740.63	3693.07	0.99
llama 8B Q6_K	16	pp2048	784.85	784.23	1.00
llama 8B Q6_K	32	pp2048	1132.08	1135.48	1.00
llama 8B Q6_K	64	pp2048	1288.80	1425.51	1.11
llama 8B Q6_K	128	pp2048	1498.22	1592.98	1.06
llama 8B Q6_K	256	pp2048	2883.23	2879.44	1.00
llama 8B Q6_K	512	pp2048	2981.80	2967.52	1.00
llama 8B Q6_K	1024	pp2048	3048.64	3019.92	0.99
llama 8B Q6_K	2048	pp2048	3285.08	3267.11	0.99
llama 8B Q8_0	16	pp2048	990.63	995.44	1.00
llama 8B Q8_0	32	pp2048	1197.01	1200.50	1.00
llama 8B Q8_0	64	pp2048	1876.51	2189.85	1.17
llama 8B Q8_0	128	pp2048	2269.11	2494.85	1.10
llama 8B Q8_0	256	pp2048	3219.55	3502.93	1.09
llama 8B Q8_0	512	pp2048	3800.51	3796.81	1.00
llama 8B Q8_0	1024	pp2048	3862.25	3882.96	1.01
llama 8B Q8_0	2048	pp2048	3989.81	3890.74	0.98

RDNA 4: 9070 XT

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	291.66	291.45	1.00
llama 8B IQ1_M - 1.75 bpw	32	pp2048	545.51	544.38	1.00
llama 8B IQ1_M - 1.75 bpw	64	pp2048	983.05	978.97	1.00
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1569.56	1650.51	1.05
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2673.03	2762.11	1.03
llama 8B IQ1_M - 1.75 bpw	512	pp2048	2473.06	2578.04	1.04
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	2694.51	2816.28	1.05
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	2700.57	2809.20	1.04
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1095.41	1094.96	1.00
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1541.44	1543.40	1.00
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1924.99	1914.19	0.99
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	2416.78	2605.31	1.08
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	3443.75	3578.80	1.04
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	2545.54	2658.92	1.04
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	2618.03	2732.73	1.04
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	2625.90	2720.94	1.04
llama 8B IQ2_M - 2.7 bpw	16	pp2048	707.56	797.45	1.13
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1286.16	1299.42	1.01
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1722.99	1782.57	1.03
llama 8B IQ2_M - 2.7 bpw	128	pp2048	2409.22	2416.60	1.00
llama 8B IQ2_M - 2.7 bpw	256	pp2048	3400.39	3403.72	1.00
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3481.32	3476.82	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3516.38	3518.17	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3465.52	3465.58	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	695.27	785.08	1.13
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1268.71	1281.49	1.01
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1751.53	1772.77	1.01
llama 8B IQ2_S - 2.5 bpw	128	pp2048	2481.62	2484.96	1.00
llama 8B IQ2_S - 2.5 bpw	256	pp2048	3464.57	3466.49	1.00
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3561.38	3560.00	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3559.57	3561.68	1.00
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3503.26	3502.63	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	684.07	780.31	1.14
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1226.38	1224.15	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1489.68	1505.68	1.01
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	2018.94	2148.19	1.06
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2795.43	2877.44	1.03
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	2176.12	2254.44	1.04
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	2246.01	2324.46	1.03
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	2253.70	2320.14	1.03
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	902.82	901.23	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1056.35	1056.74	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1810.54	1793.50	0.99
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	2253.56	2418.61	1.07
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	3222.19	3332.89	1.03
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	2432.88	2529.37	1.04
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	2505.55	2602.07	1.04
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	2514.13	2591.18	1.03
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	877.30	877.68	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	1311.17	1428.47	1.09
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	2060.13	2116.11	1.03
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2803.03	2800.70	1.00
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	4066.42	4071.60	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	4303.71	4303.69	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	4205.22	4206.69	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	4128.19	4129.55	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	889.21	887.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1339.72	1442.45	1.08
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	2088.40	2134.85	1.02
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2823.46	2821.66	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	4111.29	4090.75	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	4352.00	4343.31	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	4253.97	4245.77	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	4178.72	4168.92	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	880.03	880.21	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	1166.99	1216.89	1.04
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	2042.55	2123.39	1.04
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2788.29	2785.89	1.00
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	4074.37	4084.49	1.00
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	4313.62	4316.49	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	4218.56	4221.96	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	4135.76	4137.46	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	835.03	862.92	1.03
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	1116.09	1125.07	1.01
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1959.08	2044.26	1.04
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	2711.71	2709.81	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	3989.99	3988.51	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	4222.47	4221.38	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	4118.60	4116.58	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	4046.38	4043.89	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1102.04	1101.39	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1806.49	1806.60	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2521.27	2523.85	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	3245.60	3244.07	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	4757.77	4763.02	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	5079.21	5086.64	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	4986.73	4995.14	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	4937.20	4945.00	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1157.33	1154.93	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1793.94	1786.36	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2527.08	2522.08	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	3229.31	3244.85	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	4744.97	4737.52	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	5037.30	5028.17	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	4953.47	4944.17	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	4931.30	4917.60	1.00
llama 8B Q2_K_M	16	pp2048	813.20	901.11	1.11
llama 8B Q2_K_M	32	pp2048	878.27	895.10	1.02
llama 8B Q2_K_M	64	pp2048	467.58	528.28	1.13
llama 8B Q2_K_M	128	pp2048	583.46	646.24	1.11
llama 8B Q2_K_M	256	pp2048	659.70	758.97	1.15
llama 8B Q2_K_M	512	pp2048	271.92	301.30	1.11
llama 8B Q2_K_M	1024	pp2048	298.67	330.56	1.11
llama 8B Q2_K_M	2048	pp2048	298.92	335.08	1.12
llama 8B Q2_K_S	16	pp2048	930.89	929.42	1.00
llama 8B Q2_K_S	32	pp2048	688.86	703.67	1.02
llama 8B Q2_K_S	64	pp2048	342.31	371.67	1.09
llama 8B Q2_K_S	128	pp2048	400.63	482.15	1.20
llama 8B Q2_K_S	256	pp2048	463.16	538.87	1.16
llama 8B Q2_K_S	512	pp2048	186.24	206.19	1.11
llama 8B Q2_K_S	1024	pp2048	204.52	226.65	1.11
llama 8B Q2_K_S	2048	pp2048	205.92	229.11	1.11
llama 8B Q3_K_L	16	pp2048	792.05	935.48	1.18
llama 8B Q3_K_L	32	pp2048	1531.75	1529.64	1.00
llama 8B Q3_K_L	64	pp2048	2123.01	2118.01	1.00
llama 8B Q3_K_L	128	pp2048	2781.56	2766.41	0.99
llama 8B Q3_K_L	256	pp2048	3925.60	3923.49	1.00
llama 8B Q3_K_L	512	pp2048	4157.07	4163.49	1.00
llama 8B Q3_K_L	1024	pp2048	4115.09	4115.26	1.00
llama 8B Q3_K_L	2048	pp2048	4129.16	4133.29	1.00
llama 8B Q3_K_M	16	pp2048	794.23	937.92	1.18
llama 8B Q3_K_M	32	pp2048	1529.01	1526.05	1.00
llama 8B Q3_K_M	64	pp2048	2134.45	2130.00	1.00
llama 8B Q3_K_M	128	pp2048	2789.97	2785.76	1.00
llama 8B Q3_K_M	256	pp2048	3949.90	3950.09	1.00
llama 8B Q3_K_M	512	pp2048	4191.07	4187.55	1.00
llama 8B Q3_K_M	1024	pp2048	4141.24	4139.56	1.00
llama 8B Q3_K_M	2048	pp2048	4162.88	4157.62	1.00
llama 8B Q3_K_S	16	pp2048	677.52	871.84	1.29
llama 8B Q3_K_S	32	pp2048	1482.52	1483.86	1.00
llama 8B Q3_K_S	64	pp2048	2027.69	2025.52	1.00
llama 8B Q3_K_S	128	pp2048	2661.85	2646.82	0.99
llama 8B Q3_K_S	256	pp2048	3672.67	3674.28	1.00
llama 8B Q3_K_S	512	pp2048	3889.58	3888.92	1.00
llama 8B Q3_K_S	1024	pp2048	3859.29	3859.14	1.00
llama 8B Q3_K_S	2048	pp2048	3871.58	3872.04	1.00
llama 8B Q4_0	16	pp2048	1123.39	1122.41	1.00
llama 8B Q4_0	32	pp2048	1788.75	1782.91	1.00
llama 8B Q4_0	64	pp2048	2393.76	2377.58	0.99
llama 8B Q4_0	128	pp2048	3285.34	3287.03	1.00
llama 8B Q4_0	256	pp2048	4781.63	4761.13	1.00
llama 8B Q4_0	512	pp2048	5097.95	5067.75	0.99
llama 8B Q4_0	1024	pp2048	5043.90	5023.28	1.00
llama 8B Q4_0	2048	pp2048	5057.47	5034.12	1.00
llama 8B Q4_1	16	pp2048	1035.04	1034.17	1.00
llama 8B Q4_1	32	pp2048	1715.96	1711.45	1.00
llama 8B Q4_1	64	pp2048	2432.61	2424.07	1.00
llama 8B Q4_1	128	pp2048	3145.74	3134.99	1.00
llama 8B Q4_1	256	pp2048	4566.55	4549.99	1.00
llama 8B Q4_1	512	pp2048	4868.57	4839.15	0.99
llama 8B Q4_1	1024	pp2048	4820.76	4802.93	1.00
llama 8B Q4_1	2048	pp2048	4839.23	4818.93	1.00
llama 8B Q4_K_M	16	pp2048	952.56	954.26	1.00
llama 8B Q4_K_M	32	pp2048	1525.61	1526.91	1.00
llama 8B Q4_K_M	64	pp2048	2182.61	2183.22	1.00
llama 8B Q4_K_M	128	pp2048	2813.89	2812.25	1.00
llama 8B Q4_K_M	256	pp2048	4023.16	4028.04	1.00
llama 8B Q4_K_M	512	pp2048	4263.82	4266.84	1.00
llama 8B Q4_K_M	1024	pp2048	4290.16	4294.64	1.00
llama 8B Q4_K_M	2048	pp2048	4318.51	4318.08	1.00
llama 8B Q4_K_S	16	pp2048	991.84	992.46	1.00
llama 8B Q4_K_S	32	pp2048	1607.65	1611.44	1.00
llama 8B Q4_K_S	64	pp2048	2351.49	2353.80	1.00
llama 8B Q4_K_S	128	pp2048	3074.25	3089.20	1.00
llama 8B Q4_K_S	256	pp2048	4490.33	4494.39	1.00
llama 8B Q4_K_S	512	pp2048	4814.64	4813.87	1.00
llama 8B Q4_K_S	1024	pp2048	4766.47	4769.80	1.00
llama 8B Q4_K_S	2048	pp2048	4799.31	4798.99	1.00
llama 8B Q5_0	16	pp2048	1004.49	1004.06	1.00
llama 8B Q5_0	32	pp2048	1594.15	1596.24	1.00
llama 8B Q5_0	64	pp2048	2253.79	2259.26	1.00
llama 8B Q5_0	128	pp2048	2195.01	3007.98	1.37
llama 8B Q5_0	256	pp2048	3409.90	4346.50	1.27
llama 8B Q5_0	512	pp2048	3650.48	4648.96	1.27
llama 8B Q5_0	1024	pp2048	3655.50	4623.19	1.26
llama 8B Q5_0	2048	pp2048	3648.41	4632.95	1.27
llama 8B Q5_1	16	pp2048	770.91	770.37	1.00
llama 8B Q5_1	32	pp2048	1342.46	1342.39	1.00
llama 8B Q5_1	64	pp2048	2107.23	2106.30	1.00
llama 8B Q5_1	128	pp2048	2065.56	2829.85	1.37
llama 8B Q5_1	256	pp2048	3275.62	4174.64	1.27
llama 8B Q5_1	512	pp2048	3523.32	4534.53	1.29
llama 8B Q5_1	1024	pp2048	3563.57	4531.35	1.27
llama 8B Q5_1	2048	pp2048	3577.54	4550.52	1.27
llama 8B Q5_K_M	16	pp2048	959.68	961.11	1.00
llama 8B Q5_K_M	32	pp2048	1542.65	1543.40	1.00
llama 8B Q5_K_M	64	pp2048	2146.96	2146.43	1.00
llama 8B Q5_K_M	128	pp2048	2788.34	2773.03	0.99
llama 8B Q5_K_M	256	pp2048	3963.27	3972.45	1.00
llama 8B Q5_K_M	512	pp2048	4190.81	4196.70	1.00
llama 8B Q5_K_M	1024	pp2048	4215.09	4220.84	1.00
llama 8B Q5_K_M	2048	pp2048	4236.22	4240.20	1.00
llama 8B Q5_K_S	16	pp2048	997.55	997.35	1.00
llama 8B Q5_K_S	32	pp2048	1624.68	1624.54	1.00
llama 8B Q5_K_S	64	pp2048	2304.18	2301.41	1.00
llama 8B Q5_K_S	128	pp2048	3044.17	3037.50	1.00
llama 8B Q5_K_S	256	pp2048	4409.68	4413.85	1.00
llama 8B Q5_K_S	512	pp2048	4691.42	4698.29	1.00
llama 8B Q5_K_S	1024	pp2048	4658.90	4664.34	1.00
llama 8B Q5_K_S	2048	pp2048	4676.86	4681.23	1.00
llama 8B Q6_K	16	pp2048	828.61	828.95	1.00
llama 8B Q6_K	32	pp2048	1233.56	1234.44	1.00
llama 8B Q6_K	64	pp2048	1573.46	1573.67	1.00
llama 8B Q6_K	128	pp2048	1888.45	1889.26	1.00
llama 8B Q6_K	256	pp2048	2505.31	2506.72	1.00
llama 8B Q6_K	512	pp2048	2536.10	2537.40	1.00
llama 8B Q6_K	1024	pp2048	2647.83	2649.15	1.00
llama 8B Q6_K	2048	pp2048	2647.25	2647.14	1.00
llama 8B Q8_0	16	pp2048	903.07	902.91	1.00
llama 8B Q8_0	32	pp2048	1517.76	1516.95	1.00
llama 8B Q8_0	64	pp2048	2339.34	2327.98	1.00
llama 8B Q8_0	128	pp2048	3050.58	3034.93	0.99
llama 8B Q8_0	256	pp2048	4562.07	4527.00	0.99
llama 8B Q8_0	512	pp2048	4934.37	4891.40	0.99
llama 8B Q8_0	1024	pp2048	4924.12	4889.63	0.99
llama 8B Q8_0	2048	pp2048	4976.32	4941.78	0.99

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, to make the sweep script

am17an · 2026-07-29T15:57:44Z

I now see that you're sweeping over llama 8b, which is a dense model. Instead you should sweep over a MoE model as well, which is a more representative use-case

ggml-gh-bot · 2026-07-29T15:59:23Z

Hi @itterative, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

itterative · 2026-07-29T16:05:55Z

@am17an I had a few shapes in the sweep scripts, but indeed these were taken from llama 8B (ffn-up, ffn-down, attn, etc). I can do a different set of sweeps with the moe shapes, it will take a few hours to get new results in. Problem will be I can't really run the moe on either card at Q8 for llama-bench. I will try to get qwen and gemma shapes in the sweep.

pwilkin · 2026-07-29T16:38:00Z

@itterative you can try LFM2.5-8B-A1B for a small MoE model.

Geramy · 2026-07-29T19:23:54Z

@itterative I would also suggest a gemma4 moe model and a qwen3.6 moe model too, those are pretty popular if you guys dont mind my suggestions of course.

itterative · 2026-07-30T16:20:28Z

@am17an Went into a bit of a rabbit hole with this tuning for moe. In short, RDNA 4 doesn't seem like it needs a different table, however RDNA 3 is all over the place, seems like optimizing for moe hurts dense a fair bit (at least for ub < 256 in any case, ub > 1024 barely moves). In any case, I'll redo the llama-bench results with lfm2.5, qwen, and gemma.

Regarding what I found, I tried changing selecting a different J (sram tile width) when using a moe model, and that seems to make a difference. However, even with this, it seems like some moe wants different tuning compared to dense. I'll try a more conservative tuning before proposing any other changes outside of mmq-config-rdna3.cuh and mma-config-rdna4.cuh.

am17an · 2026-07-30T16:25:13Z

Is it possible to different tables for MoE vs Dense? Because AFAIK they are quite different workloads

itterative · 2026-07-30T16:26:49Z

It would be better to have a separate table for each imo, but I didn't want to increase the scope of the pr too much. I can try to do that if that's alright.

Geramy · 2026-07-30T16:31:42Z

Is it possible to different tables for MoE vs Dense? Because AFAIK they are quite different workloads

There are huge speed increases when tuning for MoE over Dense which I noticed myself. I think using stream-k I got over 20% increase for Dense. I do believe there would be a bit of work involved in picking different configs for MMQ based on MoE or Dense, but I believe its worth the added complexity, which shouldn't be much.

itterative · 2026-07-30T20:37:11Z

Updated RDNA 3 with more conservative changes for dense so the moe don't have any regressions. I'll rebase since Q2_0 was added.

RDNA 3 llama 3 8B

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	366.55	370.80	1.01
llama 8B IQ1_M - 1.75 bpw	32	pp2048	660.14	666.10	1.01
llama 8B IQ1_M - 1.75 bpw	64	pp2048	946.76	1021.02	1.08
llama 8B IQ1_M - 1.75 bpw	96	pp2048	1302.17	1319.61	1.01
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1583.27	1605.49	1.01
llama 8B IQ1_M - 1.75 bpw	192	pp2048	2364.34	2514.69	1.06
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2983.67	3062.45	1.03
llama 8B IQ1_M - 1.75 bpw	512	pp2048	3107.76	3096.58	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	3149.59	3157.91	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	3365.38	3354.69	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1015.73	1034.84	1.02
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1430.57	1458.59	1.02
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1424.72	1580.59	1.11
llama 8B IQ1_S - 1.5625 bpw	96	pp2048	1725.97	1899.31	1.10
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	1879.20	2023.26	1.08
llama 8B IQ1_S - 1.5625 bpw	192	pp2048	2734.93	3022.45	1.11
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	3102.26	3378.30	1.09
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	3538.19	3571.06	1.01
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	3641.48	3705.19	1.02
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	3722.02	3703.64	1.00
llama 8B IQ2_M - 2.7 bpw	16	pp2048	700.98	689.09	0.98
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1118.17	1122.84	1.00
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1449.80	1480.35	1.02
llama 8B IQ2_M - 2.7 bpw	96	pp2048	1518.14	1756.94	1.16
llama 8B IQ2_M - 2.7 bpw	128	pp2048	1724.92	1899.52	1.10
llama 8B IQ2_M - 2.7 bpw	192	pp2048	2394.28	2476.00	1.03
llama 8B IQ2_M - 2.7 bpw	256	pp2048	2901.98	3000.93	1.03
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3023.29	3022.39	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3047.93	3050.61	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3226.99	3243.20	1.01
llama 8B IQ2_S - 2.5 bpw	16	pp2048	690.53	690.09	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1108.31	1114.23	1.01
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1384.48	1509.87	1.09
llama 8B IQ2_S - 2.5 bpw	96	pp2048	1518.25	1782.64	1.17
llama 8B IQ2_S - 2.5 bpw	128	pp2048	1744.17	1907.96	1.09
llama 8B IQ2_S - 2.5 bpw	192	pp2048	2412.21	2496.47	1.03
llama 8B IQ2_S - 2.5 bpw	256	pp2048	2932.51	3037.69	1.04
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3032.29	3036.89	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3060.31	3091.03	1.01
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3250.27	3245.08	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	683.87	692.77	1.01
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1110.10	1105.78	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1160.81	1236.84	1.07
llama 8B IQ2_XS - 2.3125 bpw	96	pp2048	1406.25	1604.28	1.14
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	1548.83	1651.91	1.07
llama 8B IQ2_XS - 2.3125 bpw	192	pp2048	2457.08	2507.11	1.02
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2979.31	3014.82	1.01
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	3092.05	3094.75	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	3103.55	3110.87	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	3336.97	3336.28	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	819.98	836.88	1.02
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1165.13	1261.87	1.08
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1385.70	1484.58	1.07
llama 8B IQ2_XXS - 2.0625 bpw	96	pp2048	1638.34	1735.29	1.06
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	1812.73	1910.17	1.05
llama 8B IQ2_XXS - 2.0625 bpw	192	pp2048	2558.02	2711.08	1.06
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	2953.63	3115.33	1.05
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	3309.29	3273.18	0.99
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	3461.49	3467.46	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	3556.86	3479.15	0.98
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	768.32	768.21	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	917.05	936.85	1.02
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1679.02	1866.02	1.11
llama 8B IQ3_S - 3.4375 bpw	96	pp2048	1743.30	1903.62	1.09
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2016.88	2254.22	1.12
llama 8B IQ3_S - 3.4375 bpw	192	pp2048	2508.96	2709.14	1.08
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	2887.82	3126.67	1.08
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3177.61	3251.72	1.02
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3322.07	3395.36	1.02
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3344.74	3335.92	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	798.08	798.47	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1009.20	1034.57	1.03
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1697.99	1882.64	1.11
llama 8B IQ3_S mix - 3.66 bpw	96	pp2048	1775.22	1935.90	1.09
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2047.67	2269.32	1.11
llama 8B IQ3_S mix - 3.66 bpw	192	pp2048	2531.21	2751.99	1.09
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	2897.92	3144.86	1.09
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3183.66	3260.87	1.02
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3340.08	3419.90	1.02
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3366.20	3359.09	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	767.20	768.12	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	929.68	928.90	1.00
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1620.39	1848.59	1.14
llama 8B IQ3_XS - 3.3 bpw	96	pp2048	1735.27	1900.06	1.09
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2008.51	2230.14	1.11
llama 8B IQ3_XS - 3.3 bpw	192	pp2048	2506.52	2722.83	1.09
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	2871.44	3109.80	1.08
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3167.51	3238.89	1.02
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3319.72	3387.77	1.02
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3338.61	3334.46	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	745.96	745.80	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	974.85	976.30	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1533.34	1716.81	1.12
llama 8B IQ3_XXS - 3.0625 bpw	96	pp2048	1664.36	1859.05	1.12
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	1926.25	2125.04	1.10
llama 8B IQ3_XXS - 3.0625 bpw	192	pp2048	2540.80	2700.63	1.06
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	2945.27	3089.86	1.05
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3205.60	3229.05	1.01
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3285.11	3302.53	1.01
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3351.09	3319.03	0.99
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1125.83	1127.76	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1257.51	1257.73	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	1928.49	2084.21	1.08
llama 8B IQ4_NL - 4.5 bpw	96	pp2048	2034.13	2242.93	1.10
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	2303.72	2528.24	1.10
llama 8B IQ4_NL - 4.5 bpw	192	pp2048	2895.87	3176.48	1.10
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	3237.87	3525.92	1.09
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	3760.74	3781.55	1.01
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	3790.34	3850.92	1.02
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	3847.13	3800.26	0.99
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1120.99	1124.04	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1301.15	1306.86	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	1924.06	2158.46	1.12
llama 8B IQ4_XS - 4.25 bpw	96	pp2048	2048.78	2250.61	1.10
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	2269.64	2490.23	1.10
llama 8B IQ4_XS - 4.25 bpw	192	pp2048	2896.33	3157.30	1.09
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	3213.17	3481.84	1.08
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	3725.25	3740.38	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	3772.98	3828.58	1.01
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	3862.36	3803.17	0.98
llama 8B Q2_K_M	16	pp2048	602.83	870.66	1.44
llama 8B Q2_K_M	32	pp2048	657.52	658.20	1.00
llama 8B Q2_K_M	64	pp2048	413.25	416.44	1.01
llama 8B Q2_K_M	96	pp2048	609.73	559.19	0.92
llama 8B Q2_K_M	128	pp2048	552.58	557.95	1.01
llama 8B Q2_K_M	192	pp2048	1932.41	2056.06	1.06
llama 8B Q2_K_M	256	pp2048	2725.82	2973.39	1.09
llama 8B Q2_K_M	512	pp2048	2888.01	2876.20	1.00
llama 8B Q2_K_M	1024	pp2048	2976.04	3029.74	1.02
llama 8B Q2_K_M	2048	pp2048	3314.77	3294.80	0.99
llama 8B Q2_K_S	16	pp2048	542.06	907.51	1.67
llama 8B Q2_K_S	32	pp2048	510.11	515.23	1.01
llama 8B Q2_K_S	64	pp2048	269.47	271.30	1.01
llama 8B Q2_K_S	96	pp2048	455.57	427.02	0.94
llama 8B Q2_K_S	128	pp2048	400.08	402.40	1.01
llama 8B Q2_K_S	192	pp2048	1896.81	1928.75	1.02
llama 8B Q2_K_S	256	pp2048	2939.43	3002.85	1.02
llama 8B Q2_K_S	512	pp2048	3062.73	3073.97	1.00
llama 8B Q2_K_S	1024	pp2048	3095.74	3110.94	1.00
llama 8B Q2_K_S	2048	pp2048	3342.07	3363.34	1.01
llama 8B Q3_K_L	16	pp2048	946.55	947.32	1.00
llama 8B Q3_K_L	32	pp2048	1434.84	1434.24	1.00
llama 8B Q3_K_L	64	pp2048	1652.92	1841.84	1.11
llama 8B Q3_K_L	96	pp2048	1793.84	1970.55	1.10
llama 8B Q3_K_L	128	pp2048	2024.72	2215.21	1.09
llama 8B Q3_K_L	192	pp2048	2489.38	2686.01	1.08
llama 8B Q3_K_L	256	pp2048	2750.24	2978.67	1.08
llama 8B Q3_K_L	512	pp2048	3027.22	3034.60	1.00
llama 8B Q3_K_L	1024	pp2048	3231.06	3300.00	1.02
llama 8B Q3_K_L	2048	pp2048	3360.21	3334.07	0.99
llama 8B Q3_K_M	16	pp2048	952.03	949.99	1.00
llama 8B Q3_K_M	32	pp2048	1449.34	1440.38	0.99
llama 8B Q3_K_M	64	pp2048	1633.93	1844.72	1.13
llama 8B Q3_K_M	96	pp2048	1792.70	1972.42	1.10
llama 8B Q3_K_M	128	pp2048	2014.67	2218.52	1.10
llama 8B Q3_K_M	192	pp2048	2503.03	2714.53	1.08
llama 8B Q3_K_M	256	pp2048	2773.22	3004.25	1.08
llama 8B Q3_K_M	512	pp2048	3052.11	3051.93	1.00
llama 8B Q3_K_M	1024	pp2048	3253.66	3315.02	1.02
llama 8B Q3_K_M	2048	pp2048	3375.88	3351.05	0.99
llama 8B Q3_K_S	16	pp2048	868.40	868.63	1.00
llama 8B Q3_K_S	32	pp2048	1336.94	1330.53	1.00
llama 8B Q3_K_S	64	pp2048	1547.43	1720.01	1.11
llama 8B Q3_K_S	96	pp2048	1696.92	1852.33	1.09
llama 8B Q3_K_S	128	pp2048	1911.27	2096.39	1.10
llama 8B Q3_K_S	192	pp2048	2337.94	2545.06	1.09
llama 8B Q3_K_S	256	pp2048	2578.04	2811.43	1.09
llama 8B Q3_K_S	512	pp2048	2856.92	2865.56	1.00
llama 8B Q3_K_S	1024	pp2048	3046.79	3115.58	1.02
llama 8B Q3_K_S	2048	pp2048	3167.41	3139.60	0.99
llama 8B Q4_0	16	pp2048	1090.33	1073.96	0.98
llama 8B Q4_0	32	pp2048	1141.48	1124.00	0.98
llama 8B Q4_0	64	pp2048	1902.98	2092.84	1.10
llama 8B Q4_0	96	pp2048	2042.11	2211.37	1.08
llama 8B Q4_0	128	pp2048	2294.88	2504.88	1.09
llama 8B Q4_0	192	pp2048	2873.97	3104.14	1.08
llama 8B Q4_0	256	pp2048	3189.79	3433.06	1.08
llama 8B Q4_0	512	pp2048	3692.29	3616.73	0.98
llama 8B Q4_0	1024	pp2048	3733.19	3794.07	1.02
llama 8B Q4_0	2048	pp2048	3879.62	3831.65	0.99
llama 8B Q4_1	16	pp2048	1058.04	1047.98	0.99
llama 8B Q4_1	32	pp2048	1635.13	1621.37	0.99
llama 8B Q4_1	64	pp2048	1848.97	2052.89	1.11
llama 8B Q4_1	96	pp2048	1974.34	2180.36	1.10
llama 8B Q4_1	128	pp2048	2219.43	2414.00	1.09
llama 8B Q4_1	192	pp2048	2810.97	3096.06	1.10
llama 8B Q4_1	256	pp2048	3137.05	3411.68	1.09
llama 8B Q4_1	512	pp2048	3644.70	3586.37	0.98
llama 8B Q4_1	1024	pp2048	3700.67	3755.78	1.01
llama 8B Q4_1	2048	pp2048	3835.46	3802.19	0.99
llama 8B Q4_K_M	16	pp2048	992.04	990.52	1.00
llama 8B Q4_K_M	32	pp2048	1509.06	1511.63	1.00
llama 8B Q4_K_M	64	pp2048	1680.22	1904.06	1.13
llama 8B Q4_K_M	96	pp2048	1818.36	2017.71	1.11
llama 8B Q4_K_M	128	pp2048	2036.78	2241.84	1.10
llama 8B Q4_K_M	192	pp2048	2758.30	2967.82	1.08
llama 8B Q4_K_M	256	pp2048	3087.19	3275.89	1.06
llama 8B Q4_K_M	512	pp2048	3549.21	3556.50	1.00
llama 8B Q4_K_M	1024	pp2048	3649.37	3687.24	1.01
llama 8B Q4_K_M	2048	pp2048	3719.16	3698.06	0.99
llama 8B Q4_K_S	16	pp2048	1064.47	1064.74	1.00
llama 8B Q4_K_S	32	pp2048	1630.33	1627.35	1.00
llama 8B Q4_K_S	64	pp2048	1797.06	2062.86	1.15
llama 8B Q4_K_S	96	pp2048	1947.61	2165.86	1.11
llama 8B Q4_K_S	128	pp2048	2197.66	2406.41	1.09
llama 8B Q4_K_S	192	pp2048	2790.99	3082.03	1.10
llama 8B Q4_K_S	256	pp2048	3119.33	3395.59	1.09
llama 8B Q4_K_S	512	pp2048	3570.59	3559.00	1.00
llama 8B Q4_K_S	1024	pp2048	3666.73	3724.65	1.02
llama 8B Q4_K_S	2048	pp2048	3799.70	3776.83	0.99
llama 8B Q5_0	16	pp2048	957.09	954.84	1.00
llama 8B Q5_0	32	pp2048	892.10	908.45	1.02
llama 8B Q5_0	64	pp2048	1788.05	1985.31	1.11
llama 8B Q5_0	96	pp2048	1938.24	2146.92	1.11
llama 8B Q5_0	128	pp2048	1764.90	2417.54	1.37
llama 8B Q5_0	192	pp2048	2690.33	3007.78	1.12
llama 8B Q5_0	256	pp2048	2515.20	3315.20	1.32
llama 8B Q5_0	512	pp2048	2773.64	3499.17	1.26
llama 8B Q5_0	1024	pp2048	3001.75	3673.26	1.22
llama 8B Q5_0	2048	pp2048	3094.40	3675.90	1.19
llama 8B Q5_1	16	pp2048	740.64	741.35	1.00
llama 8B Q5_1	32	pp2048	1250.87	1249.98	1.00
llama 8B Q5_1	64	pp2048	1565.64	1784.76	1.14
llama 8B Q5_1	96	pp2048	1736.24	1985.36	1.14
llama 8B Q5_1	128	pp2048	1506.24	2246.41	1.49
llama 8B Q5_1	192	pp2048	2499.15	2876.76	1.15
llama 8B Q5_1	256	pp2048	2332.19	3214.48	1.38
llama 8B Q5_1	512	pp2048	2645.65	3422.31	1.29
llama 8B Q5_1	1024	pp2048	2901.80	3606.39	1.24
llama 8B Q5_1	2048	pp2048	2992.32	3630.84	1.21
llama 8B Q5_K_M	16	pp2048	1010.35	1012.35	1.00
llama 8B Q5_K_M	32	pp2048	1495.01	1493.39	1.00
llama 8B Q5_K_M	64	pp2048	1698.80	1892.02	1.11
llama 8B Q5_K_M	96	pp2048	1813.06	1994.87	1.10
llama 8B Q5_K_M	128	pp2048	2043.86	2215.03	1.08
llama 8B Q5_K_M	192	pp2048	2710.70	2908.23	1.07
llama 8B Q5_K_M	256	pp2048	3056.98	3215.87	1.05
llama 8B Q5_K_M	512	pp2048	3517.05	3509.86	1.00
llama 8B Q5_K_M	1024	pp2048	3620.91	3642.41	1.01
llama 8B Q5_K_M	2048	pp2048	3699.70	3656.78	0.99
llama 8B Q5_K_S	16	pp2048	1089.95	1088.44	1.00
llama 8B Q5_K_S	32	pp2048	1585.36	1611.90	1.02
llama 8B Q5_K_S	64	pp2048	1830.18	2045.51	1.12
llama 8B Q5_K_S	96	pp2048	1959.70	2137.74	1.09
llama 8B Q5_K_S	128	pp2048	2206.73	2388.18	1.08
llama 8B Q5_K_S	192	pp2048	2747.23	3003.52	1.09
llama 8B Q5_K_S	256	pp2048	3098.51	3323.98	1.07
llama 8B Q5_K_S	512	pp2048	3537.20	3494.73	0.99
llama 8B Q5_K_S	1024	pp2048	3644.40	3675.39	1.01
llama 8B Q5_K_S	2048	pp2048	3783.31	3724.25	0.98
llama 8B Q6_K	16	pp2048	785.48	788.43	1.00
llama 8B Q6_K	32	pp2048	1131.36	1138.57	1.01
llama 8B Q6_K	64	pp2048	1300.39	1438.03	1.11
llama 8B Q6_K	96	pp2048	1364.92	1524.18	1.12
llama 8B Q6_K	128	pp2048	1515.79	1681.97	1.11
llama 8B Q6_K	192	pp2048	2356.77	2362.51	1.00
llama 8B Q6_K	256	pp2048	2894.72	2888.88	1.00
llama 8B Q6_K	512	pp2048	2993.65	2987.82	1.00
llama 8B Q6_K	1024	pp2048	3052.20	3034.05	0.99
llama 8B Q6_K	2048	pp2048	3303.47	3298.33	1.00
llama 8B Q8_0	16	pp2048	989.33	990.37	1.00
llama 8B Q8_0	32	pp2048	1193.25	1193.66	1.00
llama 8B Q8_0	64	pp2048	1867.38	2183.08	1.17
llama 8B Q8_0	96	pp2048	1981.47	2278.43	1.15
llama 8B Q8_0	128	pp2048	2264.13	2481.55	1.10
llama 8B Q8_0	192	pp2048	2898.87	3204.27	1.11
llama 8B Q8_0	256	pp2048	3213.13	3499.06	1.09
llama 8B Q8_0	512	pp2048	3798.56	3783.07	1.00
llama 8B Q8_0	1024	pp2048	3864.57	3879.21	1.00
llama 8B Q8_0	2048	pp2048	4006.21	3908.64	0.98

RDNA 3 LFM2.5 8B A1B

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	16	pp2048	1981.45	1981.51	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	32	pp2048	2191.29	2189.36	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	64	pp2048	3476.62	3667.44	1.05
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	96	pp2048	3656.27	3743.61	1.02
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	128	pp2048	3956.31	4049.04	1.02
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	192	pp2048	5858.97	5913.97	1.01
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	256	pp2048	6508.82	6564.53	1.01
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	512	pp2048	9587.32	9753.46	1.02
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	1024	pp2048	12469.73	12365.50	0.99
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	2048	pp2048	13547.15	13481.66	1.00
lfm2moe 8B.A1B Q2_K_M	16	pp2048	1174.29	1617.10	1.38
lfm2moe 8B.A1B Q2_K_M	32	pp2048	1176.26	1194.04	1.02
lfm2moe 8B.A1B Q2_K_M	64	pp2048	657.94	659.43	1.00
lfm2moe 8B.A1B Q2_K_M	96	pp2048	1196.97	1050.23	0.88
lfm2moe 8B.A1B Q2_K_M	128	pp2048	1063.99	1070.16	1.01
lfm2moe 8B.A1B Q2_K_M	192	pp2048	1480.81	1816.89	1.23
lfm2moe 8B.A1B Q2_K_M	256	pp2048	1767.45	2221.04	1.26
lfm2moe 8B.A1B Q2_K_M	512	pp2048	2639.05	3435.77	1.30
lfm2moe 8B.A1B Q2_K_M	1024	pp2048	4445.18	5431.35	1.22
lfm2moe 8B.A1B Q2_K_M	2048	pp2048	6281.91	7185.62	1.14
lfm2moe 8B.A1B Q3_K_M	16	pp2048	1711.00	1717.45	1.00
lfm2moe 8B.A1B Q3_K_M	32	pp2048	2506.32	2511.03	1.00
lfm2moe 8B.A1B Q3_K_M	64	pp2048	2932.39	3124.52	1.07
lfm2moe 8B.A1B Q3_K_M	96	pp2048	3102.42	3214.56	1.04
lfm2moe 8B.A1B Q3_K_M	128	pp2048	3349.99	3477.02	1.04
lfm2moe 8B.A1B Q3_K_M	192	pp2048	4964.29	5043.91	1.02
lfm2moe 8B.A1B Q3_K_M	256	pp2048	5511.77	5610.31	1.02
lfm2moe 8B.A1B Q3_K_M	512	pp2048	8076.01	8336.15	1.03
lfm2moe 8B.A1B Q3_K_M	1024	pp2048	10524.35	10599.12	1.01
lfm2moe 8B.A1B Q3_K_M	2048	pp2048	11604.96	11729.55	1.01
lfm2moe 8B.A1B Q4_0	16	pp2048	1941.10	1932.63	1.00
lfm2moe 8B.A1B Q4_0	32	pp2048	2062.45	2052.06	0.99
lfm2moe 8B.A1B Q4_0	64	pp2048	3310.54	3518.87	1.06
lfm2moe 8B.A1B Q4_0	96	pp2048	3515.14	3622.06	1.03
lfm2moe 8B.A1B Q4_0	128	pp2048	3832.03	3964.73	1.03
lfm2moe 8B.A1B Q4_0	192	pp2048	5677.96	5723.24	1.01
lfm2moe 8B.A1B Q4_0	256	pp2048	6330.26	6404.70	1.01
lfm2moe 8B.A1B Q4_0	512	pp2048	9351.47	9539.72	1.02
lfm2moe 8B.A1B Q4_0	1024	pp2048	12189.44	12111.30	0.99
lfm2moe 8B.A1B Q4_0	2048	pp2048	13210.48	13239.88	1.00
lfm2moe 8B.A1B Q4_K_M	16	pp2048	1846.09	1850.73	1.00
lfm2moe 8B.A1B Q4_K_M	32	pp2048	2710.50	2707.30	1.00
lfm2moe 8B.A1B Q4_K_M	64	pp2048	3100.16	3351.77	1.08
lfm2moe 8B.A1B Q4_K_M	96	pp2048	3273.47	3430.58	1.05
lfm2moe 8B.A1B Q4_K_M	128	pp2048	3558.20	3675.86	1.03
lfm2moe 8B.A1B Q4_K_M	192	pp2048	5084.53	5246.72	1.03
lfm2moe 8B.A1B Q4_K_M	256	pp2048	5801.31	5952.07	1.03
lfm2moe 8B.A1B Q4_K_M	512	pp2048	8473.27	8711.47	1.03
lfm2moe 8B.A1B Q4_K_M	1024	pp2048	11055.43	11152.26	1.01
lfm2moe 8B.A1B Q4_K_M	2048	pp2048	11613.20	11787.95	1.02
lfm2moe 8B.A1B Q5_0	16	pp2048	1717.35	1720.17	1.00
lfm2moe 8B.A1B Q5_0	32	pp2048	1619.36	1663.90	1.03
lfm2moe 8B.A1B Q5_0	64	pp2048	3084.96	3308.76	1.07
lfm2moe 8B.A1B Q5_0	96	pp2048	3310.06	3439.40	1.04
lfm2moe 8B.A1B Q5_0	128	pp2048	2907.25	3850.38	1.32
lfm2moe 8B.A1B Q5_0	192	pp2048	5268.35	5576.09	1.06
lfm2moe 8B.A1B Q5_0	256	pp2048	4865.02	6210.94	1.28
lfm2moe 8B.A1B Q5_0	512	pp2048	7308.60	9255.64	1.27
lfm2moe 8B.A1B Q5_0	1024	pp2048	9611.42	11800.37	1.23
lfm2moe 8B.A1B Q5_0	2048	pp2048	11077.68	13047.09	1.18
lfm2moe 8B.A1B Q5_1	16	pp2048	1622.48	1613.38	0.99
lfm2moe 8B.A1B Q5_1	32	pp2048	2571.21	2552.13	0.99
lfm2moe 8B.A1B Q5_1	64	pp2048	3092.86	3318.64	1.07
lfm2moe 8B.A1B Q5_1	96	pp2048	3285.75	3446.54	1.05
lfm2moe 8B.A1B Q5_1	128	pp2048	2779.39	3763.90	1.35
lfm2moe 8B.A1B Q5_1	192	pp2048	5140.47	5473.37	1.06
lfm2moe 8B.A1B Q5_1	256	pp2048	4631.13	6160.68	1.33
lfm2moe 8B.A1B Q5_1	512	pp2048	6895.19	9120.67	1.32
lfm2moe 8B.A1B Q5_1	1024	pp2048	9269.63	11734.48	1.27
lfm2moe 8B.A1B Q5_1	2048	pp2048	10771.27	12901.81	1.20
lfm2moe 8B.A1B Q5_K_M	16	pp2048	1802.62	1811.85	1.01
lfm2moe 8B.A1B Q5_K_M	32	pp2048	2635.13	2622.22	1.00
lfm2moe 8B.A1B Q5_K_M	64	pp2048	3012.61	3235.67	1.07
lfm2moe 8B.A1B Q5_K_M	96	pp2048	3152.04	3308.81	1.05
lfm2moe 8B.A1B Q5_K_M	128	pp2048	3440.79	3582.25	1.04
lfm2moe 8B.A1B Q5_K_M	192	pp2048	4906.12	5123.44	1.04
lfm2moe 8B.A1B Q5_K_M	256	pp2048	5651.60	5842.86	1.03
lfm2moe 8B.A1B Q5_K_M	512	pp2048	8274.64	8518.67	1.03
lfm2moe 8B.A1B Q5_K_M	1024	pp2048	10839.56	10952.10	1.01
lfm2moe 8B.A1B Q5_K_M	2048	pp2048	11552.05	11675.98	1.01
lfm2moe 8B.A1B Q6_K	16	pp2048	1469.02	1470.51	1.00
lfm2moe 8B.A1B Q6_K	32	pp2048	1957.79	1961.28	1.00
lfm2moe 8B.A1B Q6_K	64	pp2048	2200.38	2324.40	1.06
lfm2moe 8B.A1B Q6_K	96	pp2048	2242.00	2387.17	1.06
lfm2moe 8B.A1B Q6_K	128	pp2048	2398.14	2518.76	1.05
lfm2moe 8B.A1B Q6_K	192	pp2048	3023.33	3242.22	1.07
lfm2moe 8B.A1B Q6_K	256	pp2048	3703.19	4082.27	1.10
lfm2moe 8B.A1B Q6_K	512	pp2048	5164.26	5703.97	1.10
lfm2moe 8B.A1B Q6_K	1024	pp2048	6704.06	7624.92	1.14
lfm2moe 8B.A1B Q6_K	2048	pp2048	8410.00	8732.03	1.04
lfm2moe 8B.A1B Q8_0	16	pp2048	1623.37	1617.51	1.00
lfm2moe 8B.A1B Q8_0	32	pp2048	1899.79	1906.52	1.00
lfm2moe 8B.A1B Q8_0	64	pp2048	3230.40	3590.93	1.11
lfm2moe 8B.A1B Q8_0	96	pp2048	3450.94	3700.26	1.07
lfm2moe 8B.A1B Q8_0	128	pp2048	3832.50	3984.62	1.04
lfm2moe 8B.A1B Q8_0	192	pp2048	5611.72	5907.56	1.05
lfm2moe 8B.A1B Q8_0	256	pp2048	6424.47	6482.73	1.01
lfm2moe 8B.A1B Q8_0	512	pp2048	9540.38	9639.94	1.01
lfm2moe 8B.A1B Q8_0	1024	pp2048	12603.54	12406.24	0.98
lfm2moe 8B.A1B Q8_0	2048	pp2048	14139.00	14002.73	0.99

RDNA 3 Gemma 4

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
gemma4 12B Q4_0	8	pp4096	371.91	372.35	1.00
gemma4 12B Q4_0	16	pp4096	601.40	599.34	1.00
gemma4 12B Q4_0	32	pp4096	628.51	627.76	1.00
gemma4 12B Q4_0	64	pp4096	1037.84	1130.84	1.09
gemma4 12B Q4_0	512	pp4096	1939.74	2040.37	1.05
gemma4 12B Q4_0	1024	pp4096	2045.12	2023.07	0.99
gemma4 12B Q4_0	2048	pp4096	2029.90	2008.18	0.99
gemma4 26B.A4B Q4_0	8	pp4096	563.55	564.40	1.00
gemma4 26B.A4B Q4_0	16	pp4096	749.91	749.88	1.00
gemma4 26B.A4B Q4_0	32	pp4096	763.14	763.39	1.00
gemma4 26B.A4B Q4_0	64	pp4096	1211.13	1316.18	1.09
gemma4 26B.A4B Q4_0	512	pp4096	2948.92	2977.92	1.01
gemma4 26B.A4B Q4_0	1024	pp4096	3757.41	3758.58	1.00
gemma4 26B.A4B Q4_0	2048	pp4096	4038.17	4047.26	1.00

RDNA 3 Qwen3.6 35B A3B

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
qwen35moe 35B.A3B Q4_K_M	8	pp4096	459.57	460.89	1.00
qwen35moe 35B.A3B Q4_K_M	16	pp4096	661.79	664.03	1.00
qwen35moe 35B.A3B Q4_K_M	32	pp4096	901.82	902.21	1.00
qwen35moe 35B.A3B Q4_K_M	64	pp4096	1116.85	1231.24	1.10
qwen35moe 35B.A3B Q4_K_M	512	pp4096	2636.04	2694.36	1.02
qwen35moe 35B.A3B Q4_K_M	1024	pp4096	3520.65	3520.54	1.00
qwen35moe 35B.A3B Q4_K_M	2048	pp4096	4184.40	4203.54	1.00

RDNA 4 LFM2.5 8B A1B

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	16	pp2048	1624.50	1632.33	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	32	pp2048	2685.41	2692.24	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	64	pp2048	3789.03	3786.92	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	96	pp2048	4710.91	4656.28	0.99
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	128	pp2048	4752.37	4752.72	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	192	pp2048	7370.45	7369.10	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	256	pp2048	8138.03	8123.92	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	512	pp2048	11858.39	11844.98	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	1024	pp2048	15018.72	15016.07	1.00
lfm2moe 8B.A1B IQ4_XS - 4.25 bpw	2048	pp2048	16016.48	15985.83	1.00
lfm2moe 8B.A1B Q2_K_M	16	pp2048	1315.47	1364.98	1.04
lfm2moe 8B.A1B Q2_K_M	32	pp2048	1373.67	1377.21	1.00
lfm2moe 8B.A1B Q2_K_M	64	pp2048	655.10	697.17	1.06
lfm2moe 8B.A1B Q2_K_M	96	pp2048	1310.86	1522.28	1.16
lfm2moe 8B.A1B Q2_K_M	128	pp2048	1048.98	1106.16	1.05
lfm2moe 8B.A1B Q2_K_M	192	pp2048	1357.82	1431.94	1.05
lfm2moe 8B.A1B Q2_K_M	256	pp2048	1581.78	1721.51	1.09
lfm2moe 8B.A1B Q2_K_M	512	pp2048	836.29	935.28	1.12
lfm2moe 8B.A1B Q2_K_M	1024	pp2048	1031.31	1144.53	1.11
lfm2moe 8B.A1B Q2_K_M	2048	pp2048	1146.48	1277.09	1.11
lfm2moe 8B.A1B Q3_K_M	16	pp2048	1273.30	1399.93	1.10
lfm2moe 8B.A1B Q3_K_M	32	pp2048	2334.67	2329.94	1.00
lfm2moe 8B.A1B Q3_K_M	64	pp2048	3184.16	3177.58	1.00
lfm2moe 8B.A1B Q3_K_M	96	pp2048	3776.06	3794.34	1.00
lfm2moe 8B.A1B Q3_K_M	128	pp2048	3972.42	3973.47	1.00
lfm2moe 8B.A1B Q3_K_M	192	pp2048	5977.42	5982.75	1.00
lfm2moe 8B.A1B Q3_K_M	256	pp2048	6770.57	6787.43	1.00
lfm2moe 8B.A1B Q3_K_M	512	pp2048	9921.85	9892.31	1.00
lfm2moe 8B.A1B Q3_K_M	1024	pp2048	12528.66	12526.91	1.00
lfm2moe 8B.A1B Q3_K_M	2048	pp2048	13668.62	13598.49	0.99
lfm2moe 8B.A1B Q4_0	16	pp2048	1590.54	1590.54	1.00
lfm2moe 8B.A1B Q4_0	32	pp2048	2633.39	2632.05	1.00
lfm2moe 8B.A1B Q4_0	64	pp2048	3583.68	3570.76	1.00
lfm2moe 8B.A1B Q4_0	96	pp2048	4431.89	4445.66	1.00
lfm2moe 8B.A1B Q4_0	128	pp2048	4685.78	4666.59	1.00
lfm2moe 8B.A1B Q4_0	192	pp2048	7016.94	7017.19	1.00
lfm2moe 8B.A1B Q4_0	256	pp2048	8004.15	8045.02	1.01
lfm2moe 8B.A1B Q4_0	512	pp2048	11814.67	11767.38	1.00
lfm2moe 8B.A1B Q4_0	1024	pp2048	14975.58	14968.01	1.00
lfm2moe 8B.A1B Q4_0	2048	pp2048	16068.38	16020.17	1.00
lfm2moe 8B.A1B Q4_K_M	16	pp2048	1498.60	1495.35	1.00
lfm2moe 8B.A1B Q4_K_M	32	pp2048	2488.63	2459.19	0.99
lfm2moe 8B.A1B Q4_K_M	64	pp2048	3378.38	3358.66	0.99
lfm2moe 8B.A1B Q4_K_M	96	pp2048	3982.06	3952.19	0.99
lfm2moe 8B.A1B Q4_K_M	128	pp2048	4147.94	4132.50	1.00
lfm2moe 8B.A1B Q4_K_M	192	pp2048	6233.82	6201.71	0.99
lfm2moe 8B.A1B Q4_K_M	256	pp2048	7050.24	7031.89	1.00
lfm2moe 8B.A1B Q4_K_M	512	pp2048	10427.27	10416.85	1.00
lfm2moe 8B.A1B Q4_K_M	1024	pp2048	13196.85	13165.75	1.00
lfm2moe 8B.A1B Q4_K_M	2048	pp2048	14366.74	14375.10	1.00
lfm2moe 8B.A1B Q5_0	16	pp2048	1453.73	1446.25	0.99
lfm2moe 8B.A1B Q5_0	32	pp2048	2372.99	2364.61	1.00
lfm2moe 8B.A1B Q5_0	64	pp2048	3366.25	3347.54	0.99
lfm2moe 8B.A1B Q5_0	96	pp2048	4152.68	4148.91	1.00
lfm2moe 8B.A1B Q5_0	128	pp2048	3327.69	4335.11	1.30
lfm2moe 8B.A1B Q5_0	192	pp2048	6362.08	6552.64	1.03
lfm2moe 8B.A1B Q5_0	256	pp2048	5632.62	7399.28	1.31
lfm2moe 8B.A1B Q5_0	512	pp2048	8697.77	10860.47	1.25
lfm2moe 8B.A1B Q5_0	1024	pp2048	11320.63	13879.74	1.23
lfm2moe 8B.A1B Q5_0	2048	pp2048	12465.55	15069.32	1.21
lfm2moe 8B.A1B Q5_1	16	pp2048	1443.63	1449.36	1.00
lfm2moe 8B.A1B Q5_1	32	pp2048	2438.28	2429.39	1.00
lfm2moe 8B.A1B Q5_1	64	pp2048	3410.96	3420.54	1.00
lfm2moe 8B.A1B Q5_1	96	pp2048	4099.13	4117.27	1.00
lfm2moe 8B.A1B Q5_1	128	pp2048	3350.10	4355.61	1.30
lfm2moe 8B.A1B Q5_1	192	pp2048	6315.36	6454.33	1.02
lfm2moe 8B.A1B Q5_1	256	pp2048	5637.17	7369.26	1.31
lfm2moe 8B.A1B Q5_1	512	pp2048	8603.23	10881.40	1.26
lfm2moe 8B.A1B Q5_1	1024	pp2048	11221.72	13843.10	1.23
lfm2moe 8B.A1B Q5_1	2048	pp2048	12355.04	14981.69	1.21
lfm2moe 8B.A1B Q5_K_M	16	pp2048	1454.75	1457.36	1.00
lfm2moe 8B.A1B Q5_K_M	32	pp2048	2395.44	2395.89	1.00
lfm2moe 8B.A1B Q5_K_M	64	pp2048	3282.96	3289.28	1.00
lfm2moe 8B.A1B Q5_K_M	96	pp2048	3864.35	3846.05	1.00
lfm2moe 8B.A1B Q5_K_M	128	pp2048	4049.65	4032.18	1.00
lfm2moe 8B.A1B Q5_K_M	192	pp2048	6044.70	6055.65	1.00
lfm2moe 8B.A1B Q5_K_M	256	pp2048	6859.74	6886.75	1.00
lfm2moe 8B.A1B Q5_K_M	512	pp2048	10192.70	10169.92	1.00
lfm2moe 8B.A1B Q5_K_M	1024	pp2048	12940.62	12920.29	1.00
lfm2moe 8B.A1B Q5_K_M	2048	pp2048	14170.00	14121.52	1.00
lfm2moe 8B.A1B Q6_K	16	pp2048	1242.61	1247.71	1.00
lfm2moe 8B.A1B Q6_K	32	pp2048	1880.08	1858.84	0.99
lfm2moe 8B.A1B Q6_K	64	pp2048	2350.12	2315.51	0.99
lfm2moe 8B.A1B Q6_K	96	pp2048	2513.74	2507.76	1.00
lfm2moe 8B.A1B Q6_K	128	pp2048	2578.62	2573.92	1.00
lfm2moe 8B.A1B Q6_K	192	pp2048	3964.49	3950.43	1.00
lfm2moe 8B.A1B Q6_K	256	pp2048	4396.18	4359.71	0.99
lfm2moe 8B.A1B Q6_K	512	pp2048	6562.99	6528.20	0.99
lfm2moe 8B.A1B Q6_K	1024	pp2048	8225.36	8187.53	1.00
lfm2moe 8B.A1B Q6_K	2048	pp2048	9518.45	9496.19	1.00
lfm2moe 8B.A1B Q8_0	16	pp2048	1275.64	1276.31	1.00
lfm2moe 8B.A1B Q8_0	32	pp2048	2150.15	2153.23	1.00
lfm2moe 8B.A1B Q8_0	64	pp2048	3377.93	3352.05	0.99
lfm2moe 8B.A1B Q8_0	96	pp2048	4346.00	4280.42	0.98
lfm2moe 8B.A1B Q8_0	128	pp2048	4486.11	4447.73	0.99
lfm2moe 8B.A1B Q8_0	192	pp2048	6816.20	6801.47	1.00
lfm2moe 8B.A1B Q8_0	256	pp2048	7645.89	7620.08	1.00
lfm2moe 8B.A1B Q8_0	512	pp2048	11463.09	11324.91	0.99
lfm2moe 8B.A1B Q8_0	1024	pp2048	14910.83	14779.49	0.99
lfm2moe 8B.A1B Q8_0	2048	pp2048	16426.98	16328.43	0.99

RDNA 4 Gemma 4

Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
gemma4 12B Q4_0	8	pp4096	273.76	273.33	1.00
gemma4 12B Q4_0	16	pp4096	589.08	588.77	1.00
gemma4 12B Q4_0	32	pp4096	859.97	858.37	1.00
gemma4 12B Q4_0	64	pp4096	1117.07	1114.65	1.00
gemma4 12B Q4_0	512	pp4096	2298.71	2299.09	1.00
gemma4 12B Q4_0	1024	pp4096	2297.34	2296.29	1.00
gemma4 12B Q4_0	2048	pp4096	2245.91	2245.75	1.00
gemma4 26B.A4B Q4_0	8	pp2048	458.81	458.95	1.00
gemma4 26B.A4B Q4_0	16	pp2048	651.41	650.09	1.00
gemma4 26B.A4B Q4_0	32	pp2048	991.57	989.96	1.00
gemma4 26B.A4B Q4_0	64	pp2048	1367.03	1367.02	1.00
gemma4 26B.A4B Q4_0	512	pp2048	3601.43	3586.60	1.00
gemma4 26B.A4B Q4_0	1024	pp2048	4421.17	4417.47	1.00

dfriehs · 2026-07-30T23:08:45Z

Would you be willing to share the scripts you use for sweeps? I'd like to run them for my own changes (and possibly sm_60, which isn't using mmq yet).

itterative · 2026-07-31T06:17:58Z

Tried to tune Q2_0 as well, now that it's in master. RDNA 4 results were basically the same (same changes, occupancy 4 vs occupancy 2), so I skipped those. RDNA 3 had a small speedup.

RDNA 3 LFM 2.5 8B A1B

Model	Microbatch size	Test	t/s `cad1cd8`	t/s q20-rdna3-tune	Speedup
lfm2moe 8B.A1B Q2_0	16	pp2048	1923.16	1930.59	1.00
lfm2moe 8B.A1B Q2_0	32	pp2048	2423.96	2430.57	1.00
lfm2moe 8B.A1B Q2_0	64	pp2048	3325.71	3555.77	1.07
lfm2moe 8B.A1B Q2_0	96	pp2048	3571.96	3722.70	1.04
lfm2moe 8B.A1B Q2_0	128	pp2048	3947.23	3946.07	1.00
lfm2moe 8B.A1B Q2_0	192	pp2048	5841.91	5964.86	1.02
lfm2moe 8B.A1B Q2_0	256	pp2048	6612.88	6673.29	1.01
lfm2moe 8B.A1B Q2_0	512	pp2048	9777.96	9847.29	1.01
lfm2moe 8B.A1B Q2_0	1024	pp2048	12692.70	12767.50	1.01
lfm2moe 8B.A1B Q2_0	2048	pp2048	13518.86	13618.22	1.01

RDNA 3 llama 3 8B

Model	Microbatch size	Test	t/s `cad1cd8`	t/s q20-rdna3-tune	Speedup
llama 8B Q2_0	16	pp2048	1152.78	1148.42	1.00
llama 8B Q2_0	32	pp2048	1445.49	1445.49	1.00
llama 8B Q2_0	64	pp2048	1930.97	2133.18	1.10
llama 8B Q2_0	96	pp2048	2061.36	2257.50	1.10
llama 8B Q2_0	128	pp2048	2317.43	2313.23	1.00
llama 8B Q2_0	192	pp2048	2929.43	3205.07	1.09
llama 8B Q2_0	256	pp2048	3297.08	3292.25	1.00
llama 8B Q2_0	512	pp2048	3819.61	3820.18	1.00
llama 8B Q2_0	1024	pp2048	3851.29	3849.91	1.00
llama 8B Q2_0	2048	pp2048	3970.85	3963.95	1.00

@dfriehs I've uploaded a patch with the scripts Opus made. You'll need to add some arch specific changes to it when you want to tune for your case.

mmq-sweep.patch

IMbackK · 2026-08-10T11:11:03Z

There are huge speed increases when tuning for MoE over Dense which I noticed myself. I think using stream-k I got over 20% increase for Dense. I do believe there would be a bit of work involved in picking different configs for MMQ based on MoE or Dense, but I believe its worth the added complexity, which shouldn't be much.

I agree but i think it should be a separate pr.

IMbackK

Looks good from static analysis

I did a sweeped over llama2 (q4_0, q4_k, q5_k, q8_0), gpt-oss-20b and qwen 3.5 (i13, q4_k) on gfx1100 and found no regressions

Cant test the gfx12 changes but im going to trust in your results.
I will merge once ci finishes

Geramy · 2026-08-10T18:46:48Z

-- COMMENT WAS A MISTAKE, WRONG PR --
Fixes #26746

Patt92 · 2026-08-11T05:46:44Z

Fixes #26746

Is this PR really expected to affect the GGML_OP_TOP_K failure I’m seeing on gfx1151, or was the intended reference perhaps #26592 because of its hipCUB TOP_K/ARGSORT changes?

Do you think the crash could actually be caused by an RDNA3.5-specific kernel configuration / LDS / launch-layout issue similar to what this PR addresses, rather than the existing HIP TOP_K implementation itself?
I’m asking because the GPU appears capable of executing the workload: the same model and RPC setup works with the Vulkan backend, while only the ROCm backend aborts with invalid configuration argument.

With Vulkan, the RPC worker no longer crashes at the same TOP_K point, which suggests that gfx1151 itself is capable of executing the workload. However, DeepSeek V4 currently has a separate RPC + Vulkan issue resulting in garbled output, tracked in #26685, so Vulkan unfortunately isn’t a usable workaround for me.
That’s why I assumed that the missing hipCUB/rocPRIM path might be closer to the root cause, since Vulkan has its own implementation. Or am I misunderstanding how these two PRs are related?

Geramy · 2026-08-11T17:42:57Z

@Patt92 Sorry about that, it was an accident I meant to post that to another PR haha, thanks for catching that.

cherry-pick ggml-org#26284 中 mmq-config-rdna3.cuh 的最终内容 (3 commits: c256b87 + cad1cd8 + 81a7d2e)。不取 rdna4 配置 (非本 fork 目标)。 7900 XTX 实测: prefill 大 batch 1.1~1.5x (Q4_K_M/Q5_K/Q8_0)。注: 该 PR 在上游因 CI 的 VGPR 预算检查 (CDNA 目标) 未合, RDNA3 上 VGPR 上限 512 不受影响。

IMbackK

I rebased this and tested it again against gfx1201 and gfx1100. This pr makes no improvments for gfx12, and in fact seams to interact poorly with 15e755f.

Please remove the rdna4 changes and then we can merge this.

Benchmarks gfx11

GPU	Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
RX 7900 XTX	gemma4 26B.A4B Q6_K	64	pp2048	965.33	1068.56	1.11
RX 7900 XTX	gemma4 26B.A4B Q6_K	64	pp2048@d4096	941.15	1035.53	1.10
RX 7900 XTX	gemma4 26B.A4B Q6_K	64	tg64	88.49	88.70	1.00
RX 7900 XTX	gemma4 26B.A4B Q6_K	64	tg64@d4096	84.03	84.14	1.00
RX 7900 XTX	gemma4 26B.A4B Q6_K	512	pp2048	2448.18	2497.75	1.02
RX 7900 XTX	gemma4 26B.A4B Q6_K	512	pp2048@d4096	2212.35	2255.94	1.02
RX 7900 XTX	gemma4 26B.A4B Q6_K	512	tg64	88.47	88.43	1.00
RX 7900 XTX	gemma4 26B.A4B Q6_K	512	tg64@d4096	83.93	82.23	0.98
RX 7900 XTX	gemma4 26B.A4B Q6_K	1024	pp2048	3301.97	3326.84	1.01
RX 7900 XTX	gemma4 26B.A4B Q6_K	1024	pp2048@d4096	2766.62	2793.60	1.01
RX 7900 XTX	gemma4 26B.A4B Q6_K	1024	tg64	88.27	83.86	0.95
RX 7900 XTX	gemma4 26B.A4B Q6_K	1024	tg64@d4096	84.19	84.20	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	64	pp2048	1488.90	1631.56	1.10
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	64	pp2048@d4096	1415.62	1550.90	1.10
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	64	tg64	167.95	167.97	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	64	tg64@d4096	163.54	164.02	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	512	pp2048	3857.56	3914.78	1.01
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	512	pp2048@d4096	3516.22	3553.29	1.01
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	512	tg64	167.52	168.07	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	512	tg64@d4096	163.45	163.77	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	1024	pp2048	4856.87	4874.06	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	1024	pp2048@d4096	4252.22	4280.42	1.01
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	1024	tg64	167.65	167.92	1.00
RX 7900 XTX	gpt-oss 20B MXFP4 MoE	1024	tg64@d4096	153.98	163.06	1.06
RX 7900 XTX	lfm2moe 8B.A1B F16	64	pp2048	2473.02	2469.07	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	64	pp2048@d4096	2449.95	2443.06	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	64	tg64	161.02	161.24	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	64	tg64@d4096	158.98	158.74	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	512	pp2048	6108.64	6110.41	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	512	pp2048@d4096	5970.00	5988.31	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	512	tg64	160.80	160.73	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	512	tg64@d4096	158.61	159.00	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	1024	pp2048	8400.09	8483.94	1.01
RX 7900 XTX	lfm2moe 8B.A1B F16	1024	pp2048@d4096	8045.51	8071.03	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	1024	tg64	159.98	160.23	1.00
RX 7900 XTX	lfm2moe 8B.A1B F16	1024	tg64@d4096	158.35	158.79	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	64	pp2048	3277.55	3493.63	1.07
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	64	pp2048@d4096	3235.02	3441.52	1.06
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	64	tg64	254.26	254.12	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	64	tg64@d4096	249.28	249.30	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	512	pp2048	9183.04	9405.92	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	512	pp2048@d4096	8766.94	8968.52	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	512	tg64	253.66	253.82	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	512	tg64@d4096	248.96	249.47	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	1024	pp2048	11835.13	11918.00	1.01
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	1024	pp2048@d4096	11103.12	11174.83	1.01
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	1024	tg64	253.22	253.48	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_0	1024	tg64@d4096	248.70	249.18	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	64	pp2048	3009.05	3262.60	1.08
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	64	pp2048@d4096	2894.74	3215.67	1.11
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	64	tg64	259.50	258.28	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	64	tg64@d4096	253.87	253.43	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	512	pp2048	8380.44	8632.38	1.03
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	512	pp2048@d4096	8063.60	8288.19	1.03
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	512	tg64	259.71	257.87	0.99
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	512	tg64@d4096	255.26	253.18	0.99
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	1024	pp2048	10984.11	11023.06	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	1024	pp2048@d4096	10333.75	10358.72	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	1024	tg64	259.71	245.36	0.94
RX 7900 XTX	lfm2moe 8B.A1B Q4_K_M	1024	tg64@d4096	255.22	254.02	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	64	pp2048	2967.70	3182.77	1.07
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	64	pp2048@d4096	2939.70	3145.22	1.07
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	64	tg64	248.97	248.57	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	64	tg64@d4096	244.01	244.11	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	512	pp2048	8191.12	8461.29	1.03
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	512	pp2048@d4096	7874.64	8120.87	1.03
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	512	tg64	248.38	247.49	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	512	tg64@d4096	244.00	244.13	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	1024	pp2048	10771.46	10827.59	1.01
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	1024	pp2048@d4096	10105.93	10197.94	1.01
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	1024	tg64	247.43	248.49	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q5_K_M	1024	tg64@d4096	243.82	244.16	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	64	pp2048	2118.73	2299.05	1.09
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	64	pp2048@d4096	2098.64	2279.20	1.09
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	64	tg64	238.37	238.15	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	64	tg64@d4096	235.37	234.90	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	512	pp2048	5799.48	6086.85	1.05
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	512	pp2048@d4096	5655.74	5912.90	1.05
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	512	tg64	239.06	238.51	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	512	tg64@d4096	235.01	234.46	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	1024	pp2048	7705.44	7865.29	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	1024	pp2048@d4096	7378.59	7505.45	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	1024	tg64	238.52	238.08	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q6_K	1024	tg64@d4096	234.85	234.40	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	64	pp2048	3191.76	3551.77	1.11
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	64	pp2048@d4096	3143.39	3495.62	1.11
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	64	tg64	229.07	229.30	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	64	tg64@d4096	225.16	225.75	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	512	pp2048	9196.74	9354.14	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	512	pp2048@d4096	8806.61	8940.35	1.02
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	512	tg64	228.44	229.03	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	512	tg64@d4096	224.46	225.85	1.01
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	1024	pp2048	12029.91	12016.72	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	1024	pp2048@d4096	11287.62	11254.23	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	1024	tg64	228.44	228.91	1.00
RX 7900 XTX	lfm2moe 8B.A1B Q8_0	1024	tg64@d4096	224.78	225.53	1.00
RX 7900 XTX	llama 8B Q2_K_M	64	pp2048	1177.61	1204.22	1.02
RX 7900 XTX	llama 8B Q2_K_M	64	pp2048@d4096	1083.45	1106.00	1.02
RX 7900 XTX	llama 8B Q2_K_M	64	tg64	131.04	131.14	1.00
RX 7900 XTX	llama 8B Q2_K_M	64	tg64@d4096	119.90	119.41	1.00
RX 7900 XTX	llama 8B Q2_K_M	512	pp2048	2054.06	2152.31	1.05
RX 7900 XTX	llama 8B Q2_K_M	512	pp2048@d4096	1814.31	1902.04	1.05
RX 7900 XTX	llama 8B Q2_K_M	512	tg64	131.07	131.06	1.00
RX 7900 XTX	llama 8B Q2_K_M	512	tg64@d4096	119.78	119.60	1.00
RX 7900 XTX	llama 8B Q2_K_M	1024	pp2048	2205.82	2227.87	1.01
RX 7900 XTX	llama 8B Q2_K_M	1024	pp2048@d4096	1935.52	1954.02	1.01
RX 7900 XTX	llama 8B Q2_K_M	1024	tg64	130.44	130.91	1.00
RX 7900 XTX	llama 8B Q2_K_M	1024	tg64@d4096	119.83	119.61	1.00
RX 7900 XTX	llama 8B Q3_K_M	64	pp2048	1780.33	1987.53	1.12
RX 7900 XTX	llama 8B Q3_K_M	64	pp2048@d4096	1587.87	1760.29	1.11
RX 7900 XTX	llama 8B Q3_K_M	64	tg64	106.57	106.90	1.00
RX 7900 XTX	llama 8B Q3_K_M	64	tg64@d4096	99.43	99.18	1.00
RX 7900 XTX	llama 8B Q3_K_M	512	pp2048	3121.53	3208.49	1.03
RX 7900 XTX	llama 8B Q3_K_M	512	pp2048@d4096	2593.19	2676.36	1.03
RX 7900 XTX	llama 8B Q3_K_M	512	tg64	106.15	99.04	0.93
RX 7900 XTX	llama 8B Q3_K_M	512	tg64@d4096	99.29	99.04	1.00
RX 7900 XTX	llama 8B Q3_K_M	1024	pp2048	3223.76	3076.65	0.95
RX 7900 XTX	llama 8B Q3_K_M	1024	pp2048@d4096	2663.56	2747.33	1.03
RX 7900 XTX	llama 8B Q3_K_M	1024	tg64	106.14	106.11	1.00
RX 7900 XTX	llama 8B Q3_K_M	1024	tg64@d4096	99.15	98.92	1.00
RX 7900 XTX	llama 8B Q4_0	64	pp2048	2010.97	2213.97	1.10
RX 7900 XTX	llama 8B Q4_0	64	pp2048@d4096	1773.67	1941.36	1.09
RX 7900 XTX	llama 8B Q4_0	64	tg64	126.99	126.90	1.00
RX 7900 XTX	llama 8B Q4_0	64	tg64@d4096	116.83	116.90	1.00
RX 7900 XTX	llama 8B Q4_0	512	pp2048	3546.60	3572.70	1.01
RX 7900 XTX	llama 8B Q4_0	512	pp2048@d4096	2911.55	2933.80	1.01
RX 7900 XTX	llama 8B Q4_0	512	tg64	126.54	126.72	1.00
RX 7900 XTX	llama 8B Q4_0	512	tg64@d4096	116.59	116.71	1.00
RX 7900 XTX	llama 8B Q4_0	1024	pp2048	3300.03	3694.36	1.12
RX 7900 XTX	llama 8B Q4_0	1024	pp2048@d4096	2954.97	2707.06	0.92
RX 7900 XTX	llama 8B Q4_0	1024	tg64	126.27	126.43	1.00
RX 7900 XTX	llama 8B Q4_0	1024	tg64@d4096	116.51	116.65	1.00
RX 7900 XTX	llama 8B Q4_K_M	64	pp2048	1761.56	1984.36	1.13
RX 7900 XTX	llama 8B Q4_K_M	64	pp2048@d4096	1563.87	1752.00	1.12
RX 7900 XTX	llama 8B Q4_K_M	64	tg64	106.00	106.14	1.00
RX 7900 XTX	llama 8B Q4_K_M	64	tg64@d4096	98.82	98.79	1.00
RX 7900 XTX	llama 8B Q4_K_M	512	pp2048	3189.36	3226.54	1.01
RX 7900 XTX	llama 8B Q4_K_M	512	pp2048@d4096	2677.85	2693.13	1.01
RX 7900 XTX	llama 8B Q4_K_M	512	tg64	105.75	105.81	1.00
RX 7900 XTX	llama 8B Q4_K_M	512	tg64@d4096	98.67	98.54	1.00
RX 7900 XTX	llama 8B Q4_K_M	1024	pp2048	3294.26	3390.81	1.03
RX 7900 XTX	llama 8B Q4_K_M	1024	pp2048@d4096	2740.33	2796.20	1.02
RX 7900 XTX	llama 8B Q4_K_M	1024	tg64	105.47	105.59	1.00
RX 7900 XTX	llama 8B Q4_K_M	1024	tg64@d4096	98.64	98.61	1.00
RX 7900 XTX	llama 8B Q5_K_M	64	pp2048	1793.85	1969.98	1.10
RX 7900 XTX	llama 8B Q5_K_M	64	pp2048@d4096	1597.72	1749.27	1.09
RX 7900 XTX	llama 8B Q5_K_M	64	tg64	100.63	100.47	1.00
RX 7900 XTX	llama 8B Q5_K_M	64	tg64@d4096	93.85	93.88	1.00
RX 7900 XTX	llama 8B Q5_K_M	512	pp2048	3154.94	3192.94	1.01
RX 7900 XTX	llama 8B Q5_K_M	512	pp2048@d4096	2387.36	2670.79	1.12
RX 7900 XTX	llama 8B Q5_K_M	512	tg64	100.07	100.14	1.00
RX 7900 XTX	llama 8B Q5_K_M	512	tg64@d4096	93.72	93.88	1.00
RX 7900 XTX	llama 8B Q5_K_M	1024	pp2048	3251.91	3338.50	1.03
RX 7900 XTX	llama 8B Q5_K_M	1024	pp2048@d4096	2692.42	2765.54	1.03
RX 7900 XTX	llama 8B Q5_K_M	1024	tg64	99.79	100.13	1.00
RX 7900 XTX	llama 8B Q5_K_M	1024	tg64@d4096	93.65	93.86	1.00
RX 7900 XTX	llama 8B Q6_K	64	pp2048	1292.16	1448.24	1.12
RX 7900 XTX	llama 8B Q6_K	64	pp2048@d4096	1185.95	1320.60	1.11
RX 7900 XTX	llama 8B Q6_K	64	tg64	97.77	98.12	1.00
RX 7900 XTX	llama 8B Q6_K	64	tg64@d4096	91.42	91.51	1.00
RX 7900 XTX	llama 8B Q6_K	512	pp2048	2221.37	2270.41	1.02
RX 7900 XTX	llama 8B Q6_K	512	pp2048@d4096	1951.66	1986.43	1.02
RX 7900 XTX	llama 8B Q6_K	512	tg64	97.65	97.71	1.00
RX 7900 XTX	llama 8B Q6_K	512	tg64@d4096	91.40	91.57	1.00
RX 7900 XTX	llama 8B Q6_K	1024	pp2048	2272.85	2355.11	1.04
RX 7900 XTX	llama 8B Q6_K	1024	pp2048@d4096	1913.81	1871.39	0.98
RX 7900 XTX	llama 8B Q6_K	1024	tg64	97.57	93.67	0.96
RX 7900 XTX	llama 8B Q6_K	1024	tg64@d4096	91.30	91.31	1.00
RX 7900 XTX	llama 8B Q8_0	64	pp2048	1994.62	2250.83	1.13
RX 7900 XTX	llama 8B Q8_0	64	pp2048@d4096	1757.03	1969.95	1.12
RX 7900 XTX	llama 8B Q8_0	64	tg64	86.15	86.09	1.00
RX 7900 XTX	llama 8B Q8_0	64	tg64@d4096	81.19	81.09	1.00
RX 7900 XTX	llama 8B Q8_0	512	pp2048	3596.07	3581.06	1.00
RX 7900 XTX	llama 8B Q8_0	512	pp2048@d4096	2947.37	2947.03	1.00
RX 7900 XTX	llama 8B Q8_0	512	tg64	83.91	85.90	1.02
RX 7900 XTX	llama 8B Q8_0	512	tg64@d4096	81.32	80.84	0.99
RX 7900 XTX	llama 8B Q8_0	1024	pp2048	3687.08	3711.38	1.01
RX 7900 XTX	llama 8B Q8_0	1024	pp2048@d4096	2999.74	3020.18	1.01
RX 7900 XTX	llama 8B Q8_0	1024	tg64	86.18	85.71	0.99
RX 7900 XTX	llama 8B Q8_0	1024	tg64@d4096	81.29	80.99	1.00

Benchmarks gfx12

GPU	Model	Microbatch size	Test	t/s master	t/s rdna-mmq-config	Speedup
AI PRO R9700	gemma4 26B.A4B Q6_K	64	pp2048	1032.67	1041.66	1.01
AI PRO R9700	gemma4 26B.A4B Q6_K	64	pp2048@d4096	963.75	967.72	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	64	tg64	81.20	81.12	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	64	tg64@d4096	77.43	73.87	0.95
AI PRO R9700	gemma4 26B.A4B Q6_K	512	pp2048	2695.33	2708.61	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	512	pp2048@d4096	2324.65	2333.52	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	512	tg64	81.13	80.97	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	512	tg64@d4096	77.39	77.32	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	1024	pp2048	3503.55	3511.79	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	1024	pp2048@d4096	2796.28	2801.52	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	1024	tg64	81.19	80.94	1.00
AI PRO R9700	gemma4 26B.A4B Q6_K	1024	tg64@d4096	77.39	77.28	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	64	pp2048	1947.96	1954.30	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	64	pp2048@d4096	1850.08	1854.87	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	64	tg64	143.24	143.61	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	64	tg64@d4096	138.50	138.30	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	512	pp2048	4973.33	4981.19	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	512	pp2048@d4096	4509.39	4522.77	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	512	tg64	143.33	143.75	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	512	tg64@d4096	138.51	138.82	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	1024	pp2048	6103.34	6111.14	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	1024	pp2048@d4096	5412.99	5421.14	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	1024	tg64	143.18	143.72	1.00
AI PRO R9700	gpt-oss 20B MXFP4 MoE	1024	tg64@d4096	138.43	138.88	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	64	pp2048	1501.20	1502.28	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	64	pp2048@d4096	1500.12	1501.33	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	64	tg64	127.92	128.12	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	64	tg64@d4096	125.72	126.19	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	512	pp2048	1967.32	1984.25	1.01
AI PRO R9700	lfm2moe 8B.A1B F16	512	pp2048@d4096	1955.95	1974.88	1.01
AI PRO R9700	lfm2moe 8B.A1B F16	512	tg64	126.94	127.32	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	512	tg64@d4096	125.55	125.86	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	1024	pp2048	2476.17	2493.76	1.01
AI PRO R9700	lfm2moe 8B.A1B F16	1024	pp2048@d4096	2453.55	2473.83	1.01
AI PRO R9700	lfm2moe 8B.A1B F16	1024	tg64	127.12	127.37	1.00
AI PRO R9700	lfm2moe 8B.A1B F16	1024	tg64@d4096	125.50	125.73	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_0	64	pp2048	3508.87	3528.11	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_0	64	pp2048@d4096	3472.66	3498.07	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_0	64	tg64	250.73	249.98	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_0	64	tg64@d4096	244.69	243.67	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_0	512	pp2048	11447.48	11500.02	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_0	512	pp2048@d4096	10944.05	11043.31	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_0	512	tg64	250.49	249.16	0.99
AI PRO R9700	lfm2moe 8B.A1B Q4_0	512	tg64@d4096	245.70	244.02	0.99
AI PRO R9700	lfm2moe 8B.A1B Q4_0	1024	pp2048	14605.71	14655.82	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_0	1024	pp2048@d4096	13678.79	13756.35	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_0	1024	tg64	250.87	249.21	0.99
AI PRO R9700	lfm2moe 8B.A1B Q4_0	1024	tg64@d4096	244.93	244.09	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	64	pp2048	3280.46	3296.35	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	64	pp2048@d4096	3244.23	3256.13	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	64	tg64	209.53	211.44	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	64	tg64@d4096	205.48	205.94	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	512	pp2048	10033.18	10061.88	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	512	pp2048@d4096	9673.21	9698.75	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	512	tg64	209.50	209.66	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	512	tg64@d4096	206.01	205.78	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	1024	pp2048	12782.96	12866.25	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	1024	pp2048@d4096	12077.30	12145.67	1.01
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	1024	tg64	210.36	209.72	1.00
AI PRO R9700	lfm2moe 8B.A1B Q4_K_M	1024	tg64@d4096	205.75	205.70	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	64	pp2048	3210.07	3222.81	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	64	pp2048@d4096	3175.41	3190.55	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	64	tg64	207.95	208.22	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	64	tg64@d4096	203.27	203.76	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	512	pp2048	9834.08	9835.93	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	512	pp2048@d4096	9457.55	9471.63	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	512	tg64	207.11	207.74	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	512	tg64@d4096	202.97	203.91	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	1024	pp2048	11924.02	12678.12	1.06
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	1024	pp2048@d4096	11932.21	11978.28	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	1024	tg64	207.99	207.20	1.00
AI PRO R9700	lfm2moe 8B.A1B Q5_K_M	1024	tg64@d4096	204.12	203.86	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	64	pp2048	2271.63	2277.36	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	64	pp2048@d4096	2249.92	2225.99	0.99
AI PRO R9700	lfm2moe 8B.A1B Q6_K	64	tg64	219.64	219.29	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	64	tg64@d4096	214.87	215.33	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	512	pp2048	6410.33	6430.96	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	512	pp2048@d4096	6267.98	6289.21	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	512	tg64	218.50	218.74	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	512	tg64@d4096	214.22	214.68	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	1024	pp2048	8350.38	8378.82	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	1024	pp2048@d4096	8055.48	8088.65	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	1024	tg64	217.48	218.36	1.00
AI PRO R9700	lfm2moe 8B.A1B Q6_K	1024	tg64@d4096	214.25	214.39	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	64	pp2048	3307.42	3304.86	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	64	pp2048@d4096	3271.21	3275.59	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	64	tg64	201.45	200.96	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	64	tg64@d4096	198.09	196.82	0.99
AI PRO R9700	lfm2moe 8B.A1B Q8_0	512	pp2048	11409.99	11460.98	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	512	pp2048@d4096	10941.08	10984.02	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	512	tg64	201.86	200.64	0.99
AI PRO R9700	lfm2moe 8B.A1B Q8_0	512	tg64@d4096	198.06	196.92	0.99
AI PRO R9700	lfm2moe 8B.A1B Q8_0	1024	pp2048	14965.48	15026.66	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	1024	pp2048@d4096	14015.91	14081.29	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	1024	tg64	201.76	201.13	1.00
AI PRO R9700	lfm2moe 8B.A1B Q8_0	1024	tg64@d4096	198.44	198.38	1.00
AI PRO R9700	llama 8B Q2_K_M	64	pp2048	1215.82	1504.05	1.24
AI PRO R9700	llama 8B Q2_K_M	64	pp2048@d4096	1144.22	1380.79	1.21
AI PRO R9700	llama 8B Q2_K_M	64	tg64	103.72	104.31	1.01
AI PRO R9700	llama 8B Q2_K_M	64	tg64@d4096	95.19	95.65	1.00
AI PRO R9700	llama 8B Q2_K_M	512	pp2048	2238.58	1760.59	0.79
AI PRO R9700	llama 8B Q2_K_M	512	pp2048@d4096	2003.96	1611.00	0.80
AI PRO R9700	llama 8B Q2_K_M	512	tg64	102.85	103.42	1.01
AI PRO R9700	llama 8B Q2_K_M	512	tg64@d4096	94.96	95.38	1.00
AI PRO R9700	llama 8B Q2_K_M	1024	pp2048	2394.48	1919.38	0.80
AI PRO R9700	llama 8B Q2_K_M	1024	pp2048@d4096	2113.70	1660.95	0.79
AI PRO R9700	llama 8B Q2_K_M	1024	tg64	102.75	103.44	1.01
AI PRO R9700	llama 8B Q2_K_M	1024	tg64@d4096	94.99	95.36	1.00
AI PRO R9700	llama 8B Q3_K_M	64	pp2048	2338.03	2335.86	1.00
AI PRO R9700	llama 8B Q3_K_M	64	pp2048@d4096	2075.16	2075.66	1.00
AI PRO R9700	llama 8B Q3_K_M	64	tg64	93.60	93.78	1.00
AI PRO R9700	llama 8B Q3_K_M	64	tg64@d4096	86.44	86.69	1.00
AI PRO R9700	llama 8B Q3_K_M	512	pp2048	4007.96	4008.50	1.00
AI PRO R9700	llama 8B Q3_K_M	512	pp2048@d4096	3317.23	3315.60	1.00
AI PRO R9700	llama 8B Q3_K_M	512	tg64	92.87	93.10	1.00
AI PRO R9700	llama 8B Q3_K_M	512	tg64@d4096	86.19	86.31	1.00
AI PRO R9700	llama 8B Q3_K_M	1024	pp2048	4040.71	4041.22	1.00
AI PRO R9700	llama 8B Q3_K_M	1024	pp2048@d4096	3151.19	3297.17	1.05
AI PRO R9700	llama 8B Q3_K_M	1024	tg64	92.68	92.77	1.00
AI PRO R9700	llama 8B Q3_K_M	1024	tg64@d4096	86.06	85.93	1.00
AI PRO R9700	llama 8B Q4_0	64	pp2048	2675.91	2680.63	1.00
AI PRO R9700	llama 8B Q4_0	64	pp2048@d4096	2342.35	2346.06	1.00
AI PRO R9700	llama 8B Q4_0	64	tg64	105.84	105.89	1.00
AI PRO R9700	llama 8B Q4_0	64	tg64@d4096	96.66	96.66	1.00
AI PRO R9700	llama 8B Q4_0	512	pp2048	4814.91	4856.32	1.01
AI PRO R9700	llama 8B Q4_0	512	pp2048@d4096	3883.94	3890.68	1.00
AI PRO R9700	llama 8B Q4_0	512	tg64	105.88	105.64	1.00
AI PRO R9700	llama 8B Q4_0	512	tg64@d4096	96.79	96.91	1.00
AI PRO R9700	llama 8B Q4_0	1024	pp2048	4882.15	4919.05	1.01
AI PRO R9700	llama 8B Q4_0	1024	pp2048@d4096	3879.07	3892.24	1.00
AI PRO R9700	llama 8B Q4_0	1024	tg64	105.71	105.93	1.00
AI PRO R9700	llama 8B Q4_0	1024	tg64@d4096	92.96	96.75	1.04
AI PRO R9700	llama 8B Q4_K_M	64	pp2048	2386.18	2382.02	1.00
AI PRO R9700	llama 8B Q4_K_M	64	pp2048@d4096	2110.32	2111.28	1.00
AI PRO R9700	llama 8B Q4_K_M	64	tg64	95.32	95.43	1.00
AI PRO R9700	llama 8B Q4_K_M	64	tg64@d4096	87.59	88.03	1.00
AI PRO R9700	llama 8B Q4_K_M	512	pp2048	3853.91	4095.11	1.06
AI PRO R9700	llama 8B Q4_K_M	512	pp2048@d4096	3381.67	3378.01	1.00
AI PRO R9700	llama 8B Q4_K_M	512	tg64	94.40	94.89	1.01
AI PRO R9700	llama 8B Q4_K_M	512	tg64@d4096	87.75	87.78	1.00
AI PRO R9700	llama 8B Q4_K_M	1024	pp2048	4156.83	4158.61	1.00
AI PRO R9700	llama 8B Q4_K_M	1024	pp2048@d4096	3395.09	3397.66	1.00
AI PRO R9700	llama 8B Q4_K_M	1024	tg64	94.54	94.59	1.00
AI PRO R9700	llama 8B Q4_K_M	1024	tg64@d4096	87.61	87.71	1.00
AI PRO R9700	llama 8B Q5_K_M	64	pp2048	2360.82	2360.59	1.00
AI PRO R9700	llama 8B Q5_K_M	64	pp2048@d4096	2094.59	2099.50	1.00
AI PRO R9700	llama 8B Q5_K_M	64	tg64	87.93	87.97	1.00
AI PRO R9700	llama 8B Q5_K_M	64	tg64@d4096	81.53	81.60	1.00
AI PRO R9700	llama 8B Q5_K_M	512	pp2048	4028.54	4029.13	1.00
AI PRO R9700	llama 8B Q5_K_M	512	pp2048@d4096	3330.08	3335.28	1.00
AI PRO R9700	llama 8B Q5_K_M	512	tg64	87.22	87.72	1.01
AI PRO R9700	llama 8B Q5_K_M	512	tg64@d4096	81.41	81.47	1.00
AI PRO R9700	llama 8B Q5_K_M	1024	pp2048	4083.93	4087.51	1.00
AI PRO R9700	llama 8B Q5_K_M	1024	pp2048@d4096	3320.11	3345.87	1.01
AI PRO R9700	llama 8B Q5_K_M	1024	tg64	87.14	87.50	1.00
AI PRO R9700	llama 8B Q5_K_M	1024	tg64@d4096	81.14	81.36	1.00
AI PRO R9700	llama 8B Q6_K	64	pp2048	1687.22	1688.73	1.00
AI PRO R9700	llama 8B Q6_K	64	pp2048@d4096	1537.27	1539.55	1.00
AI PRO R9700	llama 8B Q6_K	64	tg64	79.27	79.40	1.00
AI PRO R9700	llama 8B Q6_K	64	tg64@d4096	74.12	74.15	1.00
AI PRO R9700	llama 8B Q6_K	512	pp2048	2580.20	2584.98	1.00
AI PRO R9700	llama 8B Q6_K	512	pp2048@d4096	2268.64	2275.95	1.00
AI PRO R9700	llama 8B Q6_K	512	tg64	79.04	79.18	1.00
AI PRO R9700	llama 8B Q6_K	512	tg64@d4096	74.02	74.04	1.00
AI PRO R9700	llama 8B Q6_K	1024	pp2048	2603.19	2606.63	1.00
AI PRO R9700	llama 8B Q6_K	1024	pp2048@d4096	2274.46	2277.32	1.00
AI PRO R9700	llama 8B Q6_K	1024	tg64	79.00	79.16	1.00
AI PRO R9700	llama 8B Q6_K	1024	tg64@d4096	74.02	74.16	1.00
AI PRO R9700	llama 8B Q8_0	64	pp2048	2513.61	2505.78	1.00
AI PRO R9700	llama 8B Q8_0	64	pp2048@d4096	2220.37	2218.63	1.00
AI PRO R9700	llama 8B Q8_0	64	tg64	65.61	65.56	1.00
AI PRO R9700	llama 8B Q8_0	64	tg64@d4096	56.38	61.94	1.10
AI PRO R9700	llama 8B Q8_0	512	pp2048	4841.85	4834.69	1.00
AI PRO R9700	llama 8B Q8_0	512	pp2048@d4096	3878.16	3864.85	1.00
AI PRO R9700	llama 8B Q8_0	512	tg64	65.57	65.50	1.00
AI PRO R9700	llama 8B Q8_0	512	tg64@d4096	61.97	58.10	0.94
AI PRO R9700	llama 8B Q8_0	1024	pp2048	4966.90	4972.72	1.00
AI PRO R9700	llama 8B Q8_0	1024	pp2048@d4096	3917.94	3923.50	1.00
AI PRO R9700	llama 8B Q8_0	1024	tg64	65.46	65.48	1.00
AI PRO R9700	llama 8B Q8_0	1024	tg64@d4096	61.96	61.93	1.00

itterative · 2026-08-27T15:20:38Z

@IMbackK dropped rdna 4 configs and rebased on master. I do notice some regressions for tg64 on your end, I can try to adjust the script later to look into those later if you want.

IMbackK · 2026-08-29T21:42:53Z

(wait for ci)

ggerganov · 2026-08-30T10:22:56Z

(wait for ci)

Only the gpu-rocm is failing as usual (no idea why): https://github.com/ggml-org/llama.cpp/actions/runs/33087212983/job/99164631869?pr=26284

Ignore it?

IMbackK · 2026-08-30T10:34:56Z

Yeah something must be just wrong with this runner, i also cant see the logs from this runner so its useless to me:

llama.cpp/ci/run.sh

Line 52 in e422148

    
           # gpu-rocm self-hosted runner can't upload logs to blob; keep each run's logs in

.

Who is hosting this runner?

ggerganov · 2026-08-30T10:47:11Z

Yeah something must be just wrong with this runner, i also cant see the logs from this runner so its useless to me:

llama.cpp/ci/run.sh

Line 52 in e422148

# gpu-rocm self-hosted runner can't upload logs to blob; keep each run's logs in

.
Who is hosting this runner?

More info about the runner in this thread: #26544 (comment)

itterative requested a review from a team as a code owner July 29, 2026 15:55

itterative mentioned this pull request Jul 29, 2026

HIP: RDNA 4 MUL_MAT optimizations #25940

Draft

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Jul 29, 2026

itterative force-pushed the rdna-mmq-config branch from d430345 to cad1cd8 Compare July 30, 2026 20:45

IMbackK self-assigned this Aug 5, 2026

IMbackK approved these changes Aug 10, 2026

View reviewed changes

IMbackK requested changes Aug 26, 2026

View reviewed changes

itterative force-pushed the rdna-mmq-config branch from 81a7d2e to 49fd725 Compare August 27, 2026 15:19

hip: tune rdna 3 mmq config

a80abf5

itterative force-pushed the rdna-mmq-config branch from 49fd725 to a80abf5 Compare August 27, 2026 15:19

itterative changed the title ~~HIP: tune rdna 3 and rdna 4 mmq configs~~ HIP: tune rdna 3 mmq configs Aug 27, 2026

IMbackK approved these changes Aug 29, 2026

View reviewed changes

IMbackK added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 29, 2026

ggerganov merged commit 0b5be7e into ggml-org:master Aug 30, 2026
20 of 22 checks passed

ravel7524 mentioned this pull request Aug 30, 2026

CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 #24546

Open

ggerganov mentioned this pull request Sep 4, 2026

ggml : bump version to 0.23.0 ggml-org/ggml#1618

Merged

BrewTestBot mentioned this pull request Sep 4, 2026

ggml 0.23.0 Homebrew/homebrew-core#302393

Merged

1 task

fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request Sep 5, 2026

hip: tune rdna 3 mmq config (ggml-org#26284)

196d013

Uh oh!

Conversation

itterative commented Jul 29, 2026

Overview

Additional information

Requirements

Uh oh!

am17an commented Jul 29, 2026

Uh oh!

ggml-gh-bot Bot commented Jul 29, 2026

Uh oh!

itterative commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Jul 29, 2026

Uh oh!

Geramy commented Jul 29, 2026

Uh oh!

itterative commented Jul 30, 2026

Uh oh!

am17an commented Jul 30, 2026

Uh oh!

itterative commented Jul 30, 2026

Uh oh!

Geramy commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itterative commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfriehs commented Jul 30, 2026

Uh oh!

itterative commented Jul 31, 2026

Uh oh!

IMbackK commented Aug 10, 2026

Uh oh!

IMbackK left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Geramy commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Patt92 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Geramy commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IMbackK left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itterative commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IMbackK commented Aug 29, 2026

Uh oh!

ggerganov commented Aug 30, 2026

Uh oh!

IMbackK commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

itterative commented Jul 29, 2026 •

edited

Loading

Geramy commented Jul 30, 2026 •

edited

Loading

itterative commented Jul 30, 2026 •

edited

Loading

IMbackK left a comment •

edited

Loading

Geramy commented Aug 10, 2026 •

edited

Loading

Patt92 commented Aug 11, 2026 •

edited

Loading

Geramy commented Aug 11, 2026 •

edited

Loading

IMbackK left a comment •

edited

Loading

itterative commented Aug 27, 2026 •

edited

Loading

IMbackK commented Aug 30, 2026 •

edited

Loading