Question: Low generation throughput on Mac Studio

[teddythinh]

Hi team. First of all, thank you for building and maintaining this great software. I’m new to oMLX and I have a question regarding performance when running models on a Mac Studio. I’m currently testing the following two models from the mlx-community repository:

• Qwen3.6-35B-A3B-4bit
• gemma-4-26b-a4b-it-4bit

I ran the throughput benchmark, but the generation performance seems quite slow. In my tests, the generation throughput is only around 10–12 tokens/s. I'd like to understand whether this is expected for my hardware/model configuration, or if there may be some settings or optimizations I should check.

Benchmark of Qwen 3.6

Single Request Results

Test	TTFT (ms)	TPOT (ms)	ppTPS	tgTPS	E2E (s)	Throughput	Peak Mem
pp 4096 / tg 128	19,427.2	95.7	210.8	10.5	31.6	133.7	20.0 GB
pp 16384 / tg 128	107,023.3	119.3	153.1	8.4	122.2	135.2	21.0 GB

Batch Results

Batch	tgTPS	ppTPS	Avg TTFT (ms)	E2E (s)	Speedup
1x baseline	10.5	210.8	19,427.2	31.6	1.00x
2x	8.9	135.0	15,171.9	43.8	0.85x
4x	14.5	155.4	25,469.1	61.5	1.38x

Benchmark of Gemma 4

Single Request Results

Test	TTFT (ms)	TPOT (ms)	ppTPS	tgTPS	E2E (s)	Throughput	Peak Mem
pp 4096 / tg 128	5,937.4	91.8	689.9	11.0	17.6	240.0	14.4 GB
pp 16384 / tg 128	120,399.7	117.1	136.1	8.6	135.3	122.1	16.7 GB

Batch Results

Batch	tgTPS	ppTPS	Avg TTFT (ms)	E2E (s)	Speedup
1x baseline	11.0	689.9	5,937.4	17.6	1.00x
2x	14.7	131.8	15,537.7	33.0	1.34x
4x	18.8	130.5	30,287.3	58.6	1.71x

Screenshot

Additional Info

• macOS Version: 26.5 (25F71)
• oMLX Version: 0.4.4 (Build 1692)
• Device: Mac Studio 2025
• Processor: Apple M3 Ultra
• Memory: 256 GB

[ilukashin]

i'm not sure, but it looks strange for me:
hot cache only = true
but hot cache size = 0

it means you don't use any memory cache. And this can affect on large context requests.

try to set your hot cache size > 0, for example 5-10% of your memory.

[teddythinh]

Thanks for your notice. I have turned it off.

[teddythinh]

One thing I notice is that the Thermal State is in a critical state that may drastically reduce performance.

[teddythinh]

I investigated the issue, and the root cause is that my Mac is overheating, which triggers thermal throttling. After letting it cool down, the performance improved significantly.

[StevePierce]

You might want to consider software like Macs Fan Control to set a more aggressive fan curve

[teddythinh]

You might want to consider software like Macs Fan Control to set a more aggressive fan curve

Yes, I used that one, thanks @StevePierce! 😃