Hybrid Mamba-2 models (Falcon-H1, Granite-4.0-H, Nemotron-3 Nano) on an M4 Max: mlx-lm 0.31.3 vs llama.cpp b8680, measured

[john-rocky]

Thanks for landing the hybrid architectures (falcon_h1, granitemoehybrid, nemotron_h) in mlx-lm — being able to run all three families from one mlx_lm generate is what made this comparison possible. I measured the same models through mlx-lm and llama.cpp on one machine (M4 Max, 128 GB, macOS 27.0), batch 1, greedy, so anyone looking for hybrid Mamba-2 throughput on Apple Silicon has a same-model comparison in one place. Where Apple's own runtime had a measured artifact I added that column too.

Decode tok/s, batch 1, greedy, warm (MLX = median of three processes; llama.cpp = llama-bench mean ± std, -r 3):

Model	mlx-lm 0.31.3 / mlx 0.32.2 (4-bit)	llama.cpp Metal b8680 (Q4_K_M)	Apple Core AI (int8 / fp16 export)
Falcon-H1 1.5B-Instruct	300.0 (mlx-community 4-bit, 0.88 GB) †	148.6 ± 0.3 (tiiuae, 0.90 GB)	—
Falcon-H1 3B-Instruct	167.9 (4-bit, converted from tiiuae bf16 with 0.31.3) †	86.1 ± 0.4 (Q4_K_M converted from the same bf16 with b8680's converter, 1.76 GiB)	—
Falcon-H1 7B-Instruct	(not measured)	52.9 ± 0.4 (tiiuae, 4.28 GiB)	—
Granite-4.0-H 350M	521.7 (mlx-community, 0.20 GB)	282.1 ± 1.0 (ibm-granite, 0.21 GB)	191.1 (fp16)
Granite-4.0-H 1B	275.8 (mlx-community, 0.83 GB)	144.0 ± 2.0 (ibm-granite, 0.86 GB)	136.5 (int8)
Granite-4.0-H Tiny 7B-A1B	202.2 (lmstudio-community, 3.91 GB)	117.3 ± 0.9 (unsloth, 4.25 GB)	—
Nemotron-3 Nano 4B	176.8 (mlx-community, 2.24 GB)	88.4 ± 0.2 (nvidia, 2.84 GB)	85.2 (int8 head)
Nemotron-3 Nano 30B-A3B	159.7 (mlx-community, 17.8 GB)	86.2 ± 0.5 (unsloth, 24.6 GB)	—

Prompt side: llama.cpp pp512 runs 687 (Falcon 7B) to 8,496 tok/s (Granite 350M); the MLX prompt column is a 132-145 token chat prompt, so I left it out of the table rather than compare it with pp512 (it is in the linked file). Metal peak allocation for MLX vs max RSS for llama.cpp is also in the file; the two are not the same metric.

Three things to keep in mind when reading across columns. Q4_K_M and mlx 4-bit gs64 affine do not move the same bytes per token (about 4.8 vs 4.5 bits per weight), and the unsloth 30B-A3B GGUF is closer to 6.2. The Core AI column is int8 or fp16 decode-only exports measured earlier on the same Mac, so it answers "what does Apple's runtime do on this architecture", not an equal-quant race. And MLX numbers are mlx_lm generate's own Generation: figure on 256 tokens, while llama.cpp's is tg256 from llama-bench.

† One observation on Falcon-H1 at 4-bit, one prompt, greedy, not a quality evaluation: mlx-community/Falcon-H1-1.5B-Instruct-4bit (the re-upload after #504) is fluent for two sentences and then repeats itself on a 130-token technical prompt, and writes "Bubble Track Algorithm" on the bubble-sort prompt from #504. Converting tiiuae's bf16 myself with 0.31.3 (-q --q-bits 4 --q-group-size 64) reproduces that text word for word, so the checkpoint is a current conversion and the effect sits in the 4-bit gs64 affine path. The same weights at 8-bit (204.9 tok/s) and bf16 (128.1 tok/s) do not repeat; the 3B behaves the same way (bf16 65.1 and 8-bit 109.4 tok/s coherent, 4-bit 167.9 tok/s confused). llama.cpp with Q4_K_M of the same weights answers both sizes cleanly, which I mention as a data point, not a diagnosis: I did not look at template handling or numerics, and I did not try group size 32, mxfp4 or DWQ. If one of those is the recommended recipe for this family, I would be glad to re-run the rows with it. The mlx-community/Falcon-H1-3B-Instruct-4bit card still says it was converted with 0.25.2 (before #231 was complete); I could not verify its output today.

Commands, revisions, raw outputs and the coherence samples: https://github.com/john-rocky/apple-silicon-llm-bench/blob/main/results/hybrid/hybrid-apple-silicon-bench-2.md (Falcon-H1 / Granite small sizes, today) and https://github.com/john-rocky/apple-silicon-llm-bench/blob/main/results/hybrid/nemotron3-nano-apple-silicon-bench.md (Nemotron and Granite Tiny, last week). Happy to re-run any row with a different quant, prompt length or version if that is useful.