Model support: IBM Granite-4.0-H-Micro

[reneleonhardt]

🚀 The feature, motivation and pitch

IBM is at the forefront of the most efficient LLMs on edge devices, for example Granite-4.0-H-Micro.

Could Granite models be supported for responsible, high throughput and quality with extremely low memory consumption?

Alternatives

Qwen3-Next achieves many similar breakthroughs, but no small model yet.
https://qwen3-next.com/

Additional context

https://www.storagereview.com/news/ibm-granite-4-0-efficient-open-weight-llms-target-enterprise-grade-performance-cost-and-governance

RFC (Optional)

No response

[mergennachin]

A few weeks ago, we tried Granite 3.3 and it was exportable to ExecuTorch via optimum-executorch

https://github.com/huggingface/optimum-executorch?tab=readme-ov-file

huggingface/optimum-executorch#144

4.0-micro is definitely interesting -- thanks for the suggestion

cc @jackzhxng

[reneleonhardt]

Good to see some similar support somewhere!

Please note that Granite-4.0-Micro (consumes 9GB on a RTX 3060 12GB) is a dense model.
The innovation about Granite 4 are the hybrid models using Mamba-2 like Granite-4.0-H-Micro (consumes 4GB on a Raspberry-Pi 8GB) with much lower memory consumption.

https://huggingface.co/ibm-granite/granite-4.0-h-micro
https://huggingface.co/mlx-community/granite-4.0-h-micro-4bit

$ mlx_lm.benchmark --model mlx-community/granite-4.0-h-micro-4bit -p 2048 -g 128
Averages: prompt_tps=854.521, generation_tps=89.960, peak_memory=4.744