Tokenizer loading fails for models with v5-style `extra_special_tokens`

[DePasqualeOrg]

$ mlx_lm.generate --model mlx-community/Ministral-3-3B-Reasoning-2512-4bit --prompt "Hello"

AttributeError: 'list' object has no attribute 'keys'

The model's tokenizer_config.json has extra_special_tokens as a list, but transformers v4.x expects a dict. transformers v5 changes this field to a list, and some models already use the new format.

Upgrading to transformers v5 (which requires some migration work) would resolve this issue.

[awni]

I'm not able to reproduce this locally with:

pip show transformers
Name: transformers
Version: 5.0.0.dev0

Can you share more details on how to reproduce it?

[DePasqualeOrg]

Yes, this works with transformers v5, but not with the current version used by mlx-lm. I wanted to flag this, since models like mlx-community/Ministral-3-3B-Reasoning-2512-4bit are not working out of the box with mlx-lm at the moment.

[awni]

I see. In general new models often require new versions of Transformers because of their tokenizer. We don't try to back-support the tokenizer in mlx-lm. We should pin to the minimum transformers.. in this case it's not quite out yet on PyPi so it's a bit tricky. But in general I think that would be a good practice.

[ormandj]

Ran into this with mlx-community/Devstral-Small-2-24B-Instruct-2512-4bit if you need another reproducer, I was attempting to use mlx_lm.chat --model "mlx-community/Devstral-Small-2-24B-Instruct-2512-4bit". Let me know if you need any additional detail, and thank you.

[hqv1]

FYI, mlx-community/Devstral-Small-2-24B-Instruct-2512-8bit works in mlx-vlm and I'm assuming your model would work too.

[awni]

FYI an easy fix for this is to upgrade transformers. I will pin mlx-lm to transformers v5 once it's fully released. In the meantime closing this as resolved.

pip install transformers==5.0.0rc1