System info
- transformers version: 4.x (any release before v5.5.0)
- Platform: macOS (Apple Silicon) / Linux
- Python version: 3.12 / 3.13
- Model: google/gemma-4-E4B-it (requires transformers >= 5.5.0)
Background
google/gemma-4-E4B-it uses the gemma4 architecture, which was introduced in transformers v5.5.0. It cannot and should not load on v4. However, the error a user actually sees on v4 is deeply misleading and sends them down the wrong debugging path entirely.
Steps to reproduce
pip install transformers # installs v4.x by default
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")Expected error (clear and actionable)
ValueError: google/gemma-4-E4B-it requires transformers >= 5.5.0.
Please upgrade: pip install "transformers>=5.5.0"
Actual error (cryptic and misleading)
File .../transformers/tokenization_utils_base.py:1181
self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
AttributeError: 'list' object has no attribute 'keys'
Why this is a problem
The actual error gives users no indication that the real problem is a version mismatch. Instead it points to internal tokenizer code, causing users to:
- Spend time patching
tokenizer_config.json files manually
- Try reinstalling
tokenizers and transformers in different combinations
- Assume it is a bug in the model's config rather than a version requirement
- File issues against the wrong repo (model page, mlx-lm, etc.)
This is a developer experience issue — the underlying incompatibility is expected, but the error message is not.
Root cause
The tokenizer_config.json for gemma4 uses the v5-style extra_special_tokens list format:
"extra_special_tokens": ["<token_a>", "<token_b>"]
When loaded by transformers v4, _set_model_specific_special_tokens() calls .keys() on this list and crashes — before the code ever gets to check whether the gemma4 architecture is supported. The version check happens too late.
Suggested fix
Two possible approaches:
Option A — Add a guard in _set_model_specific_special_tokens() so the list format fails gracefully with a helpful message:
def _set_model_specific_special_tokens(self, special_tokens):
if isinstance(special_tokens, list):
raise ValueError(
"This model's tokenizer config uses a format introduced in transformers v5. "
"Please upgrade: pip install 'transformers>=5.0.0'"
)
...Option B — Move the architecture/version check earlier in the loading pipeline, before tokenizer config parsing, so the user sees a version error first.
System info
Background
google/gemma-4-E4B-ituses thegemma4architecture, which was introduced in transformers v5.5.0. It cannot and should not load on v4. However, the error a user actually sees on v4 is deeply misleading and sends them down the wrong debugging path entirely.Steps to reproduce
pip install transformers # installs v4.x by defaultExpected error (clear and actionable)
Actual error (cryptic and misleading)
Why this is a problem
The actual error gives users no indication that the real problem is a version mismatch. Instead it points to internal tokenizer code, causing users to:
tokenizer_config.jsonfiles manuallytokenizersandtransformersin different combinationsThis is a developer experience issue — the underlying incompatibility is expected, but the error message is not.
Root cause
The
tokenizer_config.jsonfor gemma4 uses the v5-styleextra_special_tokenslist format:When loaded by transformers v4,
_set_model_specific_special_tokens()calls.keys()on this list and crashes — before the code ever gets to check whether thegemma4architecture is supported. The version check happens too late.Suggested fix
Two possible approaches:
Option A — Add a guard in
_set_model_specific_special_tokens()so the list format fails gracefully with a helpful message:Option B — Move the architecture/version check earlier in the loading pipeline, before tokenizer config parsing, so the user sees a version error first.