Support google/gemma-4-12B-it (Gemma 4 Unified).
https://huggingface.co/google/gemma-4-12B-it
Why not supported: new arch Gemma4UnifiedForConditionalGeneration /
model_type: gemma4_unified is unregistered. It's encoder-free — raw
image
patches & audio waveforms project directly into the decoder, so the existing
vision_tower/audio_tower path and weight loader don't apply.
Key points:
- Dense decoder, 48 layers, hidden 3840 (reuse
Gemma4TextModel)
- No PLE; dual attention (sliding 1024 + full every 6th,
global_head_dim:512)
- Vision embedder: 48×48 patch → matmul + coord posemb → pool → 280 soft
tokens
- Audio embedder: 16kHz/40ms frames → linear projection
- Needs transformers ≥ 5.10.0.dev0 for the config classes
Support
google/gemma-4-12B-it(Gemma 4 Unified).https://huggingface.co/google/gemma-4-12B-it
Why not supported: new arch
Gemma4UnifiedForConditionalGeneration/model_type: gemma4_unifiedis unregistered. It's encoder-free — rawimage
patches & audio waveforms project directly into the decoder, so the existing
vision_tower/audio_towerpath and weight loader don't apply.Key points:
Gemma4TextModel)global_head_dim:512)tokens