`dtype` is silently ignored when loading composite (multimodal) checkpoints through AutoModelForCausalLM (affects all Qwen3.5 repos)

[qflen]

System Info

• transformers version: 5.10.0.dev0 (main @ effde20); also reproduced on the v5.9.0 release
• Platform: macOS-26.1-arm64-arm-64bit (Apple Silicon)
• Python version: 3.11.15
• PyTorch version (GPU?): 2.11.0 (False; MPS available, repro runs on CPU)
• Huggingface_hub version: 1.14.0
• Safetensors version: 0.7.0
• Accelerate version: 1.13.0

Who can help?

@Cyrilvallez @zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Reproduction

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.bfloat16, expected torch.float32

Same result with dtype="float32", with or without device_map, and with dtype=torch.float64. The concrete class honors the same argument, so the bug is specific to the Auto path:

from transformers import Qwen3_5ForCausalLM

model = Qwen3_5ForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.float32, correct

Fully offline reproduction (no Hub access, also a regression-test candidate):

import torch
from transformers import AutoModelForCausalLM, Qwen3_5Config, Qwen3_5ForConditionalGeneration

config = Qwen3_5Config(
    text_config=dict(
        vocab_size=64, hidden_size=32, intermediate_size=64, num_hidden_layers=2,
        num_attention_heads=2, num_key_value_heads=1, head_dim=16,
        linear_conv_kernel_dim=2, linear_key_head_dim=8, linear_value_head_dim=8,
        linear_num_key_heads=1, linear_num_value_heads=2, max_position_embeddings=128,
        layer_types=["linear_attention", "full_attention"],
        rope_parameters={"rope_type": "default", "rope_theta": 10000.0,
                         "partial_rotary_factor": 1.0, "mrope_section": [3, 3, 2]},
    ),
    vision_config=dict(depth=2, hidden_size=32, out_hidden_size=32, intermediate_size=64,
                       num_heads=2, patch_size=4, temporal_patch_size=2, spatial_merge_size=1),
)
Qwen3_5ForConditionalGeneration(config).to(torch.bfloat16).save_pretrained("/tmp/tiny_composite")

model = AutoModelForCausalLM.from_pretrained("/tmp/tiny_composite", dtype=torch.float32)
print(type(model).__name__, next(model.parameters()).dtype)
# Qwen3_5ForCausalLM torch.bfloat16, expected torch.float32

Root cause (line numbers as of effde20)

1. _BaseAutoModelClass.from_pretrained pops dtype from kwargs only when it is the string "auto" (src/transformers/models/auto/auto_factory.py:330-333). A concrete torch.dtype or dtype string stays in kwargs.

2. AutoConfig.from_pretrained(..., return_unused_kwargs=True, **kwargs) (auto_factory.py:338) then absorbs dtype into the loaded composite config's top level via standard config-kwarg handling, and it is no longer in the returned unused kwargs:

   from transformers import AutoConfig
   cfg, unused = AutoConfig.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32, return_unused_kwargs=True)
   print(type(cfg).__name__, cfg.dtype, cfg.text_config.dtype, "dtype" in unused)
   # Qwen3_5Config torch.float32 torch.bfloat16 False

3. Because the resolved model class is text-only, model_class.config_class matches the composite's text_config, so the config is swapped: config = config.get_text_config() (auto_factory.py:394-398, same pattern in from_config around lines 240-244). The top-level dtype, which now holds the user's value, is dropped here. Note this block already propagates quantization_config from the parent config for exactly this reason (Fix: propagate quantization_config to text sub-config for composite models in AutoModelForCausalLM #45494), but not dtype.

4. model_class.from_pretrained(...) is then called with no dtype in kwargs, so it falls back to its default "auto", and _get_dtype (src/transformers/modeling_utils.py:837-838) resolves it from the extracted sub-config (text_config.dtype: bfloat16 in all Qwen3.5 checkpoints) or, when that is unset, from the checkpoint weights. The user's request is silently gone.

Impact

• All Qwen/Qwen3.5-* repos ship architectures: ["Qwen3_5ForConditionalGeneration"] (MoE: Qwen3_5MoeForConditionalGeneration) with text_config.dtype: bfloat16 (verified on 0.8B/2B/4B/9B/122B-A10B), so every AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-*", dtype=...) ignores the requested dtype.
• The failure is silent: no warning, and users believe they are running fp32/fp16. As the offline repro shows, it does not even require dtype in the sub-config json (it then falls back to the weights dtype), so it affects any composite checkpoint loaded through an Auto class whose resolved class's config_class equals sub_configs["text_config"].
• Concrete fallout: in Qwen3.5 GatedDeltaNet: Large logit divergence between full-sequence forward and prefill+decode with cache #46190 the reporter's float32 control run silently executed in bfloat16, which made expected bf16 rounding look like a large correctness bug in GatedDeltaNet.

Expected behavior

dtype=torch.float32 through the Auto classes should produce fp32 parameters, identical to loading through the concrete class.

Possible fixes

1. Mirror Fix: propagate quantization_config to text sub-config for composite models in AutoModelForCausalLM #45494: in both get_text_config() extraction sites, carry the parent config's dtype over to the extracted text config when it is not None.
2. Alternatively, pop a user-provided dtype before the AutoConfig.from_pretrained call (extending the existing "auto" special case at auto_factory.py:330-333) and pass it explicitly to model_class.from_pretrained, so the user value never round-trips through the config object.

I am happy to open a PR with either approach plus an offline regression test based on the snippet above, lmk.

[Achyuthan-S]

Hi, I'd like to take a stab at this. Planning to go with Option 1 (propagating dtype at the get_text_config() extraction sites, mirroring #45494). Will open a PR shortly.

[qflen]

Hi @Achyuthan-S, please read last sentence of issue body ~

Relevant:
https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md
https://github.com/huggingface/transformers/blob/main/.ai/AGENTS.md#coordination-before-coding

[Achyuthan-S]

Apologies for jumping ahead without coordinating — I missed the 'lmk' at the end. I've already implemented the fix (Option 1, mirroring #45494) with a passing regression test. Happy to share the patch here for review before opening a PR, or defer to you if you'd prefer to take it.

[zucchini-nlp]

Ohhh, loading a multimodal model as CausalLM, nice 🫠 (never liked this idea lol)

To start with, we recently deprecated different dtype per backbone as it can lead to unexpected dtype mismatches. So when loading the whole VLM, we'd propagate fp32 in model init step and raise a deprecation warning. Loading a backbone-only is the problem here

IMO the second option makes more sense, so we keep the config's values by default and overwrite it only when user provides a different dtype @qflen ping me when you get a PR up :)