`tie_word_embeddings` not saved on customized model

[Tavish9]

System Info

transformers_version: 4.52.0.dev0
python: 3.10

Who can help?

@ArthurZucker @SunMarc @qubvel @zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoConfig, AutoModelForCausalLM, PretrainedConfig, PreTrainedModel, Trainer, TrainingArguments


class MyConfig(PretrainedConfig):
    model_type = "test"
    sub_configs = {"text_config": AutoConfig}

    def __init__(
        self,
        text_config=None,
        **kwargs,
    ):
        self.text_config = text_config

        super().__init__(**kwargs)

class MyModel(PreTrainedModel):
    config_class = MyConfig

    def __init__(self, config, language_model=None) -> None:
        super().__init__(config)
        self.language_model = language_model

        self.post_init()

text_config = AutoConfig.from_pretrained("Qwen/Qwen3-8B")
config = MyConfig(text_config=text_config)
language_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = MyModel(config, language_model)

trainer = Trainer(
    model=model,
    args=TrainingArguments(),
)

trainer.save_model()

Expected behavior

The saved config.json has no tie_word_embeddings key

Expected:

{
  "architectures": [
    "MyModel"
  ],
  "model_type": "test",
  "text_config": {
    "_name_or_path": "Qwen/Qwen3-8B",
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 1024,
    "initializer_range": 0.02,
    "intermediate_size": 3072,
    "max_position_embeddings": 40960,
    "max_window_layers": 28,
    "model_type": "qwen3",
    "num_attention_heads": 16,
    "num_hidden_layers": 28,
    "num_key_value_heads": 8,
    "rms_norm_eps": 1e-06,
    "rope_scaling": null,
    "rope_theta": 1000000,
    "sliding_window": null,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936
  },
  "torch_dtype": "float32",
  "transformers_version": "4.52.0.dev0"
}

But:

{
  "architectures": [
    "MyModel"
  ],
  "model_type": "test",
  "text_config": {
    "_name_or_path": "Qwen/Qwen3-8B",
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "initializer_range": 0.02,
    "intermediate_size": 12288,
    "max_position_embeddings": 40960,
    "max_window_layers": 36,
    "model_type": "qwen3",
    "num_attention_heads": 32,
    "num_hidden_layers": 36,
    "num_key_value_heads": 8,
    "rms_norm_eps": 1e-06,
    "rope_scaling": null,
    "rope_theta": 1000000,
    "sliding_window": null,
-   "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936
  },
  "torch_dtype": "float32",
  "transformers_version": "4.52.0.dev0"
}

[Tavish9]

More info:

If we use Qwen/Qwen3-0.6B or Qwen/Qwen3-1.7B (that has tie_word_embeddings:True in config.json), the saved config has tie_word_embeddings, which is the same as original config.

[Rocketknight1]

Hi @Tavish9, I don't think this is a bug! The base PretrainedConfig has a number of default config keys, including tie_word_embeddings, and when you inherit from it you'll need to set these in your __init__ if you don't want them to be set to the default value. Take a look at transformers/configuration_utils.py.

[Tavish9]

Hi, @Rocketknight1 thanks for replying.

However, you misunderstand my post. I don't change the original text config at all, but tie_word_embeddings is missing in the config.json file for text_config.

text_config = AutoConfig.from_pretrained("Qwen/Qwen3-8B")
config = MyConfig(text_config=text_config)
model = MyModel(config, language_model)
trainer.save_model()

Actually, we don't need to focus config.tie_word_embeddings, but config.text_config.tie_word_embeddings.

This happens for config that has tie_word_embeddings:True on the original config.

For MyConfig, it's tie_word_embeddings is True, which is the default value, I know this.

[zucchini-nlp]

@Tavish9 hey, in composite configs we first compare if the defaults are different from the model's config, which will be Qwen3Config class. You can see that the config has a default value False and thus if tie_word_embeddings=False it will not be saved

As long as the loaded config has onfig.text_config.tie_word_embeddings = False it is expected. We had similar issues earlier and IIRC now it should not have weird edge cases

[Tavish9]

@zucchini-nlp , thanks for replying.

1. Could you please show me the saving logic?

in composite configs we first compare if the defaults are different from the model's config

2. Besides, do you mean that even the config.json does not save text_config.tie_word_embeddings, the loaded config should have config.text_config.tie_word_embeddings=False?

3. However, there might be a potential issue with config.tie_word_embeddings, as AutoConfig sets tie_word_embeddings to true if tie_word_embeddings is missing in config.tie_word_embeddings within config.json.
   So, we might get loaded config: config.tie_word_embeddings=True while config.text_config.tie_word_embeddings=False.

Very Detailed Questions, thanks in advance.

[zucchini-nlp]

@Tavish9 The snippet above for saving is correct. Since you are using a custom MyConfig the issue comes from the config definition itself. We usually do not simply assign the text config as is, but rather at load time initiate an LM config as follows. Thus all missing will default to model_type's defaults. In this case it is Qwen3. The saving and loading back a composite config, will give config.text_config.tie_word_embeddings=False

transformers/src/transformers/models/llava/configuration_llava.py

Lines 127 to 131 in 04282a9

	if isinstance(text_config, dict):
	text_config["model_type"] = text_config["model_type"] if "model_type" in text_config else "llama"
	text_config = CONFIG_MAPPING[text_config["model_type"]](**text_config)
	elif text_config is None:
	text_config = CONFIG_MAPPING["llama"]()

[Tavish9]

okay, that's aligned with my experiment.

what's about the Q3? There is a misalignment between config.tie_word_embeddings and config.text_config.tie_word_embeddings if tie_word_embeddings is missing and the model_type's default is False.

I think it's a bug 🤔

[zucchini-nlp]

We should not have config.tie_word_embeddings and all text model related keys should be available only in config.text_config. For some older models we have missed that and probably configs on repo contain duplicated tie_word_embeddings, for new models we don't allow it anymore

[Tavish9]

transformers/src/transformers/configuration_utils.py

Lines 223 to 225 in 04282a9

	self.tie_word_embeddings = kwargs.pop(
	"tie_word_embeddings", True
	) # Whether input and output word embeddings should be tied for all MLM, LM and Seq2Seq models.

So, config.tie_word_embeddings is valid and would cause an issue in this case.

[zucchini-nlp]

For composite configs, we don't use config.tie_word_embeddings but rather config.get_text_config().tie_word_embeddings, so yes. If config has it defined in the upper level, then we'll ignore it and use whatever is found in text config

[Tavish9]

Okay, things solved.

BTW, composite config is a new feature of transformers update. Some old models might not have composite configs so they still use config.tie_word_embeddings.

[zucchini-nlp]

If the model config is flat, i.e. no text_config exists then config.get_text_config() returns self. We didn't change anything for usual configs and only added a standardized way to handle composite ones :)

Feel free to close the issue, if it's resolved