[ vllm x v5] nit

[ArthurZucker]

What does this PR do?

Removed the tokenizer_class attr was never there to begin with, and kwargs are now supported.
This was failing some test on vllm ci. Fixes https://buildkite.com/vllm/ci/builds/57601/steps/canvas?sid=019d1aec-aa5a-41db-bac6-4f42397279d5

In [6]: pip = pipeline("text-generation", "XiaomiMiMo/MiMo-7B-Base")
The repository XiaomiMiMo/MiMo-7B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base .
 You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base.
rue`.an avoid this prompt in future by passing the argument `trust_remote_code=T 

Do you wish to run the custom code? [y/N] y
The repository XiaomiMiMo/MiMo-7B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base .
 You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base.
rue`.an avoid this prompt in future by passing the argument `trust_remote_code=T 

Do you wish to run the custom code? [y/N] y
The repository XiaomiMiMo/MiMo-7B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base .
 You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base.
rue`.an avoid this prompt in future by passing the argument `trust_remote_code=T 

Do you wish to run the custom code? [y/N] y
modeling_mimo.py: 3.53kB [00:00, 2.34MB/s]
A new version of the following files was downloaded from https://huggingface.co/XiaomiMiMo/MiMo-7B-Base:
- modeling_mimo.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
model.safetensors.index.json: 37.0kB [00:00, 16.4MB/s]
Fetching 4 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [01:58<00:00, 29.64s/it]
Download complete: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.7G/15.7G [01:58<00:00, 132MB/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 451/451 [00:00<00:00, 6320.50it/s]
generation_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 138/138 [00:00<00:00, 582kB/s]
The repository XiaomiMiMo/MiMo-7B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base .
 You can inspect the repository content at https://hf.co/XiaomiMiMo/MiMo-7B-Base.
rue`.an avoid this prompt in future by passing the argument `trust_remote_code=T 

Do you wish to run the custom code? [y/N] y
tokenizer_config.json: 7.23kB [00:00, 6.69MB/s]
vocab.json: 2.78MB [00:00, 11.2MB/s]
merges.txt: 1.67MB [00:00, 12.6MB/s]
tokenizer.json: 7.03MB [00:00, 21.2MB/s]

In [7]: pip("Hey how are you?")
Both `max_new_tokens` (=256) and `max_length`(=20) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Out[7]: [{'generated_text': "Hey how are you? I'm in the same boat as you. I'm 18 years old and I'm currently a sophomore in college. I've been struggling with depression for about 5 years now and it's getting worse. I've been on medication for a long time to help me cope, but I still struggle. I'm not sure if I'm going to make it through college. I've been thinking about suicide a lot lately. I don't know if I can do it. I don't know if I'm strong enough. I'm just lost. I don't know what to do. I'm sorry for being a burden to you. I hope you can find some relief."}]

checked and this works expectedly.

config = AutoConfig.from_pretrained("XiaomiMiMo/MiMo-7B-Base", trust_remote_code=True)

[hmellor]

Should we also remove this from MT5 and UMT5 then?

[zucchini-nlp]

Nice, iirc we don't assume it exists in codebase so lgtm

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: mt5, umt5

[Copilot]

Pull request overview

This PR removes the tokenizer_class attribute from the base PreTrainedConfig and from the MT5/UMT5 config classes, aligning tokenizer selection away from config-level defaults and toward tokenizer config/mappings (as referenced by the vLLM CI failure in the PR description).

Changes:

• Remove tokenizer_class from MT5Config and UMT5Config (docstring + class attribute).
• Remove the tokenizer_class field (and its import dependency) from PreTrainedConfig.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/transformers/models/umt5/configuration_umt5.py	Drops tokenizer_class from UMT5 config surface and docs.
src/transformers/models/mt5/configuration_mt5.py	Drops tokenizer_class from MT5 config surface and docs.
src/transformers/configuration_utils.py	Removes the base PreTrainedConfig.tokenizer_class field and the now-unused tokenizer import.

[Copilot]

Removing tokenizer_class from PreTrainedConfig will break existing tests/utilities that treat it as a common config kwarg/field (e.g. tests/utils/test_configuration_utils.py::test_config_common_kwargs_is_complete expects tokenizer_class to be present in PreTrainedConfig().__dict__). Please update the corresponding test expectations (and any shared config_common_kwargs/common-config logic) to reflect the new base config surface, or keep a backward-compatible tokenizer_class field if it’s still considered part of the common config contract.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44971&sha=ece5f2