Persistent generation issues with MT5 models (base and fine-tuned) across environments

[Elpharran]

I'm experiencing consistent text generation failures with both pretrained google/mt5-base and custom fine-tuned MT5 models across multiple environments (local machines, Google Colab). The models produce nonsensical outputs containing <extra_id_0> and random tokens despite correct task prefixes and parameters.

Affected Models:

• google/mt5-base
• Custom MT5 variants (cointegrated/rut5-base)
• Fine-tuned for summarization task cointegrated/rut5-base

Steps to Reproduce

model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")

inputs = tokenizer(
    "translate English to Russian: Hello world!", 
    return_tensors="pt"
)

output = model.generate(
    inputs.input_ids, 
    max_new_tokens=50,
    num_beams=5,
    early_stopping=True
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Expected Behavior

Expected Russian translation: "Привет, мир!"
Actual output: <extra_id_0> Hello world! or similar garbage

Environment

• Transformers 4.50.0 (also checked 4.48.3 and 4.30.0)
• PyTorch 2.0.1+cu118
• Python 3.10.12
• Both CPU and CUDA environments affected
• Reproducible in Google Colab (T4 GPU)

Additional Context

• Issue persists across multiple task formats (translation, summarization)
• Verified correct model loading: model.config shows expected architecture
• Tokenization appears correct when inspected:

# Output: ['▁translate', '▁English', '▁to', '▁Russian', ':', '▁Hello', '▁world', '!']

• Tried multiple generation strategies (greedy, beam, sampling) without success

I am happy to provide additional code and information.

[Rocketknight1]

Hi @Elpharran, can you try some other sample inputs/languages? If it only occurs with English->Russian, it might just be the model getting confused; mT5 is very old by now, and the quality will often be much worse than a modern model.

[Elpharran]

Hi @Rocketknight1! I tried a lot of different inputs, including simple "Hello". My concern is that model does not generate a real answer, instead producing special tokens and repeating my query.
By the way same behaviour you can observe in inference window in model card.

And I forgot to mention the fact that 1.5 weeks ago everything was just fine :(

[Rocketknight1]

Oh! That last detail is extremely interesting - can you try:

1. Installing older versions of transformers (e.g. pip install transformers==4.48)
2. Checking the model repo to see if anything changed in that time

If you can identify an older version where it works correctly, then that makes it much more likely that this is a bug, and we can start to identify the cause

[Elpharran]

• I checked: there were no changes in model repos.

• Then I checked whether google/mt5base model can perform simple translation tasks bc of your previous concern that task is too complex for the model, and yes - in the original T5 paper, the models are pre-trained on a diverse dataset called C4, and they are trained with specific prefixes indicating the task, like "translate English to German: ...".
  I tried input 'translate English to German: Hello World' with 4.48.3 transformers version in Google Colab, the output is "<extra_id_0> English (US)".

And this is what I did step by step before encountering generation issues:

1. About 2 weeks ago I chose https://huggingface.co/cointegrated/rut5-base model, based on google/mt5base (key difference - only 10k English and top-20k Russian embeddings left) for my task and installed 4.50.0 version of tramsformers.
2. I fine-tuned the model on small data (10k train) for headlines generation task just to test how it will work with my prefix. It worked pretty well - produced outputs like expected: poor summarization quality, but stable - almost always it repeated the first sentence of the text and it's fine for news and such small dataset.
3. After that, 2 days before opening this issue, I modified my script to work with accelerate for distributed training and reproduced step 2 just to be sure. I got garbage as an output: answering for all texts either with the same sentence or with special tokens like <extra_id0> mixed with random texts.
4. I figured that there could be an error in my code, reran my inintial training script in the clean env on single GPU and got the same result. From that moment I could not reproduce my first result at all. I tried different versions of transformers, different VM-s, training on cpu, but result was the same.
5. I went to check rut5-base model and google orig model and got garbage output as well. Consulted with colleagues and opened this issue.

[Rocketknight1]

Hmm, that's very strange! The fact that it worked in the past, but not now, and yet can't be fixed by installing an older version of transformers suggests that it's probably not a bug that was introduced in a recent version, and that makes it hard for us to know what to do with this issue! It may be that the model or training setup never really worked that well, but whatever version you trained the first time just "got lucky" and output the first sentence rather than control tokens. We'll need you to diagnose the exact circumstances this issue occurs before we can try to fix it.

Also, I know I've mentioned this before, but if you're interested in translating or summarizing text for practical purposes, rather than just experimentation or learning, I would highly recommend a modern LLM instead of mT5. I'm sure if you ask Gemma-3 to do this task it'll perform amazingly, even with no extra training at all.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[NatalieAlexander]

Hi has there been a solution to this?