Infinite repetition bug still exists

[99991]

In some cases, Qwen3-VL and Qwen2.5-VL become stuck in an infinite loop, repeating the same text over and over. Increasing the repetition penalty is not an acceptable solution because it breaks the transcription of naturally repetitive text, for example in tables.

Output

```markdown
# NLP Pipeline

In the following, the NLP pipeline and its components are discussed in depth:

```
Text
  |
  v
Language identification
  |
  v
String -> Tokenization -> Sentence splitter -> POS tagger
  |                     |
  |                     v
  |                Sentences
  |                Tokens
  |
  |-----------------|
  | Segmentation    |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |                 |
  |

Code

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
import torch
torch.manual_seed(0)

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-VL-4B-Instruct",
    dtype="float16",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")

# Download image from github and rename to test.png
image_path = "test.png"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image_path,
            },
            {
                "type": "text",
                "text": "Transcribe this image as markdown.",
            },
        ],
    }
]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print(output_text[0])

test.png

Related

Previous issue: #241 (closed without comment)

[attashe]

Same issue

[ucaslcl]

Please use “torch_dtype=torch.bfloat16“ to load model, try again~

[99991]

Please use “torch_dtype=torch.bfloat16“ to load model, try again~

I added torch_dtype=torch.bfloat16 to the Qwen3VLForConditionalGeneration.from_pretrained, but still get the infinite repeat bug (code below).

I also get the warning torch_dtype is deprecated! Use dtype instead!`. Are you perhaps using an older PyTorch version or CUDA driver? I am currently using torch 2.9.0 and CUDA 560.35.05.

Also, torch.bfloat16 is broken on many GPUs (e.g. V100), see this issue: #182

I also tried torch.float32, but still get infinite repeat bug.

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
import torch
torch.manual_seed(0)

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-VL-4B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://raw.githubusercontent.com/99991/testing/refs/heads/master/misc/test.png",
            },
            {
                "type": "text",
                "text": "Transcribe this image as markdown.",
            },
        ],
    }
]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print(output_text[0])

[dwt]

We see the same issue running those models in Ollama or LM-Studio. I haven't tried increasing the repeat penalty yet, but we do have quite some documents with large repeating sections (with small differences, like tables with almost identical lines but different dates).

Should the repeat penalty be a solution here? Is that an otherwise known issue of the model, or is it likely that the quantizations we use are just too strong?

[scouzi1966]

I have the same issue-especially with lower quants of 4 bits. For example "generate a fully functional scientific calculator in html/css/js" will lead to Qwen3V repeating buttons forever. This is on llama.cpp (GGUF) on MacOS but also happens on Apple MLX converted models.

[ncphu]

I faced the same issue. Does anyone have final solution?

[scouzi1966]

At least we know it's a common model issue and not our code. I will try to mitigate it.

Sill a big thanks to the Qwen team for this awesome model.

[99991]

I think the problem lies in the way that the Qwen models were trained. When the same token has been generated a few times, the model is stuck in a loop and can not get out. That is true even for the biggest Qwen-models. For example, the model from https://chat.qwenlm.ai/ fails for the following simple image, which has a few repeating zeros, followed by 459. Once it has started spitting out zeros, it just keeps on going forever and ever and never generates the last thee non-zero digits 459.

Input image

Chat

I looked at the logits of the top 10 tokens, and it seems like there is a state transition from generating genuine zeros to getting stuck at generating zeros forever. Darker lines represent later tokens:

I am not sure whether it is sufficient to look at just the logits for reliable detection, but maybe there are some infinite repeat neurons in the network that can be identified and disabled.

However, it would be more desirable to fix this issue during training. Here are a few ideas:

It would probably help to train on sequences with lots of repetition that eventually transition into non-repeating sequences, so the model can learn to break out of those loops. Maybe the issue lies in the way that the training data has been prepared. Commonly, there is a preprocessing step where "low quality" data is removed. Perhaps all data with lots of repetition has been identified as low quality data and been removed, so the model has never learnt how to deal with it.

Another approach would be to generate token coordinates in addition to tokens. When too many tokens are generated at the same coordinates, the model is obviously stuck, which is much easier to detect than repeating text sequences. It is also easy to penalize this kind of generation with an appropriate loss function during training. Likewise, coordinates outside of the image boundaries are easy to penalize as well.

Alternatively, the model could indicate the image patch from which the currently generated token originated. There are only so many characters that can fit into an image patch, so it should be easy to introduce a limit here. It might make sense to combine this with some kind of weight based on font size, since larger characters take up more space in a patch.

[scouzi1966]

@99991 It actually passes on my app with Qwen3-VL-8B (fp16). Backend in this case is llama.cpp with Apple metal

[99991]

@scouzi1966 It is nice that it passes for you on a larger model with llama.cpp backend, but it would be nice if it worked all the time on all images, not just sometimes.

I have begun work on a benchmark to quantify the repetition issue.

Loopy Score	# Loopy Outputs	Decoding	Model
0.0 %	0	greedy	GLM-4.6V-Flash-Q5_K_M
2.8 %	14	greedy	gemma-3-27b-it-Q4_K_M
4.4 %	22	greedy	Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M
8.0 %	40	greedy	gemma-3-4b-it-Q4_K_M
21.4 %	107	greedy	Qwen3-VL-30B-A3B-Instruct-Q4_K_S
24.4 %	122	greedy	Qwen3-VL-8B-Instruct-Q5_K_M
30.2 %	151	greedy	InternVL3-14B-Instruct-Q4_K_M
36.0 %	180	greedy	Qwen3-VL-8B-Instruct
36.4 %	182	greedy	Nanonets-OCR-s-Q4_K_M
39.6 %	198	non-greedy	Qwen3-VL-4B-Instruct
40.8 %	204	greedy	Qwen3-VL-4B-Instruct-BF16
40.8 %	204	greedy	Qwen3-VL-4B-Instruct-Q5_K_M
41.8 %	209	non-greedy	Qwen3-VL-2B-Instruct
45.8 %	229	greedy	Qwen3-VL-4B-Instruct
49.8 %	249	greedy	Qwen3-VL-2B-Instruct
54.6 %	273	greedy	Qwen3-VL-2B-Instruct-BF16

Insights so far:

• Greedy decoding leads to more repetition
• Larger models are better
• Gemma models are much more robust than Qwen models

EDIT

I have just benchmarked GLM-4.6V-Flash, which is absolutely perfect and has zero repetition for all 500 image and prompt combinations I tested.

[nitin302]

seeing the same issue

[scruffynerf]

+1, annoying, no solution?

[AKCqhzdy]

So I wander whether there are any available methods to avoid repetition? In the production environment partial absence of results should be allowed more than continuous repetition, since it may lead to a significant consumption of time and space

[BruceHuQi]

I use the Qwen3-VL-4B model fine-tuned on A100 for furniture recognition on multiple images with a 4090 GPU, but some of the images yield duplicate outputs.

[moarseniy]

I have the same issue for bounding box detection using Qwen3-VL-32B.
Repeated same bounding boxes in output.