Chandra VLM can enter repeated-token generation loop with vLLM

[PrashanthLande]

Summary

While integrating Chandra OCR 2 with Docling through a vLLM/OpenAI-compatible endpoint, I observed that some documents can cause the VLM to enter a repeated-token generation loop.

The generation continues until max_new_tokens is reached, resulting in finish_reason="length" instead of a completed OCR response.

Observed behavior

For a small set of reproducible documents:

• Chandra starts generating normally.
• The output eventually begins repeating tokens/content.
• Generation continues until the configured max_new_tokens limit is reached.
• The request can take several minutes before returning.
• The resulting OCR output is unusable.

The issue is particularly problematic when the generated response is expected to complete within a bounded processing time.

Investigation

I implemented a small repetition detector on the generated VLM response to identify repeated output.

I also tested the sampling strategy used by Chandra's client-side retry behavior:

• Initial generation: temperature=0.0, top_p=0.1
• On repeated-token detection, retry with a higher temperature and top_p=0.95

In my testing, retrying the same document with the adjusted sampling parameters successfully produced a valid response for cases that previously entered the repetition loop.

For example, the retry logic increases temperature progressively while switching top_p to 0.95.

Challenge with retrying after completion

The main issue is that with a non-streaming request, repetition can only be detected after the generation has already completed.

For example, if the first generation consumes the full max_new_tokens budget before we detect the repetition, retrying means paying the full generation cost again.

In one of my test cases, several attempts were required before getting a successful response, resulting in roughly 20 minutes of processing time for a single page.

So simply adding multiple retries after generation may solve correctness but introduce a significant latency problem.

Possible approaches

I'm currently considering a few possible directions:

1. Detect repeated tokens during generation and terminate the current generation early, then retry with adjusted sampling parameters.
2. Add Chandra-specific retry handling at the VLM integration layer.
3. Investigate whether this can be handled through vLLM generation/repetition controls instead.
4. Introduce a retry/time budget so that repeated-token recovery does not exceed a reasonable processing limit.

I'm not sure which approach would fit best with the current Docling VLM architecture.

Question

Has this behavior been seen with the Chandra integration before?

More importantly, where would you prefer this mitigation to live — in the Chandra-specific integration, in the generic VLM pipeline, or at the vLLM/server level?

I can provide the reproducible documents, logs, and a minimal implementation of the repetition detection/retry approach if useful.

[PrashanthLande]

Just following up on this investigation. I was able to reproduce the repeated-token issue and verify that adjusting the sampling parameters (top_p and temperature) on retry can recover the generation. Would it make sense to handle this retry logic in Docling’s Chandra integration?