feat[vLLM]: Support text replacement offsets in the remaining old-format processors by harshaljanjani · Pull Request #47614 · huggingface/transformers

harshaljanjani · 2026-07-28T20:06:13Z

What does this PR do?

→ Change alluding to the discussion in vllm-project/vllm#50107 currently blocking the vLLM PR.
→ vllm-project/vllm#50107 (comment)
→ vllm-project/vllm#50107 (comment)

cc: @zucchini-nlp @eustlb

Code Agent Policy

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline, Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you fix any necessary existing tests?

zucchini-nlp · 2026-07-29T08:30:45Z

I will come to merge this PR after we settle down vLLM updates
Thanks a lot for working on this ❤️

zucchini-nlp

Niiice, looks good to me overall, just left comments about moving some things around if possible

zucchini-nlp · 2026-07-31T09:31:46Z

@@ -55,6 +51,7 @@ def __init__(
        self.image_seq_length = image_seq_length
        self.image_token_id = tokenizer.image_token_id


i think we also have to update image_token_id to boi_token_id here

Done, thanks for catching this :)
Needed an image_token_ids override with it otherwise token_type_ids marks the boi instead of the 256 soft tokens. Also updated vLLM to read the soft id off the tokenizer like native gemma3_mm.py

zucchini-nlp · 2026-07-31T09:32:19Z

+        # Create empty text to be replaced with placeholders
+        if images is not None and not text:
+            images = self.image_processor.fetch_images(images)
+            text = [" ".join([self.boi_token] * len(images)) for images in make_nested_list_of_images(images)]


better places in self.prepare_inputs_layout

zucchini-nlp · 2026-07-31T09:33:22Z

+        if images is not None and text is not None:
            batched_images = make_nested_list_of_images(images)


same, make nested list is beter placed in prepare_inputs which is usually called before valid. Here then we can expect inputs are "normalized for model"

zucchini-nlp · 2026-07-31T09:34:41Z

+    def _check_special_mm_tokens(self, text: list[str], text_inputs: "BatchFeature", modalities: list[str]):
+        """
+        Checks that number of special tokens in text and processed text is same. The count can be different
+        if tokenized text was truncated, leading to issues in model code.
+        """
+        token_str = self.tokenizer.image_token
+        token_id = self.tokenizer.image_token_id
+        if token_str is not None and token_id is not None:
+            ids_count = [list(ids).count(token_id) for ids in text_inputs["input_ids"]]
+            text_count = [sample.count(token_str) for sample in text]
+
+        if ids_count != text_count:
+            raise ValueError(
+                f"Mismatch in `image` token count between text and `input_ids`. Got ids={ids_count} and text={text_count}. "
+                "Likely due to `truncation='max_length'`. Please disable truncation or increase `max_length`."


can you add a small comment on why overriden - different tokens used for placeholder in input text and expanded text

zucchini-nlp · 2026-07-31T09:35:16Z

-            "Deprecated: `processor.image_token` will switch from returning "
-            "`tokenizer.image_token` to `tokenizer.boi_token` in v5.11."
+    def replace_image_token(self, image_inputs: dict, image_idx: int, **kwargs) -> str:
+        num_crops = to_py_obj(image_inputs["num_crops"])[image_idx]


do we need to_py_obj? 🤔

Removed, num_crops works directly

zucchini-nlp · 2026-07-31T09:37:35Z

    ) -> BatchFeature:
-        requires_backends(self, ["torch"])
-
        text = self._get_validated_text(text)


i'd rename this to common self._validate_inputs

zucchini-nlp · 2026-07-31T09:38:42Z

+        # Audio samples are already collated
+        if len(audio) == 1 and audio[0].ndim == 2:
+            audio = audio[0]


hmm, we didn't have to do that prev, any reason it's required now?

Yeah it's because make_list_of_audio sees a [2, N] tensor as one sample, so we only ended up with one replacement for two inputs. Before the tensor went straight to the feature extractor which knew it was already collated and returned two audio_embed_sizes

ahh oke, so iiuc the make_list_of_audio assumes mono-channel audio in all cases?

Effectively yes you can say that, it never inspects ndim so [2, N] and [2, 1, N] are treated the same and the mono-channel audio assumptions are really's in the FEs, e.g. VibeVoice rejects ndim != 1, Granite takes the [2, N] and gives one embed size per sample etc.

zucchini-nlp · 2026-07-31T09:42:18Z

+        # Check only if passed explicitly as another value since by default we'll use `pt`
+        for group in (kwargs, kwargs.get("text_kwargs", {}), kwargs.get("common_kwargs", {})):
+            if group.get("return_tensors", "pt") != "pt":
+                raise ValueError(f"{self.__class__.__name__} only supports `return_tensors='pt'`.")


nit: lets merge kwargs first so we dont have to infer which possible dict to check for return_tensors. For ex in dia:

transformers/src/transformers/models/dia/processing_dia.py

Lines 159 to 165 in 1ce2865

output_kwargs = self._merge_kwargs(DiaProcessorKwargs, **kwargs)

audio_kwargs = output_kwargs["audio_kwargs"]

generation = audio_kwargs.get("generation", True)

return_tensors = output_kwargs["text_kwargs"].get("return_tensors", None)

if return_tensors != "pt":

raise ValueError(f"{self.__class__.__name__} only supports `return_tensors='pt'`.")

Done in the dia shape, thanks for the precedent :)

zucchini-nlp · 2026-07-31T09:43:01Z

        if isinstance(text, str):
            text = [text]
        elif not isinstance(text, (list, tuple)):
            raise ValueError("text input must be a string or list of strings")

        if audio is not None:
+            audio = self.feature_extractor.fetch_audio(audio, sampling_rate=self.feature_extractor.sampling_rate)
            audio = make_list_of_audio(audio)
-            data = self.feature_extractor(audio, **audio_kwargs)
-            audio_lengths = data["padding_mask"].sum(dim=-1).cpu().numpy()
+            for example in audio:
+                if example.ndim != 1:
+                    raise ValueError(f"Audio should be mono, got shape: {example.shape}")

            # Replace audio duration placeholders in text
-            audio_durations = audio_lengths / self.feature_extractor.sampling_rate
-            audio_durations_iter = iter(audio_durations)
+            audio_durations = iter([len(el) / self.feature_extractor.sampling_rate for el in audio])
            audio_duration_pattern = re.compile(re.escape(self.audio_duration_token))
            for i in range(len(text)):


self.prepare_inputs? and the valuerror in self.validate

zucchini-nlp · 2026-07-31T09:45:22Z

+    @property
+    def _audio_processor(self):
+        # TODO: To be replaced with `audio_processor`
+        return getattr(self, "audio_processor", getattr(self, "feature_extractor", None))
+


huh, hopefully Eustache's upcoming refactor will make this better

github-actions · 2026-08-02T10:27:51Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: gemma3, granite_speech, vibevoice_asr

github-actions · 2026-08-02T10:39:41Z

CI recap

Dashboard: View test results in Grafana
Latest run: 30743753698:2
Result: success | Jobs: 16 | Tests: 174,907 | Failures: 0 | Duration: 16h 13m

zucchini-nlp

Great work, left one tiny comment. I will trigger slow CI (please dont push anything until it's done) and when it passes, we can merge :)

zucchini-nlp · 2026-08-03T09:05:43Z

+        if text is not None and not isinstance(text, str):
+            if not isinstance(text, list) or not isinstance(text[0], str):
+                raise TypeError("Invalid input text. Please provide a string, or a list of strings")


why not in validate_input?

Because prepare_inputs_layout runs before validate_inputs so by then the dict has already become a list of keys so it'll pass this and integer would fail earlier in list(text).copy()

ah oke, tbh this could be a general check in Mixin/ Not on you ofc, let's merge

zucchini-nlp · 2026-08-03T09:07:19Z

-            kwargs["padding"] = True
-        text_inputs = self.tokenizer(prompt_strings, **kwargs)
-        return BatchFeature(data={**text_inputs, **audio_inputs})
+        kwargs.setdefault("audio_kwargs", {}).setdefault("device", device)


ah interesting, then it makes sense, I think Eustache's audio processor PR will bring device as a standard kwarg

zucchini-nlp · 2026-08-03T09:08:56Z

+        # Audio samples are already collated
+        if len(audio) == 1 and audio[0].ndim == 2:
+            audio = audio[0]


ahh oke, so iiuc the make_list_of_audio assumes mono-channel audio in all cases?

zucchini-nlp · 2026-08-03T09:12:04Z

run-slow: gemma3, granite_speech, vibevoice_asr

github-actions · 2026-08-03T09:15:52Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/gemma3", "models/granite_speech", "models/vibevoice_asr"]
quantizations: []

HuggingFaceDocBuilderDev · 2026-08-03T09:25:54Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

harshaljanjani · 2026-08-03T09:42:26Z

Thanks a lot for your time @zucchini-nlp, looking forward to going back to the vLLM PR :)

github-actions · 2026-08-03T09:47:35Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	792e910e	workflow commit (merge commit)
PR	d69ba74c	branch commit (from PR)
main	b3a36037	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

feat: Add return_text_replacement_offsets to TextKwargs

e3847e8

harshaljanjani mentioned this pull request Jul 28, 2026

Support mm_processor_cache in the Transformers multimodal backend vllm-project/vllm#50107

Open

4 tasks

harshaljanjani added 2 commits July 29, 2026 17:05

Merge branch 'main' into feat/return-text-replacement-offsets

fc70dec

refactor: Design refactor - 1

faeea18

harshaljanjani changed the title ~~feat[vLLM]: Add return_text_replacement_offsets to TextKwargs~~ feat[vLLM]: Support text replacement offsets in the remaining old-format processors Jul 30, 2026

zucchini-nlp reviewed Jul 31, 2026

View reviewed changes

harshaljanjani added 2 commits August 2, 2026 09:48

refactor: Address review round

d27ea0d

Merge branch 'main' into feat/return-text-replacement-offsets

d69ba74

harshaljanjani requested a review from zucchini-nlp August 2, 2026 19:14

zucchini-nlp approved these changes Aug 3, 2026

View reviewed changes

zucchini-nlp added this pull request to the merge queue Aug 3, 2026

Merged via the queue into huggingface:main with commit b317ff3 Aug 3, 2026
115 checks passed

harshaljanjani deleted the feat/return-text-replacement-offsets branch August 8, 2026 06:11

		@@ -55,6 +51,7 @@ def __init__(
		self.image_seq_length = image_seq_length
		self.image_token_id = tokenizer.image_token_id

		if images is not None and text is not None:
		batched_images = make_nested_list_of_images(images)

	output_kwargs = self._merge_kwargs(DiaProcessorKwargs, **kwargs)
	audio_kwargs = output_kwargs["audio_kwargs"]
	generation = audio_kwargs.get("generation", True)
	return_tensors = output_kwargs["text_kwargs"].get("return_tensors", None)

	if return_tensors != "pt":
	raise ValueError(f"{self.__class__.__name__} only supports `return_tensors='pt'`.")

Uh oh!

Conversation

harshaljanjani commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Uh oh!

zucchini-nlp commented Jul 29, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshaljanjani Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshaljanjani Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshaljanjani Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 2, 2026

Uh oh!

github-actions Bot commented Aug 2, 2026

CI recap

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Aug 3, 2026

harshaljanjani commented Jul 28, 2026 •

edited

Loading

harshaljanjani Aug 2, 2026 •

edited

Loading

harshaljanjani Aug 2, 2026 •

edited

Loading

harshaljanjani Aug 2, 2026 •

edited

Loading