Normalize multimodal input keys in AnyToAnyPipeline

[Sunt-ing]

What does this PR do?

AnyToAnyPipeline accepts multimodal inputs through both direct keyword arguments and dict/dataset samples, but two of those paths used singular keys before calling the processor. videos= was repacked as video, and dict inputs like {"image": ..., "video": ...} were forwarded unchanged. Qwen Omni processors expect images= and videos=, so video inputs could be ignored and image inputs could crash when an image placeholder was expanded without an image grid.

This normalizes pipeline inputs before the processor call. Direct videos= stays plural, dict image / video aliases are converted to images / videos, and explicit plural dict values keep precedence.

End-to-end pipeline repro (real AnyToAnyPipeline + real Qwen3-Omni processor)

Environment:

Hardware: AutoDL c4090, NVIDIA GeForce RTX 4090 present; repro run CPU only with CUDA_VISIBLE_DEVICES=""
OS: Linux AutoDL container
Python: 3.12.3
PyTorch: 2.8.0+cu128
Transformers main: b70d02fc724d04c916832ca4ead03ff05e8fb1ee
Processor metadata: Qwen/Qwen3-Omni-30B-A3B-Instruct via AutoProcessor.from_pretrained
Model: small fake Qwen3OmniMoeForConditionalGeneration; it only replaces the 30B weight download and still exercises pipeline("any-to-any") plus the real processor input path
Command: CUDA_VISIBLE_DEVICES="" python repro_any_to_any_keys.py

import numpy as np
import torch
from PIL import Image

from transformers import AutoProcessor, GenerationConfig, pipeline
from transformers.pipelines import AnyToAnyPipeline


class Qwen3OmniMoeForConditionalGeneration(torch.nn.Module):
    input_modalities = ("image", "video", "audio", "text")
    output_modalities = ("text", "audio")

    def __init__(self):
        super().__init__()
        self.config = type("Cfg", (), {"_commit_hash": None, "model_type": "qwen3_omni_moe"})()
        self.generation_config = GenerationConfig(max_new_tokens=1)
        self.last_keys = None

    @property
    def device(self):
        return torch.device("cpu")

    def can_generate(self):
        return True

    def generate(self, input_ids=None, **kwargs):
        self.last_keys = sorted(kwargs.keys())
        batch_size = input_ids.shape[0] if input_ids is not None else 1
        token = torch.full((batch_size, 1), 42, dtype=input_ids.dtype if input_ids is not None else torch.long)
        if input_ids is None:
            return token
        return torch.cat([input_ids, token], dim=1)


processor = AutoProcessor.from_pretrained("Qwen/Qwen3-Omni-30B-A3B-Instruct", trust_remote_code=False)

direct_model = Qwen3OmniMoeForConditionalGeneration()
direct_pipe = pipeline("any-to-any", model=direct_model, processor=processor)
video = np.zeros((2, 8, 8, 3), dtype=np.uint8)
direct_output = direct_pipe(
    text="describe the video",
    videos=video,
    return_full_text=False,
    max_new_tokens=1,
    processor_kwargs={"num_frames": 2},
)
print("direct_videos", "GEN_KEYS", direct_model.last_keys, "HAS_VIDEO_GRID", "video_grid_thw" in direct_model.last_keys)
print("direct_videos", "OUT", direct_output)

dict_model = Qwen3OmniMoeForConditionalGeneration()
dict_pipe = AnyToAnyPipeline(model=dict_model, processor=processor, max_new_tokens=1)
image = Image.new("RGB", (16, 16), color="white")
image_text = f"{processor.image_token} describe"
video_text = f"{processor.video_token} describe"

for label, inputs in [
    ("dict_image", {"text": image_text, "image": image}),
    ("dict_video", {"text": video_text, "video": video}),
]:
    try:
        output = dict_pipe(inputs, return_full_text=False, processor_kwargs={"num_frames": 2})
        print(label, "OK", "GEN_KEYS", dict_model.last_keys, "OUT_LEN", len(output))
    except Exception as error:
        print(label, "ERR", type(error).__name__, repr(str(error)))

# current main
[transformers] Keyword argument `video` is not a valid argument for this processor and will be ignored.
[transformers] Keyword argument `image` is not a valid argument for this processor and will be ignored.
direct_videos GEN_KEYS ['attention_mask', 'generation_config', 'max_new_tokens'] HAS_VIDEO_GRID False
direct_videos OUT [{'input_text': 'describe the video', 'generated_text': 'K'}]
dict_image ERR StopIteration ''
dict_video ERR StopIteration ''

# after this PR
direct_videos GEN_KEYS ['attention_mask', 'generation_config', 'max_new_tokens', 'pixel_values_videos', 'video_grid_thw', 'video_second_per_grid'] HAS_VIDEO_GRID True
direct_videos OUT [{'input_text': 'describe the video', 'generated_text': 'K'}]
dict_image OK GEN_KEYS ['attention_mask', 'generation_config', 'image_grid_thw', 'pixel_values'] OUT_LEN 1
dict_video OK GEN_KEYS ['attention_mask', 'generation_config', 'pixel_values_videos', 'video_grid_thw', 'video_second_per_grid'] OUT_LEN 1

I did not add a dedicated unit test for this small key-normalization change. The repro above exercises the real pipeline plus real Qwen3-Omni processor path before and after the patch.

Touched-file checks

Checks:

python -m ruff check src/transformers/pipelines/any_to_any.py
python -m ruff format --check src/transformers/pipelines/any_to_any.py
git diff --check

Code Agent Policy

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

cc @Rocketknight1 @zucchini-nlp

[zucchini-nlp]

Can we add tests pls, for ex take a vlm with videos support and enable pipeline tester mixing on it

LKM if you have a tiny ckpt that has to be moved to testing hub repo!

[zucchini-nlp]

i think we can assume now that the keys will be correct, this is usually not called outside pipe

[Sunt-ing]

Thanks @zucchini-nlp. The follow-up narrows this to the direct videos= path and removes the dict image / video alias normalization.

It adds a focused regression with the tiny Qwen2.5-Omni processor plus a fake generate model, and extends the shared pipeline tester path to exercise videos= when a video example is available. Reverting the source line makes the focused test fail because pixel_values_videos and video_grid_thw never reach generate; with the fix it passes.

I also tried enabling the real Qwen2.5-Omni pipeline tester directly. It enters the long Omni generation path on CPU, so I kept this at the pipeline/processor boundary.

[zucchini-nlp]

why is this, lets import an existing model and use it

[Sunt-ing]

Thanks @zucchini-nlp, good point. I updated the test to load the tiny Qwen2_5OmniForConditionalGeneration.

I still stub generate on that loaded instance so the test stays focused and does not enter the slow Omni generation path.

[zucchini-nlp]

tbh I dont understand why we need the override in general. Calling a pipe on model should work ideally without any workarounds

[Sunt-ing]

Thanks @zucchini-nlp, agreed. I removed the generate override entirely; the test now calls the tiny Qwen2.5-Omni model through the pipeline with its real generate.

I only pass public Qwen generation kwargs to keep the real call short.

[zucchini-nlp]

Thanks, lets merge

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

• The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
• A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
• A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

[zucchini-nlp]

@bot /style

[github-actions]

Style fix is beginning .... View the workflow run here.

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29732995254:2
Result: success | Jobs: 15 | Tests: 168,990 | Failures: 0 | Duration: 12h 4m