Consistent speech model input names for the Seq2SeqTrainer generate function

[OllieBroadhurst]

🚀 Feature request

Could we maybe have a consistent naming convention for speech models? So far we have:

• input_features
• input_values
• input_ids

From what I can tell, these are mostly the same for the purposes of how the Seq2SeqTrainer interprets them.

Motivation

This would prevent the need for custom Seq2SeqTrainer classes and would make training more modular.

Your contribution

A change in param names would do the trick but could break a lot of code. Alternatively adding the capability to accept different key values in the generate function here would work too using a (clunky) mapping such as INPUT_MAPPING_LABELS = {"input_features": "input_ids", "input_values": "input_ids", "input_ids": "input_ids"}.

[NielsRogge]

Yes indeed. @patrickvonplaten

Also, the generate method will be extended to also work with pixel_values (for models that generate text given an image). The general input of generate could then be called inputs (which could be input_ids for text, input_features for speech, pixel_values for images, etc.).

Using a mapping is probably the way to ensure backwards compatibility.

[patrickvonplaten]

I can tackle this next week :-)

[patrickvonplaten]

Related: #14421

[patrickvonplaten]

This is resolved now with https://github.com/huggingface/transformers/pull/14802/files no?

[OllieBroadhurst]

Hey there. Looks cool! One super edge case is SpeechEncoderDecoderModel.

For example, if we use a Wav2Vec2Model encoder and a XLMRobertaForCausalLM decoder, we would have ['input_ids', 'attention_mask'] when using either processor.tokenizer.model_input_names for Wav2Vec2 or tokenizer.model_input_names for XLMRoBERTa, when in fact the model requires input_values in the generate function.

[OllieBroadhurst]

@patrickvonplaten the only way I can think of a truly generalized solution without breaking changes or mappings, is using inspect on the forward function:

import inspect
from transformers import SpeechEncoderDecoderModel

model = SpeechEncoderDecoderModel.from_pretrained('facebook/s2t-wav2vec2-large-en-de')
list(inspect.signature(model.forward).parameters)[0]

For the above example that would return input_values (as opposed to using Speech2TextProcessor.tokenizer.model_input_names which would show input_ids). But like I say, super edge case and happy to leave it as is.

[patrickvonplaten]

Great catch @OllieBroadhurst! I'll include a fix in #14856 . Hope to have it merged by tomorrow