model.generate() function raise a exception

[yumoqing]

System Info

• transformers version: 4.23.1
• Platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.35
• Python version: 3.10.6
• Huggingface_hub version: 0.10.1
• PyTorch version (GPU?): 1.13.0+cu117 (False)
• Tensorflow version (GPU?): 2.10.0 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
from datasets import load_dataset
import soundfile as sf
model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr")
def map_to_array(batch):
    speech, _ = sf.read(batch["file"])https://huggingface.co/docs/transformers/model_doc/speech_to_text
    batch["speech"] = speech
    return batch

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"])
transcription = processor.batch_decode(generated_ids)
print(f'{transcription=}')

1. the above code copy/paste from huggingface.co
2. running the script
3. get a exception

Traceback (most recent call last):
  File "/home/ymq/tmp/pretrained-models/test/t.py", line 16, in <module>
    generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"])
  File "/home/ymq/py3/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/ymq/py3/lib/python3.10/site-packages/transformers/generation_utils.py", line 1208, in generate
    self._validate_model_kwargs(model_kwargs.copy())
  File "/home/ymq/py3/lib/python3.10/site-packages/transformers/generation_utils.py", line 910, in _validate_model_kwargs
    raise ValueError(
ValueError: The following `model_kwargs` are not used by the model: ['input_ids'] (note: typos in the generate arguments will also show up in this list)

Expected behavior

get stt result text

[sgugger]

cc @sanchit-gandhi and @gante

[sanchit-gandhi]

Hey @yumoqing,

In this case, it's just the code snippet on the model README that's wrong. Pasting a corrected version of the code snippet that you can use:

import torch
from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
from datasets import load_dataset

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr")

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]

inputs = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt")
generated_ids = model.generate(input_features=inputs["input_features"], attention_mask=inputs["attention_mask"])
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(transcription)

Print Output:

['mister quilter is the apostle of the middle classes and we are glad to welcome his gospel']

I've opened a PR to update the example on the model's README card on the Hub: https://huggingface.co/facebook/s2t-small-librispeech-asr/discussions/2/files