`BlenderbotForConditionalGeneration` errors out with `list index out of range`

[ariG23498]

Running the following:

from transformers import AutoTokenizer, BlenderbotForConditionalGeneration

mname = "facebook/blenderbot-400M-distill"
model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = AutoTokenizer.from_pretrained(mname)
UTTERANCE = "My friends are cool but they eat too many carbs."
print("Human: ", UTTERANCE)

inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print("Bot: ", tokenizer.batch_decode(reply_ids, skip_special_tokens=True)[0])

REPLY = "I'm not sure"
print("Human: ", REPLY)

NEXT_UTTERANCE = (
    "My friends are cool but they eat too many carbs.</s> <s>That's unfortunate. "
    "Are they trying to lose weight or are they just trying to be healthier?</s> "
    "<s> I'm not sure."
)
inputs = tokenizer([NEXT_UTTERANCE], return_tensors="pt")
next_reply_ids = model.generate(**inputs)
print("Bot: ", tokenizer.batch_decode(next_reply_ids, skip_special_tokens=True)[0])

Fails with

IndexError: list index out of range

Colab notebook

Should work as expected. The code has been taken from the official docs

[vasqu]

Pretty sure it's due to the cache being initialised with kwargs there, i.e. the number of initialized cache layers is wrong

cc @manueldeprada @Cyrilvallez

[manueldeprada]

having a look

[manueldeprada]

Happens because num_hidden_layers is wrong in the config: It should be same as decoder_layers but it coincides with encoder_layers

@gante since you dealt with a similar problem in #40277, do you have any suggestion other than opening a PR in the Hub??

[gante]

yes, working on it (#40553 ) :D

TL;DR we need one abstraction to pull the encoder sub-config (any modality, not just text) from the full config, to then initialize the encoder cache correctly. There were other name clashes too.