[generate] Always use decoder config to init cache #40772
Conversation
gante
force-pushed
the
use_right_config
branch
from
0d88c0d to
dd57459
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
gante
changed the title
gante
changed the title
| if requires_cross_attention_cache: | ||
| cross_attention_cache_kwargs = { | ||
| "config": self.config.get_text_config(encoder=True), | ||
| "config": self.config.get_text_config(decoder=True), |
There was a problem hiding this comment.
Sorry? This one should not be changed no?
There was a problem hiding this comment.
No, we want to use the decoder cache! I had the exact same thought [use encoder config] in a recent PR :D
In a nutshell: the config is used here to
- determine which type of layers are used on cross attention
- the number of layers of cross attention we have
Cross attention is a layer in the 👉 decoder 👈 model -- it's the attention between the encoder outputs and the data (tokens on llm) being decoded
There was a problem hiding this comment.
Oh indeed! Easy to get confused with the name of the cache (EncoderDecoderCache) haha!
| # If a config is passed, use it to infer the layer types and initialize accordingly | ||
| if config is not None: | ||
| config = config.get_text_config() | ||
| config = config.get_text_config(decoder=True) |
There was a problem hiding this comment.
Could that somehow clash if we pass an encoder config to it elsewhere? I.e. for encoder/decoder cache
There was a problem hiding this comment.
We always want the decoder config for KV cache purposes
The encoder is only used once to get the encoder outputs, which are then autoregressivelly used in the decoder with cross attention. Both self-attention and cross-attention are decoder layers, and thus parameterized by the decoder.
* mega derp * fix * always use the decoder
* mega derp * fix * always use the decoder
* mega derp * fix * always use the decoder
What does this PR do?
(see title)
Fixes #40644