Repository navigation
Replies: 1 comment
|
The full-size return is intentional: it keeps the KV tensors static for compilation/export. The unfilled slots are not supposed to participate in attention. In torchtune's normal generation path, the cache and mask have the same
One caveat: if you call |
Uh oh!
There was an error while loading. Please reload this page.
Hello,
Please excuse me if the question is stupid. It seems to me that the update method in torchtune.modules.KVCache always outputs the full cache of dimension max_seq_length, even if only partially filled. Wouldn't this affect the scaling of the attention output during inference, whenever the sa_scale parameter is not set in torchtune.modules.TransformerSelfAttentionLayer? This is because the extra zeros in the cache would affect the computation of the softmax in the attention matrix.
Thank you!
All reactions