You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Batches of variable-length prompts are currently not forwarded in a single step into the decoder. Only the tokens up to the minimum length in the batch are forwarded at once, and the remaining tokens are force-decoded in the decoding loop.
It could be more efficient to forward the batch in a single step but this requires supporting padding positions on the left. It means each entry in the batch has an offset in position-aware modules:
softmax
position encodings
rotary embeddings
padder
MHA values "mask"
The text was updated successfully, but these errors were encountered:
Batches of variable-length prompts are currently not forwarded in a single step into the decoder. Only the tokens up to the minimum length in the batch are forwarded at once, and the remaining tokens are force-decoded in the decoding loop.
It could be more efficient to forward the batch in a single step but this requires supporting padding positions on the left. It means each entry in the batch has an offset in position-aware modules:
The text was updated successfully, but these errors were encountered: