You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What steps did you take and what happened:
[A clear and concise description of what the bug is.]
With decoder-only models, such as GPT-2 padding should be done on the left. This is because the output is a continuation of the input prompt -- there would be gaps in the output without left padding. Huggingface will warn if we don't use left padding for decoder models. huggingface/transformers#18388 (comment)
The huggingface will throw a warning about this
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer
What did you expect to happen:
It should pad left for decoder only models.
What's the InferenceService yaml:
[To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]
/kind bug
What steps did you take and what happened:
[A clear and concise description of what the bug is.]
With decoder-only models, such as GPT-2 padding should be done on the left. This is because the output is a continuation of the input prompt -- there would be gaps in the output without left padding. Huggingface will warn if we don't use left padding for decoder models.
huggingface/transformers#18388 (comment)
The huggingface will throw a warning about this
What did you expect to happen:
It should pad left for decoder only models.
What's the InferenceService yaml:
[To help us debug please run
kubectl get isvc $name -n $namespace -oyaml
and paste the output]Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
):/etc/os-release
):The text was updated successfully, but these errors were encountered: