You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thank you for sharing your work with us. I tried to use your model (PhoGPT tokenizer) and set the max length to 8192, but the tokenizer's output did not add any padding tokens. Here is an example:
Dear VinAI team,
Thank you for sharing your work with us. I tried to use your model (PhoGPT tokenizer) and set the max length to 8192, but the tokenizer's output did not add any padding tokens. Here is an example:
The output is:
{'input_ids': [2985, 270, 1117, 1378], 'attention_mask': [1, 1, 1, 1]}
You can see that the output token list only has 4 tokens. Should it be 8192 tokens instead?
The text was updated successfully, but these errors were encountered: