Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Probably a bug in XLMRobertaTokenizer #2795

Closed
zjujh1995 opened this issue Feb 10, 2020 · 2 comments · Fixed by #3198
Closed

Probably a bug in XLMRobertaTokenizer #2795

zjujh1995 opened this issue Feb 10, 2020 · 2 comments · Fixed by #3198
Labels
Core: Tokenization Internals of the library; Tokenization.

Comments

@zjujh1995
Copy link

zjujh1995 commented Feb 10, 2020 •
edited
Loading

(Everything goes perfect when I did experiment with MultilingualBert, but seems only the base-model is released.)

When using XLM-R, the according tokenizer (XLMRobertaTokenizer) converts <unk> and every OOV token into id = 1. However, 1 should be the number of <pad>. (And the tokenizer can convert 1 to <pad>, 3 to <unk>).
@LysandreJik LysandreJik added the Core: Tokenization Internals of the library; Tokenization. label Feb 10, 2020
@ShiroKL
Copy link

ShiroKL commented Feb 12, 2020 •
edited
Loading

Hi, if I am not wrong the pad should be 2 ?
at least the parameters tokenizer.pad_token_id for XLM-R

EDIT: 2 is eos sorry.

@dennlinger dennlinger mentioned this issue Feb 14, 2020
Closed
@LysandreJik LysandreJik mentioned this issue Mar 9, 2020
Merged
@LysandreJik
Copy link
Member

Indeed, this is a bug that will be fixed when #3198 is merged. Thanks for letting us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Core: Tokenization Internals of the library; Tokenization.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

XLM-R Tokenizer now passes common tests + Integration tests huggingface/transformers
3 participants
@LysandreJik @zjujh1995 @ShiroKL