bert-base-multilingual-uncased vocabulary not consecutive

[ntubertchen]

🐛 Bug

When I was checking out bert-base-multilingual-uncased vocabulary. I receive the warning "Saving vocabulary to ./vocab.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted"

I ran the similar command on two different machine and got the same warning.

from pytorch_transformers import *
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased',do_lower_case=True)
tokenizer.save_vocabulary('./')

I ran it on

• OS:
• Python version: python3.5
• PyTorch version: pytorch1.0.1.post2
• PyTorch Transformers version (or branch): 1.0
• Using GPU ? Yes
• Distributed of parallel setup ?no

[LysandreJik]

H! Could you please specify on which OS you have this error? I cannot reproduce this on MacOS 10.15, nor on Ubuntu 18.04 with both Python 3.5 and 3.6.

[ntubertchen]

Hi, @LysandreJik

I tried these on Ubuntu 16.04.6 LTS.

[gilnoh]

@ntubertchen
Just in case helpful for you -- I had exactly the same issue with release 1.0 of pytorch-transformer, when I worked with multilingual BERT base models. (Ubuntu 19.04, training MRPC model with run_glue.py script.) Always at the end of training (e.g. MRPC), it gave me the above warning. And the eval results were quite strange. (e.g. often much lower than expected). Due to this, and also Chinese model broken issue; I installed master branch as of now (commits after Roberta models added), and the error went away. no more such warnings, and stable result.

Maybe you should try to install from current master (which will install locally built 1.1); it helped me. I think some code has been changed on handling multilingual vocab file.

[LysandreJik]

Thanks @gilnoh. @ntubertchen can you let me know if you still have the same problem with the current (1.1.0) release ?

[thomwolf]

Yes, we had an issue in Bert tokenizer that made it lose a token in the Chinese vocabulary.
This is fixed now with the merge of #860.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.