why BertTokenizer is used instead of BartTokenizer?

[Chen-Wang-CUHK]

Thank you for your nice work!

when preprocessing data, I follow your code to use BertTokenizer to load the cpt-base tokenizer. The tokenizer is load successfully, but I get the following warning message:

"""
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'BartTokenizer'.
The class this function is called from is 'BertTokenizer'.
"""

Then I tried to use BartTokenizer to load it, but I failed.

The question is whether I should ignore the warning and still use the BertTokenizer? Thank you.

[choosewhatulike]

Thank you for your attention.

BertTokenizer should be used instead of BartTokenizer. Because our tokenizer is following the tokenizer of BERT-base-chinese.
And you can ignore the warming message from huggingface. Don't worry about it.

[Chen-Wang-CUHK]

Thank you for your attention.

BertTokenizer should be used instead of BartTokenizer. Because our tokenizer is following the tokenizer of BERT-base-chinese. And you can ignore the warming message from huggingface. Don't worry about it.

Get it. Thank you.