Making ConvBert Tokenizer independent from bert Tokenizer

[IMvision12]

What does this PR do?

Fixes #19303

Added BertTokenizer class in tokenization_convbert.py and BertTokenizerFast in tokenization_convbert_fast.py

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for working on this! Left a couple of pointers on how to finish the work :-)

[IMvision12]

Done! @sgugger do i need to do same for convbert_fast?

[sgugger]

Thanks for iterating! We're on the right path.
Now we just need to make sure the util scanning the copies is happy (e.g., the copies should match the original). I've left comments in that direction and you can check locally if the util is happy by running make repo-consistency

[IMvision12]

@sgugger Thanks for quick feeback,
For tokenization_convbert.py i have change the comment to # Copied from transformers.models.bert.tokenization_bert.BertTokenizer with ConvBertTokenizer->BertTokenizer

and for tokenization_convbert_fast.py i have change the comment to # Copied from transformers.models.bert.tokenization_bert_fast.BertTokenizerFast with ConvBertTokenizerFast->ConvBertTokenizer

for convbert_fast make repo-consistency gives error : - src/transformers\models\convbert\tokenization_convbert_fast.py: copy does not match models.bert.tokenization_bert_fast.BertTokenizerFast at line 55

[sgugger]

You'll need to add broader patterns than just the full name of the tokenizer as BERT is used in the docstrings for instance. To see what the copy utils wants to modify, you can run make fix-copies locally :-)

[IMvision12]

After running make fix-copies it changes slow_tokenizer_class = BertTokenizer but it should be ConvBertTokenizer in tokenization_convbert_fast.py

[sgugger]

Yes, that's why I made the suggestions above.

[IMvision12]

So, do i need to copy BertTokenizer and BertTokenizerFast class to tokenization_convbert_fast.py as after adding those classes it passes all tests

[sgugger]

Just accept the suggestions above.

[IMvision12]

Done @sgugger are there any more changes?

[sgugger]

I'm not following what you're doing in the fast tokenizer file. The goal is to have ConvBertTokenizerFast stop depending on BertTokenizerFast by copying the code from that file. Not put the code of BertTokenizer there.

[sgugger]

Thanks a lot for iterating! This is almost ready to merge, just a couple of last nits.

[sgugger]

Thanks a lot for iterating on this PR! Looks perfect now.