Add the fast implementation of BlenderbotTokenizer

[stancld]

🚀 Feature request

As it is the case for other models' tokenizers, add the fast implementation of BlenderbotTokenizer.

Motivation

To have faster tokenization for Blenderbot models. (Also, the implementation should be pretty straightforward considering the similarity to the RobertaTokenizer/RobertaTokenizerFast.)

Your contribution

I would like to have a look at this and will be glad to add that.

[LysandreJik]

That sounds great @stancld, we would love a PR!

[stancld]

@LysandreJik - I found a minor issue in the formatting of tokenizer_config.json at https://huggingface.co/facebook/blenderbot-3B/blob/main/tokenizer_config.json, where is "add_prefix_space": "true" instead of "add_prefix_space": true. This leads to the error (see below) during the slow->fast (tokenizer) conversion. It can be handled in the converter source code, though I believe it might be better to update a config file. Is there a way of how to send a PR to HF's hub?

Error:

TypeError: Can't convert 'true' to PyBool

[LysandreJik]

Ah ! There's no way to do that as of now - let me handle that for you.

[LysandreJik]

Should be done with huggingface#c468b23

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[stancld]

still open #13720