Add BlenderbotTokenizerFast

[stancld]

What does this PR do?

This PR add the fast (rust) implementation of BlenderbotTokenizer.

Fixes #13634

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@LysandreJik

Anyone in the community is free to review the PR :)

[patil-suraj]

LGTM, thanks a lot for adding this!

[patil-suraj]

(nit) maybe call this fast_tokenizer for consistancy

[stancld]

In test files, it seems to me the rust_tokenizer naming is used instead of fast_tokenizer. However, I can change it here :)

[SaulLu]

Thank you so much for this addition! 😄

[SaulLu]

Let me ping @Narsil if he ever has some leads to fix the failed pipeline tests. 🙂

[Narsil]

The fix is here: Narsil@605725f

Not sure it's the best fix so I will describe a bit more the issue.

Blenderbot implements for AutoModelForSeq2SeqLM (the real way to use it) and AutoModelForCausalLM (I don't think it's really used in practice, but it's implemented in the lib).

The pipeline tests every model that implements its supported architecture so BlenderbotForCausalLM are used. BUT the test config for pipelines is taken from the test modeler, which implements (understandably) the encoder/decoder config (with encoder_no_repeat_ngram_size=3). But when the tests is tried with a decoder-only BlenderbotForCausalLM then it fails.

test_pipeline_common can have a very specific override as this behavior should be very marginal (implementing both) within the lib and also very consistent (CausalLM = decoder only, and encoder_no_repeat_ngram doesn't make any sense for decoder-only)

[Narsil]

Something might have been changed since last time, since the embeddings are now an issue.
Suggesting a diff to override only the config used for pipelines tests on blenderbot.

index 33d506492..9e04ec89d 100644
--- a/tests/test_modeling_blenderbot.py
+++ b/tests/test_modeling_blenderbot.py
@@ -137,6 +137,11 @@ class BlenderbotModelTester:
             pad_token_id=self.pad_token_id,
         )
 
+    def get_pipeline_config(self):
+        config = self.get_config()
+        config.max_position_embeddings = 100
+        return config
+
     def prepare_config_and_inputs_for_common(self):
         config, inputs_dict = self.prepare_config_and_inputs()
         return config, inputs_dict

(By default max_embeddings are 20 long which is not enough for some pipelines tests)

[stancld]

@Narsil Thank you very much for your help! :) It looks like everything works now.

[LysandreJik]

This looks good to me! @SaulLu, does it look good to you too? Feel free to merge if so.

[SaulLu]

It's all good for me too! Thank you for this addition and for fixing the latest problems with the pipeline @stancld. 😄

Thank you also @Narsil for giving the right leads to fix the last tests!

[jonatanklosko]

Hey, I opened PRs to add tokenizer.json to the repos:

• facebook/blenderbot-400M-distill
• facebook/blenderbot-3B
• facebook/blenderbot-1B-distill

While the conversions work, tokenizer.json files are useful for us because they allow loading directly using the tokenizers Rust bindings, so if those can be merged it would be appreciated :)