🚨🚨🚨 [NLLB Tokenizer] Fix the prefix tokens 🚨🚨🚨

[ArthurZucker]

What does this PR do?

The NLLB tokenizer's suffix and prefix token were wrong w.r.t to the paper.
This breaking change fixes the tokenizer.
Could be none breaking if we add these to the configuration file maybe? But it is a required change
Have to update the tests but should be good.

The big problem was the prefix and suffix tokens.
The previous version adds [self.eos_token_id, self.cur_lang_code] at the end of the token sequence for both target and source tokenization. This is wrong as the NLLB paper mentions (page 48, 6.1.1. Model Architecture) :

Note that we prefix the source sequence with the source language, as opposed to the target
language as previously done in several works (Arivazhagan et al., 2019; Johnson et al.,
2017). This is primarily because we prioritize optimizing zero-shot performance of our
model on any pair of 200 languages at a minor cost to supervised performance.

Previous behaviour:

>>> from transformers import NllbTokenizer
>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[13374, 1398, 4260, 4039, 248130, 2, 256047]
>>> # 2: '</s>'
>>> # 256047 : 'eng_Latn'

New behaviour

>>> from transformers import NllbTokenizer
>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[256047, 13374, 1398, 4260, 4039, 248130, 2]

Enabling the old behaviour:

>>> from transformers import NllbTokenizer
>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", legacy_behaviour = True)

This parameter should be part of the tokenizer_config.json.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[LysandreJik]

Thank you for the implementation, @ArthurZucker!

Could you please put in the PR description the gist of the breaking change with a code sample, and how to revert to the previous behavior if users would like that?

Thank you

[sgugger]

As @LysandreJik said, we would need a way to enable the old behavior for users who rely on it.

[ArthurZucker]

Indeed thanks for the tip on how to enable that swiftly!

[sgugger]

Perfect, thanks!

[ArthurZucker]

One test is failing with NLLB (running slow ones locally, test_encode_decode_with_spaces), fixing this before merging.
Edit: Fast and slow have a different behaviour! space_between_special_tokens does not exist in rust (yet, PR coming soon)

[LysandreJik]

Cool, I like the flag :)

Can the doc be shown more prominently? Maybe to replace the disclaimer mentioning to tag me? A disclaimer mentioning that we changed it to what it is now, with the code snippet?

[sgugger]

Needs proofing :-)

[LysandreJik]

Thank you @ArthurZucker!

[ArthurZucker]

Thanks both for proof reading! 👍🏻