Fix MarianTokenizer to remove metaspace character in decode

[tanaymeh]

What does this PR do?

This PR fixes the MarianTokenizer so that it removes the metaspace character during decode (▁).

Fixes #26018

Who can review?

@ArthurZucker, @xenova

[xenova]

Is this section necessary still?

[tanaymeh]

No it's not, I accidentally left it there! Fixed it now.

[ArthurZucker]

Thanks! We need to add a test, and make sure the fast tokenizers also works! 🤗

[xenova]

make sure the fast tokenizers also works!

Unfortunately, there's no MarianTokenizerFast 😅

[tanaymeh]

Thanks for the review, @ArthurZucker!
I added the following test (it checks with starting special characters in a string like an underscore).

def test_tokenizer_decode(self):
    tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-es")
    source_text = "_This is 1 text string that starts with an _ and ends with one too _"
    ids = tokenizer(source_text)["input_ids"]
    output_text = tokenizer.decode(ids, skip_special_tokens=True)
    self.assertEqual(source_text, output_text)

Does it look good to you?

[tanaymeh]

make sure the fast tokenizers also works!

Unfortunately, there's no MarianTokenizerFast 😅

@xenova @ArthurZucker I would love to add the MarianTokenizerFast if that is considered a worthwhile addition to 🤗 transformers!

[ArthurZucker]

LGTM, not really a fan of relying on the strip for this kind of process but it's quick fix so should be okay!

[ArthurZucker]

Instead of manually adding the spiece underline we can just use the example from the issue: tokenizer.decode(tokenizer("hello world")['input_ids'], skip_special_tokens=True)

[xenova]

+1

Also, I can't quite tell on mobile, but are those just regular underscores? The sentencepiece underscore is a slightly different character (but looks similar).

[tanaymeh]

My fault. My VSCode theme really made them both look similar 🥲
Fixed it now!

[ArthurZucker]

Thanks for fixing! This is breaking but it's a bug fix so good to go!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[xenova]

LGTM!

[tanaymeh]

Thanks for the reviews and merging!
I was wondering @ArthurZucker if MarianTokenizerFast would be a worthwhile addition to HF transformers and if I can contribute to adding it.

[ArthurZucker]

It does not seem to have been requested a lot 😄 but feel free to add it if you want some good experience with tokenizers

[xenova]

I think this would be a good addition, simply because of the number of monthly downloads the >1400 models get 😇. The top 5 models alone total ~2.75M downloads in the past month.

https://huggingface.co/models?sort=downloads&search=Helsinki-NLP%2Fopus-mt