Original Llama-3 tokenizer behaves differently from transformers version

[chawins]

System Info

transformers==4.41.2
tiktoken==0.7.0 and 0.4.0

Who can help?

@ArthurZucker @younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The issue can be produced with the following snippet.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
ids = tokenizer("! ! !").input_ids
print(ids)  # >> [128000, 0, 758, 758]
print(tokenizer.decode(tokenizer("! ! !").input_ids))  # >> <|begin_of_text|>!!! (this is wrong?)

# Llama-3 from https://github.com/meta-llama/llama3
# Download tokenizer.model from https://llama.meta.com/llama-downloads/
from llama import Tokenizer
tokenizer = Tokenizer("path/to/tokenizer.model")
ids = tokenizer.encode("! ! !", bos=True, eos=False)
print(ids)  # >> [128000, 0, 758, 758]
print(tokenizer.decode(ids))  # <|begin_of_text|>! ! ! (this is expected)

Expected behavior

The string after encoding and decoding back should be the same. The original tokenizer has this behavior, but not the transformers version which throws out the whitespace. Is this expected?

I know that original Llama-3's tokenizer is based on tiktoken now. Is that the reason we see this difference?

[ArthurZucker]

Hey! This is because of a default in transformers:
print(tokenizer.decode(tokenizer("! ! !").input_ids, clean_up_tokenization_spaces=False) )
should do the trick.
Let's set it to default False and deprecated it cc @itazap !

[chawins]

Awesome! I confirmed that clean_up_tokenization_spaces=False fixed the issue. Thanks a lot for pointing it out.