[Phi-3-mini-128k-instruct] Difference of encodings for Slow and Fast Tokenizer

[GKIBMNY]

System Info

• transformers version: 4.46.2
• Platform: Linux-5.10.223-212.873.amzn2.x86_64-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.26.2
• Safetensors version: 0.4.5
• Accelerate version: 1.1.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.4.0a0+3bcc3cddb5.nv24.07 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: No
• GPU type: NVIDIA L40S

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-128k-instruct', use_fast=True)
text = '<|user|>\n I am good <|end|>\n<|endoftext|>'
ids=tokenizer(text).input_ids
print(ids)
print(tokenizer.convert_ids_to_tokens(ids))
print(tokenizer.encode(text))

tokenizer = AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-128k-instruct', use_fast=False)
ids = tokenizer(text).input_ids
print(ids)
print(tokenizer.convert_ids_to_tokens(ids))
print(tokenizer.encode(text))

Expected behavior

In expected case, both slow and fast tokenizer should return the same IDs

[atharva-navani16]

Hey @GKIBMNY try using the following script,

Try setting legacy=True when loading the tokenizer:

from transformers import AutoTokenizer

model_name = "microsoft/Phi-3-mini-128k-instruct"
text = "<|user|>\n I am good <|end|>\n<|endoftext|>"

# Using Fast Tokenizer
fast_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
fast_ids = fast_tokenizer(text).input_ids
print("🔵 Fast Tokenizer IDs:", fast_ids)
print("🔵 Fast Tokenizer Tokens:", fast_tokenizer.convert_ids_to_tokens(fast_ids))
print("🔵 Fast Tokenizer Encoded:", fast_tokenizer.encode(text))

# Using Slow Tokenizer without legacy=True
slow_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False, legacy=False)
slow_ids = slow_tokenizer(text).input_ids
print("🟢 Slow Tokenizer (legacy=False) IDs:", slow_ids)
print("🟢 Slow Tokenizer (legacy=False) Tokens:", slow_tokenizer.convert_ids_to_tokens(slow_ids))
print("🟢 Slow Tokenizer (legacy=False) Encoded:", slow_tokenizer.encode(text))

# Using Slow Tokenizer with legacy=True
slow_tokenizer_legacy = AutoTokenizer.from_pretrained(model_name, use_fast=False, legacy=True)
slow_legacy_ids = slow_tokenizer_legacy(text).input_ids
print("🟠 Slow Tokenizer (legacy=True) IDs:", slow_legacy_ids)
print("🟠 Slow Tokenizer (legacy=True) Tokens:", slow_tokenizer_legacy.convert_ids_to_tokens(slow_legacy_ids))
print("🟠 Slow Tokenizer (legacy=True) Encoded:", slow_tokenizer_legacy.encode(text))

Output:
🔵 Fast Tokenizer IDs: [32010, 306, 626, 1781, 29871, 32007, 32000]
🔵 Fast Tokenizer Tokens: ['<|user|>', '▁I', '▁am', '▁good', '▁', '<|end|>', '<|endoftext|>']
🔵 Fast Tokenizer Encoded: [32010, 306, 626, 1781, 29871, 32007, 32000]
🟢 Slow Tokenizer (legacy=False) IDs: [29871, 32010, 306, 626, 1781, 29871, 32007, 32000]
🟢 Slow Tokenizer (legacy=False) Tokens: ['▁', '<|user|>', '▁I', '▁am', '▁good', '▁', '<|end|>', '<|endoftext|>']
🟢 Slow Tokenizer (legacy=False) Encoded: [29871, 32010, 306, 626, 1781, 29871, 32007, 32000]
🟠 Slow Tokenizer (legacy=True) IDs: [32010, 29871, 306, 626, 1781, 29871, 32007, 32000]
🟠 Slow Tokenizer (legacy=True) Tokens: ['<|user|>', '▁', '▁I', '▁am', '▁good', '▁', '<|end|>', '<|endoftext|>']
🟠 Slow Tokenizer (legacy=True) Encoded: [32010, 29871, 306, 626, 1781, 29871, 32007, 32000]

[ArthurZucker]

Yep! legacy is the most important flag for the behaviour between special tokens and tokens right after them!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.