Llama tokenizer inconsistency for the newline character for convert_tokens_to_ids

[JackCai1206]

System Info

transformers 4.41.0
torch 2.3.0
GPU: NVIDIA GeForce RTX 4090, CUDA version 12.3

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I am trying to get the token id for the new line character for llama 3, and found this weird inconsistency. Basically convert_tokens_to_ids('\n') outputs None, but tokenize('\n') outputs 198. But then tokenizer.convert_ids_to_tokens(198) gives me Ċ

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B", padding_side='left')

id1 = tokenizer.convert_tokens_to_ids('\n')
print(id1) # None

id2 = tokenizer('\n')['input_ids'][1]
print(id2) # 198

newline_tok = tokenizer.convert_ids_to_tokens(198)
print(newline_tok) # Ċ

id3 = tokenizer('Ċ')['input_ids'][1]
print(id3) # 128

id4 = tokenizer.convert_tokens_to_ids('Ċ')
print(id4) # 198

Expected behavior

I expected the output of convert_tokens_to_ids('\n') to be 128

[itazap]

Hello! 🤗

The token 'Ċ' is actually represented by 2 tokens (after the begin_of_text token):

tokenizer('Ċ')['input_ids'] # [128000, 128, 232] which are [128,232].

Token 128 corresponds to another token:

tokenizer.decode([128]) # �

So, to decode 'Ċ' you will need:

tokenizer.decode([128, 232]) # Ċ

why you are seeing 'Ċ' as opposed to '\n':

This is related to the BPE algorithm which converts 'space' tokens like newline and tab into special characters, which may be represented by multiple bytes and results in them being represented by multiple ids, such as 2 values in this case. It is explained well in this comment here and here.

I hope I answered your questions! Feel free to reply with any further questions

[JackCai1206]

So when I use GenerationConfig, I want to initlialize it like so

GenerationConfig(
    max_new_tokens=config.max_new_tokens,
    num_return_sequences=1,
    return_dict_in_generate=True, 
    stop_strings='\n', # Doesn't work
    pad_token_id=tokenizer.pad_token_id)

The newline character as stop strings doesn't work for llama 3 because it is internally using something similar to convert_tokens_to_ids and returning None, which means the model.generate does not recognize the '\n' stop token. Right now the workaround is to do set eos_token_id=[198] in GenerationConfig, but I want to be able to use the stop_strings argument.

[itazap]

Can you please share a small reproducer?

[Acejoy]

So when I use GenerationConfig, I want to initlialize it like so

GenerationConfig(
    max_new_tokens=config.max_new_tokens,
    num_return_sequences=1,
    return_dict_in_generate=True, 
    stop_strings='\n', # Doesn't work
    pad_token_id=tokenizer.pad_token_id)

The newline character as stop strings doesn't work for llama 3 because it is internally using something similar to convert_tokens_to_ids and returning None, which means the model.generate does not recognize the '\n' stop token. Right now the workaround is to do set eos_token_id=[198] in GenerationConfig, but I want to be able to use the stop_strings argument.

Hey, were you able to get it working? I too have the same issue. I want to use the stop_strings parameter for stopping generation.