Use chat_template from gguf metadata

[vriesdemichael]

Is your feature request related to a problem? Please describe.
When using a model which uses a non-standard chat template it is hard to implement chat functionality using llama-cpp-python. There are quite a few chat templates predefined in llama_chat_format.py, but every time you want to add a new one it requires a new chat formatting function decorated by @register_chat_format.

When you do not have a matching chat format the output of the chat completions will be sub optimal.

Describe the solution you'd like
Huggingface has started pushing the standard to store the chat template in the tokenizer config.
The convert script in llama.cpp has started storing this chat_template too:
gguf_write_call
function to add vocab
Implementation in base model

This means that for any huggingface model with the chat_template in the tokenizer config that gets converted by llama.cpp's convert script it will have the chat_template available in the gguf metadata.

The chat template is a jinja template, the huggingface code has an example on how to apply it: https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1678
(this function is also used in hf_autotokenizer_to_chat_formatter, but the way it is implemented it requires access to the hf model/config files)

When this standard is picked up it will enable users to directly use chat completions for models with custom templates

Describe alternatives you've considered

• Disable the chat endpoint when there is no known chat_format -> no complaining about bad results
• Load the tokenizer config outside as a separate file to load the required chat_template
• supply the chat template as an argument (which is how text-generation-webui does it

[abetlen]

Hey @vriesdemichael yes finally got a chance to start on this thanks to @teleprint-me work to integrate jinja2 templating. There's now a Jinja2ChatFormatter in llama_chat_formats.py and I'm using it in #1110 to automatically pull the chat_template.

The issue right now is that the gguf doesn't supply the correct eos_token from the tokenizer_config.json for the chat format. What it includes is the eos token for the tokenizer but not for the chat format itself. ie for the chatml format I can access the jinja2 format but don't have a way to extract <|im_end|> as a stop sequence. This causes the model to start generating user tokens.

https://huggingface.co/mlabonne/NeuralBeagle14-7B/blob/main/tokenizer_config.json

Example tokenizer config for reference.

[frozen-agent]

I think it is because of a premature continue statement here:
https://github.com/ggerganov/llama.cpp/blob/381ee195721d8e747ee31a60c0751822b3072f02/gguf-py/gguf/vocab.py#L158

in the config you provided the add_eos_token is set to false, which results in skipping the eos_token in the gguf writer.

I'm not sure what the add_eos_token is supposed to do, but either the gguf writer is flawed or this tokenizer config is.

[teleprint-me]

I want to stress the importance of allowing users to define their chat templates. The current situation seems to have caused some confusion and issues with template accuracy. Even the llama templates have been found to contain errors, which is surprising given the availability of Facebook's source code.

While I understand the convenience of embedding templates into the model, I personally prefer creating and refining my own templates for my favorite models, like llama, mistral, and phi.

The introduction of template support has brought about some predictably complex challenges. However, it's essential to remember that flexibility and user control should be a priority. My approach might not align with the current design, but it was a compromise made to meet the needs of the community, which should take precedence over personal preferences.

Regarding the EOS token, it seems to be available in the model file:

18:43:58 | /mnt/valerie/llama.cpp
(.venv) git:(gguf-py | Δ) λ python gguf-py/scripts/gguf-dump.py local/models/facebook/llama-2-7b-chat-hf/ggml-model-f16.gguf --no-tensors
* Loading: local/models/facebook/llama-2-7b-chat-hf/ggml-model-f16.gguf
* File is LITTLE endian, script is running on a LITTLE endian host.

* Dumping 25 key/value pair(s)
      1: UINT32     |        1 | GGUF.version = 3
      2: UINT64     |        1 | GGUF.tensor_count = 291
      3: UINT64     |        1 | GGUF.kv_count = 22
      4: STRING     |        1 | general.architecture = 'llama'
      5: STRING     |        1 | general.name = 'LLaMA v2'
      6: UINT32     |        1 | llama.context_length = 4096
      7: UINT32     |        1 | llama.embedding_length = 4096
      8: UINT32     |        1 | llama.block_count = 32
      9: UINT32     |        1 | llama.feed_forward_length = 11008
     10: UINT32     |        1 | llama.rope.dimension_count = 128
     11: UINT32     |        1 | llama.attention.head_count = 32
     12: UINT32     |        1 | llama.attention.head_count_kv = 32
     13: FLOAT32    |        1 | llama.attention.layer_norm_rms_epsilon = 9.999999747378752e-06
     14: UINT32     |        1 | general.file_type = 1
     15: STRING     |        1 | tokenizer.ggml.model = 'llama'
     16: [STRING]   |    32000 | tokenizer.ggml.tokens
     17: [FLOAT32]  |    32000 | tokenizer.ggml.scores
     18: [INT32]    |    32000 | tokenizer.ggml.token_type
     19: [STRING]   |    61249 | tokenizer.ggml.merges
     20: UINT32     |        1 | tokenizer.ggml.bos_token_id = 1
     21: UINT32     |        1 | tokenizer.ggml.eos_token_id = 2
     22: UINT32     |        1 | tokenizer.ggml.unknown_token_id = 0
     23: BOOL       |        1 | tokenizer.ggml.add_bos_token = True
     24: BOOL       |        1 | tokenizer.ggml.add_eos_token = False
     25: STRING     |        1 | tokenizer.chat_template = "{% if messages[0]['role'] == 'system' %}{% set loop_messages"

This suggests that the EOS token isn't present and extracted from the model. It's also defined as enumerations in llama.cpp:

19:28:57 | /mnt/valerie/llama.cpp
(.venv) git:(gguf-py | Δ) λ grep -rni "eos" *.c *.h              
llama.h:645:    LLAMA_API llama_token llama_token_eos(const struct llama_model * model); // end-of-sentence
llama.h:652:    LLAMA_API int32_t         llama_add_eos_token(const struct llama_model * model);
llama.h:872:    /// @param n_predict Maximum number of tokens to predict. EOS may occur earlier.

If the EOS token appears to be missing, it might be due to issues during the conversion process, which needs further investigation. However, I'm unsure how this specific edge case would be handled in the front-end for llama.cpp.

19:29:13 | /mnt/valerie/llama.cpp
(.venv) git:(gguf-py | Δ) λ grep -rni "eos" gguf-py 
gguf-py/gguf/vocab.py:32:            self.special_token_types = ('bos', 'eos', 'unk', 'sep', 'pad')
gguf-py/gguf/gguf_writer.py:396:    def add_eos_token_id(self, id: int) -> None:
gguf-py/gguf/gguf_writer.py:397:        self.add_uint32(Keys.Tokenizer.EOS_ID, id)
gguf-py/gguf/gguf_writer.py:411:    def add_add_eos_token(self, value: bool) -> None:
gguf-py/gguf/gguf_writer.py:412:        self.add_bool(Keys.Tokenizer.ADD_EOS, value)
gguf-py/gguf/constants.py:69:        EOS_ID        = "tokenizer.ggml.eos_token_id"
gguf-py/gguf/constants.py:74:        ADD_EOS       = "tokenizer.ggml.add_eos_token"
gguf-py/gguf/constants.py:564:KEY_TOKENIZER_EOS_ID     = Keys.Tokenizer.EOS_ID

It's vital to maintain user flexibility while addressing these technical challenges. I'm still learning, juggling linear algebra, C++, the transformer architecture, and the llama.cpp source code simultaneously. The journey has been overwhelming but exceptionally rewarding.

[teleprint-me]

I've created an example script that allows for more flexible experimentation with metadata. You can check it out here.

I ran the script as follows:

python examples/low_level_api/gguf_chat_template.py local/models/facebook/llama-2-7b-chat-hf/ggml-model-f16.gguf --format

Here's what the script does:

• It successfully detected model metadata and output the available model fields, including GGUF.version, GGUF.tensor_count, GGUF.kv_count, general.architecture, general.name, and various parameters related to the model's architecture.

• It also displays information related to the tokenizer, such as tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id, and tokenizer.chat_template.

As a demonstration, I've included a sample output from the script:

[BOS][INST] <<SYS>>
I am a helpful assistant.
<</SYS>>

Hello! [/INST]

Please note that the model should start producing output from the input suffix, e.g., [/INST].

Let me know what you think.

[Rocketknight1]

Hi all, I'm the developer at Hugging Face who designed the original spec and implementation for our chat templates. I just found out about this today - we're big fans of llama.cpp, so let us know if you have any questions or you're encountering any difficulties with the templates!

[abetlen]

Hey @Rocketknight1, thank you, really appreciate any help here!

There's an open issue at ggml-org/llama.cpp#5040 but I'll include an overview here as well.

The issue we're running into at the moment is identifying the correct stop sequence(s) for end of model generation when applying a chat format. For instance in chatml there's <|im_end|> that signifies the end of a message. Is this included in the tokenizer_config.json, I've seen some models have this set as the eos_token but also include a regular </s> token so I'm a little confused.

[Rocketknight1]

Hm, I don't know if that information is actually stored anywhere! In general, base language models use eos_token to indicate when to finish generation, but as you mentioned, not all chat templates put eos_token at the end of assistant generations, so I don't know if there's any clean tokenizer property that indicates the end of assistant messages for chat models. Let me ask around at Hugging Face to see if we have a consistent way of figuring out the end of a chat generation!

[Rocketknight1]

Hi @abetlen, sorry for the delay! The situation is that standard causal language models are supposed to use tokenizer.eos_token to end their generations. However, when those models are fine-tuned for chat, the end-of-generation token may change (e.g. to something like <|im_end|>)

Our recommendation (which we're going to add to the official docs) is that when people fine-tune a model for chat, they should:

1. Add control tokens like <|im_end|> as special tokens
2. Set tokenizer.eos_token to the new end-of-generation token

However, there are some cases when this won't be possible - sometimes a model has been trained with ChatML but without adding <|im_end|> as a special token, which means it'll be tokenized as multiple tokens. This can mean the end-of-generation sequence is multiple tokens, and we don't really store that anywhere! There's nothing much you can do but manually add <|im_end|> as a string to the GGUF files in that case.

Anyway, tl;dr you should just read tokenizer.eos_token and use that to mark the end of generation. If the repo is correctly configured, then that should be all you need, but we're aware that many repos aren't right now. We'll see if we can communicate better with our users about that!

As a hack, you could also check for some common formats - for example, if the chat template contains the string <|im_end|> or [/INST] or eos_token you could probably add those as end-of-generation strings in the GGUF.

[abetlen]

@Rocketknight1 thanks yeah that was my thought as well. It does seem that tokenizer.eos_token is often correct so I'll set that to the default.

Good suggestion about using the chat_template to recognize the format, could probably even do exact string matching on the template, that should cover most of the remaining cases.