How to convert to gguf format with tokenizer.json file?

[LiamHoang26]

I have tried to convert llama-2-7b model to GGUF format to deploy with llama.cpp. But they do not include tokenizer.model file which is needed to convert process. But they have tokenizer.json file. So Is there any method to use tokenizer.json file to create model in GGUF format? If not, is there any way to generate tokenizer.model file?

Many thanks.

[goerch]

Please try the --vocabtype bpe option.

[KerfuffleV2]

Shot in the dark: what happens if you replace vocab.json here with tokenizer.json?

I don't think that will work, but you can try to generate vocab.json from tokenizer.json. For example, save this mini Python script to a file:

import json, sys
tokenizer = json.load(sys.stdin)
json.dump(tokenizer['model']['vocab'], sys.stdout)

then run python whatever.py < tokenizer.json > vocab.json (commandline example is for Linux/unix type OSes, may need to be modified for Windows, etc).

[ChaosCodes]

Hi, I have met the same issue, but if I add new special tokens during fine-tuning, the added_tokens will be stored in tokenizer['added_tokens'] rather than tokenizer['model']['vocab']. It seems such way to generate vocab.json will lost my new special tokens.

[KerfuffleV2]

It seems such way to generate vocab.json will lost my new special tokens.

There's separate logic to try to extract added tokens, so it may still work (that part will try to read tokenizer.json if I remember correctly). How well it actually works is a different question though. :)

[LiamHoang26]

Shot in the dark: what happens if you replace vocab.json here with tokenizer.json?

Hi @goerch
I tried to follow your work around, and I met this exception

I also tried to use @KerfuffleV2 script to generate vocab.json file from tokenizer.json => This solution might work, and I could use vocab.json file with --vocabtype bpe option to convert model to gguf format successfully.
However, when try to inference with llama.cpp. I meet error with log:

GGML_ASSERT: llama.cpp:2029: codepoints_from_utf8(word).size() > 0
Aborted (core dumped)

Could anyone give me any solution, please?
Many thanks.

[goerch]

I'm sorry for that. In the meantime I learned that some (older?) models like GPT2 only provide something like vocab.json, so we probably try to keep compatibility here, although I'd prefer to use tokenizer.json, because it seems to contain a more complete description. Regarding your actual question: I'd like to check that but I'm currently overwhelmed by #3502.