TheBloke's future models won't work with LocalAI

[no-skynet]

Having found models from TheBloke to be particular well documented and easy to work with, I have basically stopped looking at any other. I even built my own TheBloke filtered gallery.

This seemed to work well, for a while at least. However, The Bloke recently switched to using ChatML for all future models.

See here:
https://www.google.com/search?hl=en&q=%22All%20of%20my%20future%20models%20will%20use%20ChatML%20prompt%20format%22
"All of my future models will use ChatML prompt format" - TheBloke

...

Originally posted by @no-skynet in #67 (comment)

[mudler]

not really, LocalAI supports (thanks to the @dave-gray101 contribution ) chat message template. this doesn't look like much different from llama2 and can be accomplished among the lines. See also the model gallery with a template example: https://github.com/go-skynet/model-gallery/blob/main/llama2-7b-chat-gguf.yaml

[no-skynet]

So we can build a ChatML template that will work with OpenAI's Chat Markup Language (ChatML) format, with <|im_start|> and <|im_end|> tokens?

[mudler]

So we can build a ChatML template that will work with OpenAI's Chat Markup Language (ChatML) format, with <|im_start|> and <|im_end|> tokens?

yes definetly!

[mudler]

so if I'm reading correctly a chat template might be look like this, but note, I've didn't tried it yet:

<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>

[lunamidori5]

so if I'm reading correctly a chat template might be look like this, but note, I've didn't tried it yet:

<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>

GGUF does this auto, it would be use the one on the demo page if using a GGUF model, love the new meta data in the model <3

[mudler]

so if I'm reading correctly a chat template might be look like this, but note, I've didn't tried it yet:

<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>

GGUF does this auto, it would be use the one on the demo page if using a GGUF model, love the new meta data in the model

this is just for chat models and not instructors for what I can see - I doubt the prompt is injected inside the model file

[dave-gray101]

good to know about this, I'm in the process of improving the model gallery scraper right now, and I'll add this to the list of things to test for

[no-skynet]

@dave-gray101, another thing to look out for... the last time I checked out the scraper, it seemed to be downloading *.bin files only, however TheBloke has switched to *.gguf files. In my own gallery scraper I added support for both and found an extra 400+ new models from TheBloke 😅

[scenaristeur]

ggml seems deprecated on TheBloke models , (https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-GGML)

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

It seems like the issue is related to TheBloke's decision to switch to using ChatML for their future models. This means that any existing models that were built using a different format may not work correctly with the new version of LocalAI.
To address this issue, you could try updating your model to use the ChatML prompt format, or you could consider using a different model that is already compatible with the latest version of LocalAI.
If you need further assistance in debugging the issue or updating your model, feel free to provide more details or ask for help in the issue tracker or Discord channel.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/CONTRIBUTING.md
• https://github.com/go-skynet/LocalAI/tree/master/examples/README.md
• https://github.com/go-skynet/LocalAI/tree/master/README.md
• https://github.com/go-skynet/LocalAI/tree/master/api/api_test.go