Chat ML settings

[kajuberdut]

I read through the code for how User String and Bot String are used to try and make sure I'm setting them in a sane way for ChatML:

The ChatML format looks like:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
{prompt}<|im_end|>

a chatML model will add <|im_end|> for the end of generation so I don't think that needs to be explicitly in the User String but a user won't want to manually add <|im_end|> when using quick instructions so I added it to the start of the Bot String.

Does this seem like the correct "Custom" config for ChatML?

[bridgesense]

There's actually a bug here. The User and Bot string are switched around.

This is an amazing extension!

[bridgesense]

It could just be that way in Notebook B. It's a bit confusing but not a real issue. I just started using this, today. I suppose I should clarify, that when using the custom string, the response calls the user what is present in the Bot string.

[bridgesense]

Nevermind. It finally dawned on me what's happening here. This notebook is really geared for dialogue. You can simple tell it, who's who during the custom instructions --- ...