Unexpected behavior in ChatSession.ChatAsync methods #261
Description
I've been trying to create a chatbot with the ChatSession class and IHistoryTransform implementation to use the correct template for the LLM I am testing LLamaSharp with. I ran into unexpected behavior and could not get good responses from the model.
The only way I got a decent response was when I sent the whole history as a prompt with the correct formatting. Saving and loading the session would lead to ever growing context and significant slowdown.
I noticed that the use of the IHistoryTransform interface in the ChatSession class is leading to these unexpected results. When the ChatSession class was refactored to use the IHistoryTransform interface, the behavior changed dramatically by parsing the text in the prompt argument and adding the prompt (which may be the whole history) into the history of the session.
It's a little bit hard to untangle. I forked the repo and changed the behavior to be a bit more straightforward. With the overload method that accepts a string prompt managing the internal history for the user and the overload that accepts a ChatHistory argument allowing the user to manage the history themselves.
See here: master...philippjbauer:LLamaSharp:master
I see many more improvements here, especially in how the IHistoryTransform interface could be changed to accommodate better templating support. Making use of public properties for role tokens and end tokens so that these tokens can be replaced internally before adding the generated message to the history instead of relying on the inferenceParam.AntiPrompt property.
If this is missing the mark completely let me know, but I couldn't figure out how this should be used after reading the documentation and looking at the code.
Edit: A little bit lengthy but here is my IHistoryTransform implementation:
public class HistoryTransform : IHistoryTransform
{
private string _nl = Environment.NewLine;
private string _roleTemplate = "<|{0}|>";
private string _endToken = "</s>";
private string _regexPattern = @"(?:<\|([a-z]+)\|>\n)([^<]*)(?:</s>)";
public string RoleTemplate => _roleTemplate;
public string EndToken => _endToken;
public HistoryTransform(
string? roleTemplate = null,
string? endToken = null,
string? regexPattern = null)
{
_roleTemplate = roleTemplate ?? _roleTemplate;
_endToken = endToken ?? _endToken;
_regexPattern = regexPattern ?? _regexPattern;
}
public string HistoryToText(ChatHistory history)
{
string text = "";
foreach (Message message in history.Messages)
{
text += FormatMessage(message);
}
AuthorRole nextRole = history.Messages.Last().AuthorRole == AuthorRole.User
? AuthorRole.Assistant
: AuthorRole.User;
text += $"{FormatRole(nextRole)}{_nl}";
return text;
}
public ChatHistory TextToHistory(AuthorRole role, string text)
{
ChatHistory history = new();
Match match = Regex.Match(text, _regexPattern);
string message = match.Groups[2].Value.Trim();
history.AddMessage(role, message);
return history;
}
// Not part of interface, use in program to load saved history
public ChatHistory FullTextToHistory(string text)
{
ChatHistory history = new();
MatchCollection matches = Regex.Matches(text, _regexPattern);
foreach (Match match in matches.Cast<Match>())
{
AuthorRole role = match.Groups[1].Value switch
{
"system" => AuthorRole.System,
"user" => AuthorRole.User,
"assistant" => AuthorRole.Assistant,
_ => AuthorRole.System
};
string message = match.Groups[2].Value.Trim();
history.AddMessage(role, message);
}
return history;
}
public string FormatRole(AuthorRole role)
{
return string.Format(_roleTemplate, role.ToString().ToLower());
}
public string FormatMessage(Message message)
{
return $"{FormatRole(message.AuthorRole)}{_nl}{message.Content.Replace(_endToken, "")}{_endToken}{_nl}";
}
}The FullTextToHistory method is a crutch to load the history I save alongside the session data. I think this should be handled by the ChatSession class and saved as a JSON formatted file or similar for easy portability when necessary.