When context length is reached, we should forget example dialogue instead of recent chat

[baobabKoodaa]

Currently oobabooga seems to work such that when context length is reached, the chatbot starts forgetting oldest parts of recent chat. It would be better if it started forgetting oldest parts of example dialogue. It doesn't make sense to prioritize example dialogue above recent chat. Changing this behavior would have the following benefits:

1. Users would experience chatbots as having better, more long-term memory
2. Character creators would be able to create more immersive characters, since they wouldn't be self-limited to 600-1000 tokens anymore. After this change it would be perfectly safe to create characters that consume even 1500 tokens of context.

[ghost]

I would love having this as a user-configurable option.

For Pygmalion models I already do this manually by placing the example dialogue in the chat itself rather than have it in the yaml. The example dialog is really only intended as a primer for the model to learn the intended writing style and background, and it's not required after the chat has gone on and the context is full.

[miles-du]

I would love having this as a user-configurable option.

For Pygmalion models I already do this manually by pasting the example dialogue in the chat box rather than have it in the yaml. The example dialog is really only intended as a primer for the model to learn the intended writing style and background, and it's not required after the chat has gone on and the context is full.

Didn't think of that eiery, great tip! I have been getting by trying to thin out the existing chat when I start a new day, but your idea is much better. Thanks.

[ossirytk]

I somewhat disagree with this opinion. What currently irks me the most is the standard the most community sites use for the character file. I guess this is due to compatibility issues, but nearly all of the files have duplicate fields. This causes the character file to take double the size. The descriptions are also often very verbose without really needing to be. It's so unbearably bad that I rewrote all of the character files I downloaded. I'm extra triggered that at the same time there is a trend of writing without line breaks in these files.

[baobabKoodaa]

The file size is insignificant, because we're talking about a kilobyte or something. What matters is the length of context that is passed to LLM, and those duplicate fields will not be sent twice inside the context.

Even if you rewrite all the character files to be very small, you still run into the context length issue once you start chatting. At some point the chatlog will grow so much that you reach the max context length, and at that point you have to make choice about which parts you need to cut out. You mentioned that you "somewhat disagree", indicating that you would prefer to cut out the beginning of the prompt that has character and scenario info, as opposed to cutting out example dialogue, but you didn't really explain why you think this.

[ossirytk]

I was unaware of how the files were parsed, so I can admit being wrong here. I thought that the whole file gets parsed and the fields for user and character would get filled to the final prompt. I guess I confused this for when I ran llama.cpp from terminal, since it does no parsing at all, and so the context can get overfilled. If the parser in webui can tell the difference between the duplicate fields then it's not an issue.

For the descriptions I just think that the w++ style pseudo code for the character description is very compact and effective. I find that it's better tokenwise than writing in natural language. On the other hand I think that example dialogue has a large impact in the way that the model handles responses especially with storytelling and role play models.

I usually use a setting that keeps the initial prompt in memory indefinitely just so the model remembers the context and the example dialogue. I think this gives better results than remembering more lines of dialogue.

[baobabKoodaa]

You can launch oobabooga with the --verbose option so you can actually see what gets placed into the prompt that is sent to the LLM. This will allow you to check if any duplicate stuff is being sent.

[ossirytk]

Thanks man. I have been looking for this. Guess my first instinct of writing print lines in the server code wasn't the best one.