-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple sessions on one model. #41
Comments
At the moment the model weights and the context are all bound together in one object. You need to save and restore "states" to have two contexts in one set of weights. This is due to how llama.cpp itself used to work. They've since made a change which splits model weights and model contexts into two separate things, so you can make multiple contexts from one set of shared weights. My PR (#64) partially addresses this by adding in support for the new loading system. Future PRs will modify the higher level APIs to use this. |
Thanks for the response. I am trying it right now but my responses seems to lose some context. I assumed that when a state is loaded / saved it retains the chat and prompt history, or am I mistaken? |
I tried including the chat history after loading the session again aswell but this seemed to reset the "memory" of the previous conversation aswell, |
Is this testing all being done on top of my PR (#64), with |
Not yet, this was my next course of action. Was hoping I understoot the behaviour correctly first. |
I'm not too sure, sorry. I've been contributing PRs on some of the lower level bits of the stack but not the "higher level "stuff yet. I do know there are a few layers, which should all save and reload state together (executor, context etc), so maybe try backtracing some of that to check it all looks reasonable. |
Thanks, I just build your PR and it seems to work better without changing a lot of code which is great! Might be the wrong place to ask but I couldn't find it in sourcecode. Do you know if LLamaSharp adds things like 'Assistant:', and 'User:' to the chat? |
As far as I know it does not, but that'd be in the higher level parts that I'm not too familiar with so I'm not too sure on that! |
0.4.2 is out now. Does that resolve this issue? |
Hi there, would love to have multiple sessions on the same model but th sessions seem to remember new information given by the other chat sessions. In the docs and settings I couldn't find anyhting. I am curious if this is somemthing I am doing wrong?
The text was updated successfully, but these errors were encountered: