feedback

[ricardosantos79]

just a fwd, as i do not have other place to strictly discuss this. if this is breaking rules please close and archive.

i have been exploring the LMQL in python, testing how to make a conversational bot, that can stay in character and store memory, i absolutely love how the dataclass works so far and how easy is to plug it in LMQL.

https://gist.github.com/ricardosantos79/6fc2daa281281dcc49986345b12395a5

my tests with LMQL so far lead to this, but i need to figure out..
1: how to restrict the LLM to stay in character.. (asking a psychologist for a strip as a client would be a straight no..)
2: keep a "consultation diary" or a class for each patient like a archive of different users 😅 to organize/compress memory.

need a higher parameter model to run than using langchain or a prebuilt chat interface like GPT4ALL(a 7b model usually works alright for low context conversational content depending on model)..

• mxlewd20b works great(except for above issues see chat example at end of provided gist), mxlewd13b +-, did not get any 7b or less to work correctly. i am sure there could also be improvements i can do on the prompt side, but.. 😬
• for example the 13b only tries to read chat memory if you prompt it to, while the 20b does it naturally.

as a disclaimer, i do not have any background on CS/ML or alike so please forgive my bluntness, just poking around with it.
and thank you for the developers for providing this wonderful tool.

[ricardosantos79]

• i have solved (1) the memory blowing the context window by using txtai, this stores the memory in a embeddings db and searchs for a number of most similar interactions between the user and the ai restricting the active context available to the on topic conversation (probably would use some tweeks), storing the memory could be as easy as pickle it into a file and retrieving the chat logs since the chat log no longer goes in the prompt (the reason context would explode).
• on the model front since then starling7b came out and looks to be a solid model, but likes to run off on msg size (hard to make it stop at the moment)
  i updated the gist with new additions: https://gist.github.com/ricardosantos79/6fc2daa281281dcc49986345b12395a5