Repository navigation
Having trouble getting Hindsight to retain "experience" memories #6523
|
I'm running Hindsight with a local Ollama model, and it works great, except the only problem I'm having is it's really inconsistent about storing "experience" memories. The way I'm retaining messages is raw JSON conversations just like it suggests in Best Practices:
And I set the context field as "Conversation between user and [name of bank] about [concise topic]." I've also tried "Conversation between user and assistant..." and "Conversation between me and user..." but it's always inconsistent. Very rarely it stores a fact as an experience, and usually it just stores it as a world fact "Assistant told user X" or "Assistant did X." I'm only using a 9b model, so maybe that's the issue. But like I said, everything works great except for this one thing. Any advice? Thank you! |
Replies: 3 comments 7 replies
|
A few things are working against you here, and the 9B model is only part of it. 1. Hindsight doesn't know the assistant is the bank. In the extraction prompt, "experience" facts are what it calls Spell out the role mapping explicitly: 2. Put the same rule in the bank's curl -X PATCH http://localhost:8888/v1/default/banks/<bank_id>/config \
-H 'Content-Type: application/json' \
-d '{"updates": {"retain_mission": "This bank is the memory of the AI assistant in these conversations. Whatever the assistant said, recommended, explained, decided or did is its own first-person experience and should be extracted as an assistant fact, not a world fact."}}'3. Use a bigger model just for retain. Also worth knowing: user facts are supposed to be |
|
If you retried with the same On v1.2.1, you can check this with
|
|
Update: I added "context" like "Log is dated with the dates that entries occurred at" and that helped extract occurred at dates, so that seemed to work. I'm unsure if world facts extracted from conversations with a timestamp deserve an "occurred at" date -- I'm leaning toward "no." For example, something like "The user is looking into X." Maybe I mentioned that on a certain date, but I'm not sure that fits the Hindsight intended use-case for "occurred". |
A few things are working against you here, and the 9B model is only part of it.
1. Hindsight doesn't know the assistant is the bank. In the extraction prompt, "experience" facts are what it calls
assistantfacts: things the agent whose memory this is actually did, recommended, decided or answered, ideally first person ("I suggested X"). Retain no longer passes a narrator or the bank name into the extraction prompt (it was removed because bank names were leaking into stored facts). So the only way the model learns thatrole: "assistant"means "me" is yourcontext. With "Conversation between user and [bank name]", a small model just sees two third parties, which is how you end up with world…