How we share long-term memory across AI agents from different vendors — feedback & criticism welcome

[indamices]

How we share long-term memory across AI agents from different vendors — would love your feedback and criticism

TL;DR: We open-sourced our cross-agent shared memory layer (cross-agent-memory). It's been running in production for months with ~2M chunks across our agent fleet. We're sharing it here specifically because memory seems to be one of the most requested features in this community, and we'd genuinely like your criticism: what's wrong with our design, what would you do differently, and how it could (or couldn't) work with DeepSeek Harness.

Why we built this

We run a small fleet of AI agents (built on Claude Code, OpenCode/GLM, and a few orchestration sessions) that maintain a quantitative trading system together — collecting market data, running backtests, monitoring 270+ cron jobs, writing reports. The recurring problem: every session starts with amnesia. Lessons learned by one agent evaporate; the next agent repeats the same mistakes; context files grow until they get truncated.

We noticed discussion #14 on long-term memory has 50+ comments and multiple memory plugin requests. The Super-agent article also names cross-session memory as a scalability bottleneck. So we thought: this is the right community to poke holes in what we built.

What it does (short version)

• One memory service, many agents: a local HTTP service (localhost:5370) that any CLI agent can query — Claude Code hooks, OpenCode/GLM sessions, cron-driven scripts, whatever. Agents stop being the unit of memory.
• Provenance and weights on every entry: each memory carries its source, timestamp, and a trust weight — so an agent can decide whether to believe it before acting on it. This was our answer to "stale memory is worse than no memory."
• Three-tier storage (hot / warm / cold): recent, frequently-used memories stay hot; older ones cool down to cold storage automatically. A daily health check verifies the rotation actually happens (we learned the hard way that "we configured rotation" ≠ "rotation works").
• ~2M chunks across file/session domains, with sqlite-vec for retrieval.
• Behavior rules that survive context loss: the four rules every agent must follow when writing to shared memory (e.g., "cite your source or don't write").

What we're NOT claiming

To be clear about scope, because overselling helps nobody:

• We have not solved agent-to-agent task coordination (queuing, dispatch, transactions). Reading the Super-agent architecture doc, your effects queue design is well ahead of anything we have — we're studying it.
• Our system is Chinese-language-first; English docs are basic.
• It's a monorepo tied to our trading system's layout; extracting a standalone package is on our TODO list, not done.

Questions we'd love your input on

1. Retrieval quality vs. scale: at ~2M chunks, our naive vector search works, but we worry about 10M+. If you've built memory systems at scale, where did naive retrieval break for you?
2. Trust weights: we hand-tune source weights. Is there a principled way to derive them (usage feedback? contradiction detection? something else)?
3. The "stale memory" problem: how do you decide when a memory should be deleted rather than just deprioritized? We currently never delete, which feels wrong long-term.
4. Integration with Harness: has anyone here wired an external memory service into DeepSeek Harness sessions? The session-persistence seam looks related but we haven't studied it deeply — pointers welcome.
5. Anything we're obviously doing wrong: honestly, the blunter the better. We'd rather hear it here than discover it in production.

Links

• Repo: indamices/cross-agent-memory (also mirrored on Gitee for users in China)
• A 2-minute demo video showing the memory service in action

Thanks for reading. Fire away.

[ajdelaguila]

On your third question, I think "delete or deprioritize" is missing a third state that does most of the work, which is "replaced". A memory that went stale because a newer one contradicts it is a different case from one that is merely old, and they want different handling. When the newer entry names the one it replaces, the old one can drop out of anything an agent acts on while staying retrievable by id, so an agent that later meets a reference to it can see what replaced it. It also helps to require that only an entry which is itself current can retire another, so a draft or an expired note cannot knock out a live one. Age-based cooling is then left for what nobody has replaced, and deletion for what was never true or must not be kept, which is a much smaller set than "old".

That takes pressure off the trust weights too. A hand-tuned weight is being asked whether something is still true, which no weight can answer; an explicit replacement link answers it directly, and the weight only has to rank among entries that are all current.

I maintain Data Olympus, which works this way for team rules and decisions rather than session memory, so it is a different layer from yours. A document declares supersedes, in-force queries exclude whatever an in-force document has replaced, and a lookup by id still returns the old one with its successor. The rules are in section 4.2 of the spec if the shape is useful: https://github.com/knaisoma/data-olympus/blob/main/SPEC.md

[antropos17]

I'd test a high-weight memory that asks the next agent to change its own hooks or permissions. Provenance helps explain where that text came from, but shouldn't authorize the change. I maintain AEGIS; local profile review is one checkpoint for such files. It doesn't authenticate your memory service or resolve instruction meaning.

Drafted with AI assistance.