Repository navigation
Replies: 4 comments 6 replies
|
Super cool idea. I gotta jump into this rabbit hole soon! |
|
Digging a bit. I think I kind of like this idea of passive injection of memory, more like how our mind works, human memory is not just an active recall system, it's often passive: "made me think of that". I have to look more on the converging interfaces for these systems, and try them too, i'd be very curious to review what exactly gets injected and how. Seems as useful as potential dangerous, with risks of burning tokens, noise or pure recall of bad/obsolete memories. Almost points to some active memory management to prevent undesired outcomes, unless these systems are super accurate / smart. If I had more cycles I'd make it a weekend project to do a POC, but super busy right this moment. One thing I can commit to is to try it if it ever lands in a branch. |
|
Thanks for digging in! The "made me think of that" framing is exactly why we proposed this. Two things from actually running this stuff (self-hosted Agor with teammates on a nightly cron loop, plus a separate reviewer agent that's been on Honcho for about a month): The risks you list are real; we've hit all three. Burned tokens, noise, and stale memories popping back up are exactly why our other agent uses Honcho in "tools" mode instead of passive injection: the agent asks its memory when it thinks it needs to, so cost and noise stay bounded. Passive injection feels much nicer when it works, but I'd want the interface to support both, with a token budget on the passive side to prevent it from running away. "Active memory management" is spot on, and it doesn't need the memory system to be super smart. It mostly needs boring hygiene, which we ended up building by hand: a rule that nothing becomes a "fact" until it's shown up 3+ times (one anecdote never becomes policy), a weekly cleanup pass that merges duplicates and retires stale stuff, and checking citations before trusting derived memories (we actually caught memories citing sources that didn't back them up). Honcho does part of this out of the box; the interface just needs the hooks so any backend can. Why an interface instead of a feature: even inside stock Agor today, a teammate's memory is split across two stores that don't talk, Agor Knowledge and Claude Code's own auto-memory. We had facts we explicitly taught a teammate become invisible to its own nightly loop because they landed in the wrong store. One provider surface fixes that whole class of problem, whatever backend people pick. On "lands in a branch", that's the plan. We'll post a short interface sketch first (capture hook, recall hook with a budget, per-teammate provider config, native = current behavior as default, Honcho as the reference impl) so we agree on shape before code. Any preference for whether it lives on the executor side or the daemon side? |
|
Right, would love to help define the right interface ahead of prompting for a feature. From what you said, it seems the injection point (or one of them) might be the system prompt. Though the system prompt should generally be treated as immutable, so that would limited context-injection to "right after first prompt", which prevents in-conversation "magic recall" ("... oh that reminds me about ..."). Reason why system prompt should be treated as immutable is that changing any of the context window with invalidate the cache, which isn't good. So. Does that inform pre-pending / post-pending prompts as they come? It seems heavy, like every prompt would have some sort of "let's ask the memory system whether anything here from memory is relevant", and I'm guessing that the memory sub-system probably always has something to recall or say (🔥🪙🔥!). I'm not sure how that may work. Or maybe it's an alternative button to the prompt button, like some sort of button 🧠 that goes and fetches things from memory relevant to the current context window. Personally if I'd push that button, I'd want to review, and maybe even edit the output prior to looping it back into the LLM. So. Interface. What does theses tool take as input? Like do you typically give it a single prompt, or you feed the entire context window into that thing? |
|
Good questions. We've been running this setup for real with our reviewer agent for about a month, so most of this is "what actually happens" rather than theory. "Single prompt or the whole context window?" Neither. It's two different things: Saving: after each turn, the new messages get shipped off in the background. A worker chews on them later, on a cheap model, and turns them into notes. Nothing gets added to the conversation, zero extra tokens on the hot path. The memory system has already seen all the earlier turns, so it never needs the entire window. On the 🔥🪙: yeah, memory systems always have something to say. The fix is simple: nothing gets injected unless someone asked. Default mode is the agent asking when it wants to, like any other tool. Auto-injection can exist, but opt-in and with a hard token budget. Saving costs nothing in-session because all the digestion happens later on the cheap tier. The 🧠 button is a genuinely good idea and I'd make it the third mode. In Agor terms, it's basically a widget: you press it, memory comes back with its answer, you read it, maybe edit it, then approve it into the conversation. That review step matches everything we've learned running this stuff. Memory systems are very confident about stale things, and a human check before it goes in is cheap insurance. So the sketch I'll post: observe(sessionId, peerId, newMessages) fire-and-forget for saving; query(question, scope) -> answer + sources exposed to sessions as MCP tools (same pattern as Agor's own MCP server, so no new plumbing idea); optional primer(peerId, budget) at session start; per-teammate provider config with native = what Agor does today, as the default. Saving lives daemon-side, since it owns the messages and knows who's who; recall rides the MCP tool path that already exists. I'll put the sketch on this issue. Tear it apart when it lands. |
|
So back to the interface layer, seems one part of the interface is some sort of session synchronization. As is, based on configuration, all conversation data/metadata driven by Agor would be sent to this memory repository. Unclear whether that's only configured globally and/or per user. Some potential "config hell" here, layering on top of the already complex Agor configuration. Streams? Batches? Clearly one could write an offline batch sync, where message history out of Agor would dump or get injected into that memory system. Streaming would be more challenging but potentially doable. Important question remains, is this user-driven config, or admin/global configuration or a mix of both. For active recall, seems a normal link to the external memory system, through MCP, would work. Note that Agor is already fairly self-aware (agents have access to session history, knowledge base (vector-searchable, ...)), but doesn't "dream", meaning it doesn't do active memory management outside of knowledge base indexing. One option to consider is a comparable sub-system: session indexing, fact extraction, ... That has the clear limitation of being self-scoped, only aware of itself, so if a large team uses an array of different tools, an external, centralized memory store offers advantages over Agor who would potentially only know about itself. |
|
Good framing, "session synchronization" is exactly the right name for it. Answers from how we'd build it (and partly how we already run it): Streams vs batches: batches, and small ones. Fire the sync once per completed task, carrying just the messages that task produced. Ordered, easy to retry, and memory doesn't need sub-second freshness anyway: our learning loop deliberately waits for sessions to go quiet before learning from them, because a live conversation isn't a settled one. Streaming buys nothing here and costs you ordering and failure handling. History import is then the same call at a larger size: an admin "backfill" that walks through old sessions in the same sync order. One mechanism for both. Config hell: contained by giving every knob exactly one owner. Three layers: Operator (config.yaml, where secrets already live): which provider exists, endpoint, credentials. MCP for active recall: agreed, that's how our reviewer runs today. On Agor growing its own "dream" subsystem: yes, and the seam makes that a small decision instead of a big one. The native provider can grow session indexing and fact extraction later behind the same interface, and nobody who chose an external store is affected. Your self-scoped point is honestly our strongest reason for wanting the interface rather than a feature: our memory store also listens to a separate MR-review agent, so the teammate knows things Agor never saw. Sketch drops shortly with all of this in it. |
|
Disclosure, I maintain one of the category of tool you would be plugging in here, a local first memory server for coding agents at github.com/samvallad33/vestige, AGPL, so take the rest as one data point rather than a vote for a specific provider. The distinction you are drawing between explicit knowledge and modeled understanding is the right one, and it maps to something worth deciding early, whether the provider is allowed to write back automatically or only ever gets read from. Automatic write back is where most of the actual risk lives, since a wrong inference about who someone is or what they meant becomes durable context for every future session with that person, and it can be hard to tell apart from something the team actually taught the teammate. One thing that helped for me was gating the write path on how surprising the new observation is relative to what is already stored, so routine restatements get skipped and genuinely novel or conflicting claims get flagged rather than silently accepted, and a correction supersedes the old claim instead of sitting next to it as an unrelated new fact. If it is useful while scoping the interface, happy to compare notes on where that gating approach held up and where it did not. A feature flag with nothing changing when unconfigured is the right default for something that touches modeled understanding of real people. |
|
Thanks for the disclosure and the data point; this is exactly the experience worth hearing. Agreed that the read/write split is the decision to make early. Where the sketch lands on it: The only Agor-to-provider write is the observation stream (message deltas after a task completes). Whether the provider derives inferences from that is the provider's internal policy, but two things stay in the interface because of the risk you describe: recalled claims carry provenance (taught vs derived, with citations either way, so a wrong inference can never masquerade as something the team actually said), and corrections supersede the old claim instead of piling up next to it. We enforce that second rule today with a weekly consolidation pass at the prompt level; your surprise-gating is the provider-side version of the same idea and honestly the better place for it. Would genuinely like to hear where it held up and where it didn't, especially the failure cases where "novel" and "noise" were hard to tell apart. And yes, on the default: feature flag, nothing changes when unconfigured. That's in the sketch verbatim. |
|
Here's the interface sketch as promised. Written against 0.24.5, grounded in a month of running Honcho behind a production reviewer agent plus Agor teammates on a nightly distillation loop. Tear it apart. Design constraints (from operating this, not preference)
The interfaceinterface MemoryProvider {
// Capture. Called by the daemon after a task completes, with only the
// messages that task produced. Fire-and-forget: failures are logged,
// never block the session. The provider keeps its own state, so it
// never needs the whole context window.
observe(input: {
sessionId: string;
branchId: string; // the teammate, if the branch is one
userId: string; // the human peer
messages: MessageDelta[]; // only what's new since the last observe
}): Promise<void>;
// Recall. Exposed to sessions as MCP tools (same pattern as the built-in
// agor server, so no new plumbing concept). Question in, short
// synthesized answer out. Never a page of raw memories.
query(input: {
question: string;
scope: { userId?: string; branchId?: string; sessionId?: string };
maxTokens?: number;
}): Promise<{ answer: string; citations?: string[] }>;
// Optional session-start context ("who am I talking to, what's live").
// Computed once at session creation, hard token budget, stable for the
// session's lifetime, so cache-safe. Providers may return null.
primer(input: {
userId: string;
branchId: string;
budgetTokens: number;
}): Promise<string | null>;
}Recall modes (all three are
|
| Mode | Caller | Default? |
|---|---|---|
| Tools | The agent, mid-conversation, when it decides it needs memory | Yes: bounded by agent judgment, cache-safe append |
| 🧠 button | The human; the answer comes back as a reviewable, editable block, approved into the context | Opt-in UI (a widget, basically) |
| Passive | The daemon, appending a budgeted block before a prompt | Opt-in, per teammate, hard maxTokens |
Sync shape
Batches, small ones: one observe() per completed task. Ordered, retry-friendly, and memory doesn't need sub-second freshness (our loop deliberately waits for sessions to go quiet before learning from them). History import is the same call at a bigger size: an admin-triggered backfill(scope, since) walking old sessions through observe() in order. One mechanism for both.
Config: three layers, each with one owner
| Layer | Owner | Decides |
|---|---|---|
| Deployment | operator (config.yaml, where secrets already live) | which provider exists, endpoint, credentials |
| Teammate | whoever configures the teammate (branch config, like mcp_server_ids today) |
on/off, and mode: tools only / plus primer / plus passive budget |
| Human user | the person themselves | consent to being modeled as an individual; without it the provider receives only role-anonymous data |
The third layer is a consent flag, not a tuning knob. We run exactly this split in production for GDPR reasons, and it must be enforced at the observe() call site (daemon filters before shipping), not by provider goodwill.
Write path and provenance
The only Agor-to-provider write is observe(). Whether the provider derives inferences from what it observes is provider-internal policy, but the interface keeps taught and inferred distinguishable forever:
query()answers carry provenance: each claim is taught (traceable to an explicit statement) or derived (the provider's inference), with citations either way. A wrong inference must never be able to masquerade as something the team actually said.- Corrections supersede rather than accumulate: a contradicted claim gets replaced (history kept), not filed next to its correction as an unrelated fact. Provider-side gating of the write path (skip routine restatements, flag novel or conflicting claims for review) is the right home for this, per the discussion above.
Reference implementation: Honcho
The mapping is nearly 1:1, which is what makes it a good reference backend: observe appends messages to a Honcho session (an async worker digests them on a cheap model tier), query is the dialectic endpoint (natural-language question over derived conclusions, which carry citations), primer is Honcho's token-budgeted context block. Honest labeling: we run the observe + tools-mode query halves in production today; primer and passive are capabilities we haven't exercised, and they're the part a branch would prove out.
What native does (the default)
observe is a no-op (the framework's teammate-memory tools keep working as today), query delegates to existing teammate memory / Knowledge search, primer returns null. Zero behavioral change for existing deployments. And if Agor later grows its own "dream" subsystem (session indexing, fact extraction), it grows behind this same interface as a smarter native provider, without affecting anyone who chose an external store.
Open questions
- Tenancy: provider instances per tenant, or tenant passed in scope?
- Primer vs USER.md: complement or replacement when a provider supplies one?
- Failure posture: provider down means sessions proceed memory-less with a logged warning (our vote), or fail loudly?
- Where the per-user consent flag lives so the daemon can enforce it at the
observe()call site: user preferences feel right, but the filtering has to happen before anything ships.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Teammates currently get durable memory through their Knowledge-base namespace. That works well for explicit, shared knowledge: decisions, runbooks, things the team teaches them. What's missing is a seam for modeled memory, meaning an external backend that observes conversations, builds an evolving picture of the people a teammate works with (and of what the teammate itself has learned), and injects that synthesis into context when a session starts.
This proposal adds a small
MemoryProviderinterface so teams can plug in a self-hosted backend like Honcho, Letta or Mem0 per teammate. When no provider is configured, nothing changes. I'm offering to build the interface and a Honcho reference implementation behind a feature flag.Motivation
Long-lived teammates on the Message Gateway talk to the same colleagues for weeks. Two kinds of memory matter there:
Right now the only route to the second kind is attaching a memory backend's tools via MCP. That works, but it is agent-initiated only. The teammate has to remember to remember. There is no ambient ingestion of conversations into the backend and no context injection at session start, and those two things are what make external memory feel like memory instead of a database the agent occasionally queries.
Prior art
Hermes Agent ships this as a first-class plugin system: eight external memory providers, one active at a time next to the built-in memory. When a provider is active, Hermes automatically injects provider context into the system prompt, prefetches relevant memories in the background before each turn, and syncs conversation turns to the provider afterwards. Write cadence is configurable (async, per turn, per session, or every N turns).
Two design details there translate directly to Agor:
The interface shape is also converging across the ecosystem. The same backends ship integrations for several harnesses already, so this is a proven seam.
Closer to home, a memory layer came up in #147 (the Zep sub-thread), where the open questions were where such a layer plugs into the framework and how to scope and namespace it. This proposal is one concrete answer to both: the daemon's message pipeline and context assembly as the plug points, scoped per teammate, with gateway identity as the namespace key for people.
Proposed design
Names are illustrative. Happy to be redirected on where this should live.
Wiring, based on reading the architecture docs (corrections welcome):
ingestMessagesas an after-create hook on the messages service in the daemon. FeathersJS hooks look like the natural mechanism here. Enqueued, so it never sits in the request path.getContextcalled wherever teammate context gets assembled before a session starts, prepended as its own block.Configuration per teammate, next to skills / MCP / gateway / schedule:
Non-goals:
Why this needs a small seam in core
The fair question is whether MCP tools already cover this. Partially, yes, but only in the agent-initiated form described above. The two touchpoints that make memory ambient, the message pipeline and context assembly, are not reachable from outside the daemon. Everything else can live in provider packages. So the ask is deliberately small: one seam, no change to default behavior, and providers stay out of the core path unless configured. Since these backends already ship integrations for other harnesses, a stable interface here is also likely to attract provider contributions beyond Honcho.
Offer
I'll contribute the interface, the Feathers wiring, and the Honcho reference implementation behind a feature flag, plus docs. Types centralized in
packages/coreper the contribution guide, and unit tests included since I saw those are explicitly wanted. I'd develop it in Agor itself, branch per feature on our instance, which seems to be the spirit of the house.For context: we run self-hosted Honcho in production at Voys (Dutch telecom) behind Hermes-based agents, a GitLab MR reviewer and a voicemail agent. So we'd dogfood this with a real workload. We also have GDPR and works-council review experience with exactly this class of feature, if that's useful for the docs.
Questions for maintainers:
All reactions