RFC: extension surface for processAgentResponse (middleware) — related to #2332

[cmyoya]

RFC: extension surface for processAgentResponse (middleware) — related to #2332

Posting as a Discussion because this repo restricts PR creation to collaborators. The patch is implemented and validated; the actual diff lives on a fork branch (link below) and can be inspected, fetched, or cherry-picked directly. Happy to open a real PR if granted access.

Branch with the patch (one commit, applies cleanly to current main @ 13d5fa71c):
https://github.com/cmyoya/claude-mem/tree/feat/response-processor-middleware

Single commit: 48129359 — 4 files changed, 251 insertions, 1 deletion.

---

Motivation

processAgentResponse() in src/services/worker/agents/ResponseProcessor.ts is the single point where parsed observations land before being persisted via sessionStore.storeObservations(...). Today there is no way for an external consumer to inspect, mutate, or reject a parsed batch at this point without forking the file. This proposal adds a small in-process middleware surface so consumers can:

• Run dedup or scope checks that go beyond the byte-exact content_hash (src/services/sqlite/observations/store.ts) — e.g. semantic dedup, plan-scoped file-path checks (related: Observer SDK session has filesystem-write tools despite system prompt declaring "no access to tools" — autonomously edits user source files #2332).
• Validate provenance before write — e.g. require that an observation citing a files_modified path is within an allowed plan scope.
• Plug in attestation — e.g. verify a referenced git tag exists before treating an observation as a closure record.

No existing consumer is degraded; default behavior is unchanged.

API

// src/services/worker/agents/middleware.ts
import type { ParseResult } from '../../../sdk/parser.js';

export type ValidParseResult = Extract<ParseResult, { valid: true }>;

export interface MiddlewareContext {
  memorySessionId: string;
  project: string;
  agentName: string;          // 'SDK' | 'CLI' | future agents
  contentSessionId: string;
  sessionDbId: number;
}

export type ResponseProcessorMiddleware = (
  parsed: ValidParseResult,
  ctx: MiddlewareContext,
) => Promise<ValidParseResult | null>;

export function registerResponseProcessorMiddleware(mw: ResponseProcessorMiddleware): void;
export function unregisterAll(): void;

Middlewares execute FIFO. A middleware returning null rejects the batch — the rest of the chain is not invoked, no storeObservations call is made, and the claimed messages are confirmed (so they are not retried as if the worker crashed). A middleware returning a ParseResult passes the (possibly mutated) result to the next link.

A middleware that throws is treated as a rejection (logged at error, batch dropped). This is the conservative choice — silently passing the batch on a thrown gate would defeat the gate's purpose.

Default behavior

Empty array. No middleware is registered out-of-the-box. There is no env flag, no settings.json key — registration happens in code by an opt-in importer. This keeps the default install identical to today.

Backward compatibility

• No public API removed or renamed.
• processAgentResponse() signature unchanged.
• With zero middlewares registered, behavior is byte-identical to current main (verified by the no-op fast path: if (middlewares.length === 0) return parsed;).
• No new runtime dependencies.

Files touched

File	Change
src/services/worker/agents/middleware.ts	New. The surface itself: types, register/unregister, FIFO runner. ~98 LOC.
src/services/worker/agents/ResponseProcessor.ts	Insert runResponseProcessorMiddlewares(...) between parse-validation and storage. ~17 line addition.
src/utils/logger.ts	Add 'MIDDLEWARE' to the Component union (one line) so the new module can log under its own component tag.
examples/ddouns/consumer.ts	New, reference only. A first consumer that demonstrates the API (semantic-prefix dedup + closure-tag attestation). Lives outside src/ and outside the tsconfig.json include glob — does not ship in the npm package, does not affect claude-mem runtime, does not introduce a DDouns dependency.

Comparison: OpenCode HTTP plugin pattern vs in-process middleware

OpenCode's plugin model uses HTTP fire-and-forget hooks. That has two costs we cannot accept here:

1. Latency on the hot path. processAgentResponse runs per agent response in the worker. An HTTP roundtrip per batch is a regression we will hear about.
2. Failure mode is silent. Fire-and-forget cannot reject a write. For consumers that need to gate writes (dedup, scope, attestation), HTTP is the wrong shape.

In-process middleware is synchronous-by-design and can return null to reject. Consumers that prefer HTTP can still implement a middleware that calls out — that is their choice, not the framework's default.

Validation

Validated against main @ 13d5fa71c204bbb5fac79fd2052ec85f59666e98 on Windows + bun 1.3.13:

Gate	Result
git apply --check (both patches)	clean
tsc --noEmit (root + viewer)	24 baseline errors, 0 in patched files
bun test tests/worker/agents/	49 pass / 4 skip / 0 fail / 314ms

The examples/ddouns/consumer.ts is outside the tsc include glob and does not contribute to typecheck.

Suggested tests (not yet shipped)

Three suggested unit tests at tests/worker/agents/middleware.test.ts (using the existing bun:test style as in tests/worker/agents/response-processor.test.ts):

1. Empty array is no-op. With no middlewares registered, processAgentResponse storage path runs exactly as today (assert storeObservations called once with the original parsed batch).
2. Mutation propagates. A middleware that prepends a marker to every observation's title results in the SQL row carrying that marker.
3. Reject short-circuits. A middleware returning null results in zero storeObservations calls and a confirmed-message state (no retry loop).

Happy to add them as a follow-up commit on the same branch.

How to inspect locally

git clone https://github.com/cmyoya/claude-mem
cd claude-mem
git checkout feat/response-processor-middleware
bun install
bun test tests/worker/agents/   # 49 pass / 0 fail

Or compare directly:
main...cmyoya:feat/response-processor-middleware

Note on consumers

DDouns (https://github.com/cmyoya/DDouns) is the first known consumer and is what motivated this proposal. The patch ships without any DDouns dependency — the middleware surface is generic and useful for any plan-scoped, dedup, or attestation extension. The DDouns gate is a separate package that registers itself via the API above; it is not bundled here. The examples/ddouns/consumer.ts file is a non-shipping reference of how a consumer would wire in.

Open to redesigning the API shape if a different signature better matches the project's direction. The branch is the proposed concrete; the discussion is the negotiation.