memory: live write/forget tool paths fail in workflows evals (previously masked by vacuous fixtures)

[dcramer]

Summary

5 of 11 cases in packages/junior-evals/evals/memory/workflows.eval.ts are red. The evals themselves have been audited and are correct (see below) — the reds document real misbehavior on the live memory write/forget tool paths: during the turn the agent narrates that persistence failed or is unavailable ("nothing was written to an external store", "the memory skill isn't available", "could not remove the preference"). Passive extraction, recall/injection, and read paths work.

Failing cases (post-audit):

1. explicit first-person preference → agent says memory not persisted (rubric fail condition "do not say the memory failed to save" fires correctly)
2. first-person opinion with rewritten wording → agent explicitly refuses persistence
3. explicit shared task procedure → 0.50, frames shared knowledge as a "preference" (borderline)
4. passive operational knowledge recall → 0.50, first claims no relevant memory then recovers (partial)
5. forget a remembered preference → agent says it could not remove it

Why "green on main" was an illusion

The file had the same fixture bug as multi-actor-provenance.eval.ts (fixed on the #773 Track 1 branch): label-suffixed thread_ts values (e.g. "17000000.memory-explicit") fail parseSlackMessageTs, so the runtime keyed memories by per-message messageTs while the eval's readMemories filtered on the thread-based sourceKey. Every positive DB assertion could never match and every no-store assertion passed vacuously. The fixtures are now valid (17000000.000001…000011), the assertions are meaningful for the first time, and one case ("organic conversation teaches a task procedure") went green purely from the fixture fix.

Classification (provisional)

The live tool wiring (tools.ts, plugin tool registration, live requester derivation) is untouched by the #773 Track 1 branch — only passive extraction, session-log provenance, the run projection, and the actor rename changed. So these reds are most consistent with a pre-existing bug that vacuous assertions had been masking, not a #773 regression. Definitive verdict: run cases 1 and 5 on the committed baseline (stash or a main checkout) — that is the first step of this issue.

Eval audit notes (already applied on the branch)

• Fixtures: fixed (valid Slack ts).
• DB assertions: now non-vacuous, scoped by sourceKey/scope/subjectType — contract-level, no extraction-internal leakage (no overfitting).
• Rubrics: fail conditions are precise and fired against genuine non-persistence narration, not over-read.
• Scenarios: realistic and satisfiable.

Per maintainer policy: red-but-correct evals stay red; do not relax them to green. Fix the product behavior.

First steps

1. Baseline run of cases 1 and 5 against main to confirm pre-existing vs Audit actor/instruction provenance across plugins #773-branch regression.
2. Instrument the live createMemory/removeMemory tool path in the eval environment (why does the agent see the tool as unavailable/failing?).
3. Fix the product path; cases must go green without touching the eval file.

Found during #773 Track 1 verification.

[dcramer]

Deferred tool discovery plan

Current working hypothesis: the memory workflow failures are primarily a discoverability problem, not necessarily a memory storage failure. Memory tools are registered as deferred plugin tools, so the model only reaches them if it already knows to call searchTools and then executeTool. That path is too implicit.

Proposed direction:

1. Keep the deferred execution model. Plugin tools should not all become native model tools. Junior should continue exposing searchTools and executeTool directly for now.
2. Introduce source as the stable grouping concept for deferred catalog tools. Examples: memory, future mcp:sentry, future mcp:linear. Avoid separate model-facing terms like namespace/domain/owner for the same concept.
3. Update the searchTools description to advertise deferred sources, not every tool. Example:

Deferred tools are grouped by source. Use searchTools with source to inspect one source, then executeTool with the exact returned tool_name.

Available sources:
- memory: Long-term memory storage, recall, listing, and removal.

4. Add a source?: string | null filter to searchTools alongside query and max_results.
5. Define behavior:
   • omitted/null source: search all deferred catalog tools;
   • source: "memory": search/list only memory tools;
   • empty query with a source: list tools in that source;
   • unknown source: return a structured empty result with known sources;
   • empty query without source: return sources and only a bounded sample, not the whole catalog.
6. Keep results compact. Return source summaries once at the top level. Omit per-tool source when searching a single source. For cross-source searches, include only compact per-tool source: "memory", not expanded plugin/MCP metadata.
7. Auto-summarize model-visible descriptions: normalize whitespace, prefer the first line/paragraph before a break, and hard cap around 160-200 chars. Full descriptions can still be used for search indexing.
8. Carry source metadata from plugin registration into the executable catalog. Memory should expose source id memory with a short description like Long-term memory storage, recall, listing, and removal.
9. Align this with the future MCP unification path: MCP providers can become sources (mcp:<provider>) in the same catalog later, while current MCP activation rules can remain separate until we fold them in.

Verification plan:

• Deterministic coverage for source filtering, unknown source handling, empty-query listing, compact mixed-source results, filtered result shape, and description truncation.
• Integration coverage that deferred plugin tools carry source metadata into the catalog.
• Memory workflow evals for the agent-facing behavior once local Postgres/model credentials are available.
• Local agent validation for explicit remember/forget flows.

Codex prior art: Codex's tool_search description advertises that tools are available from known sources and that some tools may need discovery. Junior should adopt that source-advertising pattern, but keep executeTool as the execution bridge because Junior does not currently load newly discovered tools into the next model turn.

[dcramer]

should be good here now.. still need to merge mcp into it but future