Phase 5: narrow repository access behind domain-facing stores#16
Merged
ethanj merged 20 commits intoarchitecture2from Apr 18, 2026
Merged
Phase 5: narrow repository access behind domain-facing stores#16ethanj merged 20 commits intoarchitecture2from
ethanj merged 20 commits intoarchitecture2from
Conversation
Design exploration docs (bm25-rrf-channel, phase-1a/1b analysis, scope-access-contract) belong in the research repo per the boundary rule: if it doesn't change shipped backend behavior, it belongs in atomicmemory-research. Updated the one source reference in composed-boot-parity.test.ts to point to the new location. Files moved to atomicmemory-research/docs/core-repo/: - bm25-rrf-channel.md - phase-1a-composed-boot-parity-test.md - phase-1a-singleton-hazards.md - phase-1b-config-import-audit.md - scope-access-contract.md
…ete/list Pre-PR for Phase 5. Closes the confirmed workspace expand visibility hole and adds opt-in visibility enforcement for GET/DELETE/LIST. Repository layer: - getMemoryInWorkspace and listMemoriesInWorkspace accept optional callerAgentId. When present, the visibility clause from searchVectorsInWorkspace is applied: agent_only memories are hidden from non-owning agents, restricted memories require explicit grants. Service layer: - scopedExpand now threads scope.agentId to the repo layer - New scopedGet, scopedDelete, scopedList methods that also thread agentId for visibility enforcement - Old *InWorkspace methods marked @deprecated (no visibility) Route layer: - GET/DELETE/LIST accept optional agent_id query param. When both workspace_id and agent_id are present, uses scoped methods with visibility enforcement. When agent_id absent, unchanged behavior. 5 new regression tests. 943/943 pass. tsc clean. fallow clean.
Addresses codex review of d340f03: 1. Workspace delete now returns 404 (not 500) when the memory is not visible to the calling agent. deleteMemoryInWorkspace returns false instead of throwing; route checks result and returns 404. 2. agent_id query param on GET/DELETE/LIST now validated via optionalUuidQuery (same as episode_id) — malformed input returns 400, not a Postgres type error 500. 3. Four new route-level regression tests: invalid agent_id on GET/LIST/DELETE returns 400; workspace DELETE of invisible memory returns 404. 947/947 tests pass. tsc clean.
Step 1 of Phase 5. Defines 8 store interfaces in src/db/stores.ts: MemoryStore, EpisodeStore, SearchStore, SemanticLinkStore, RepresentationStore, ClaimStore, EntityStore, LessonStore. Each interface exposes only the methods its domain consumers need, extracted from the existing MemoryRepository, ClaimRepository, EntityRepository, and LessonRepository public surfaces. Also defines CoreStores bundle type for the runtime container. Pure type definitions — no implementations, no behavioral change.
Step 2 of Phase 5. Creates 5 Pg*Store classes that implement the store interfaces by delegating to existing repository-*.ts functions: - PgMemoryStore (56L) — memory CRUD + workspace + CMO - PgEpisodeStore (20L) — episode create/read - PgSearchStore (65L) — vector/hybrid/keyword search + dedup - PgSemanticLinkStore (22L) — inter-memory links - PgRepresentationStore (28L) — atomic facts + foresight Also fixes 4 interface return types in stores.ts that didn't match the actual repo implementations (getMemoryStats shape, deleteBySource shape, createLinks returns number, findNearDuplicates returns CandidateRow[] not SearchResult[]). Pure additions — nothing breaks. 947/947 tests pass. tsc clean.
Step 3 of Phase 5. Constructs all Pg*Store instances in createCoreRuntime() and exposes them as runtime.stores alongside the existing runtime.repos (which remains for backward compat). ClaimStore, EntityStore, and LessonStore are type aliases for their existing repository classes — no wrapper needed since the classes already implement the right contracts. Simplifies stores.ts: removes manually-specified ClaimStore/ EntityStore/LessonStore interfaces that had signature mismatches; uses direct type aliases from the class exports instead. Pure addition — repos still constructed, services still wired through repos. 947/947 tests pass. tsc clean.
Addresses codex review: 1. Replaces type aliases (ClaimStore = ClaimRepository) with Pick<> narrowing that exposes only the methods domain consumers actually call. ClaimStore: 16 methods (from 30+), EntityStore: 10 methods, LessonStore: 7 methods. The concrete class still satisfies the narrowed type, but consumers can no longer access the full repo surface through stores.*. 2. Adds 2 runtime-container tests: stores are constructed with all expected fields, and entity/lesson store presence tracks config flags. Catches future regressions where store wiring breaks. 949/949 tests pass. tsc clean.
Step 4 of Phase 5. Migrates 8 service files from deps.repo/claims/ entities/lessons to deps.stores.*: - memory-ingest.ts → stores.episode, stores.memory - ingest-fact-pipeline.ts → stores.search, stores.memory - ingest-post-write.ts → stores.memory - memory-storage.ts → stores.memory, stores.representation, stores.claim, stores.entity - memory-audn.ts → stores.memory, stores.representation, stores.claim (deps.repo.getPool() retained for deferred-audn pool access) - memory-crud.ts → stores.memory, stores.claim, stores.entity, stores.lesson (deps.repo.getPool() retained for reconciliation) - memory-search.ts → stores.claim, stores.memory, stores.search - memory-lineage.ts — uses its own LineageDeps.claims port, unchanged MemoryServiceDeps gains stores field (deprecated old fields remain for backward compat). MemoryService constructor builds a stores shim from individual repos when stores not provided (test compat). 7 test files updated with stores in mock deps. 949/949 pass. tsc clean.
Addresses codex review of 0b989c5: migrated service code was calling deps.stores.* for data access but still checking deps.entities/ deps.lessons for null guards, creating a hidden dual-contract coupling. All guards in memory-crud.ts, memory-storage.ts, memory-audn.ts, and memory-search.ts now use deps.stores.entity/lesson for both the null check and the call. lesson-service.ts and write-security.ts updated to accept LessonStore instead of LessonRepository. EntityStore Pick expanded with findDeterministicEntity and upsertRelation. Only remaining deps.repo refs are in memory-search.ts (passed to runSearchPipelineWithTrace — Step 7) and memory-crud.ts (getPool() for deferred-audn/reconciliation). 949/949 tests pass. tsc clean. fallow clean.
…ores Step 7 of Phase 5. Replaces repo: MemoryRepository + entityRepo: EntityRepository params across 15 search-pipeline functions with stores: SearchPipelineStores — a focused bundle of SearchStore, SemanticLinkStore, MemoryStore, EntityStore, and pool. Internal functions use stores.search.*, stores.link.*, stores.memory.* directly. External imported functions (iterativeRetrieval, agenticRetrieval, expandQueryViaEntities, expandTemporalQuery, expandLiteralQuery, expandSubjectQuery, augmentQueryWithEntities, coRetrieveByEntityNames) receive stores.search/stores.entity with type assertions at the boundary — their signatures will be updated when their owning modules are migrated. Callers updated: memory-search.ts and ingest-post-write.ts build SearchPipelineStores from deps.stores + deps.repo.getPool(). 5 test files updated for the new stores shape. 949/949 tests pass. tsc clean. fallow clean.
…to representations Steps 8-9 of Phase 5. Step 8: Mark MemoryRepository class as @deprecated with explicit rationale (getPool() and external helper signatures still depend on it). Step 9: Add workspace_id and agent_id columns to memory_atomic_facts and memory_foresight via idempotent ALTER TABLE ... ADD COLUMN IF NOT EXISTS. Adds partial indexes on workspace_id for workspace-scoped queries. Updates StoreAtomicFactInput and StoreForesightInput with optional workspaceId/agentId fields, and the SQL inserts to write them. NULL values for user-scoped rows (backward compat). Replaces SCOPE_TODO comments with migration references. 949/949 tests pass. tsc clean. fallow clean.
…ep 10) Behavioral change: workspace ingest now calls storeCanonicalFact and resolveAndExecuteAudn instead of the inline storeWorkspaceMemory path. Workspace memories now get: - Canonical memory object (CMO) creation - Claim/version lineage - Atomic fact decomposition with workspace_id/agent_id columns - Foresight projections with workspace_id/agent_id columns - Entity resolution and linking (when enabled) Changes: - AudnFactContext gains optional workspace field - storeProjection threads workspace into storeMemory, storeAtomicFacts, storeForesight (workspace columns from Step 9) - resolveAndExecuteAudn accepts optional workspace, threads into ctx - supersedeCanonicalFact threads workspace through storeProjection - updateCanonicalFact threads workspace into replace operations (fixes the codex-flagged scope-loss on delete-and-reinsert) - processWorkspaceFact simplified: calls storeCanonicalFact/ resolveAndExecuteAudn instead of inline AUDN. storeWorkspaceMemory deleted. - AtomicFactRow and ForesightRow types gain workspace_id/agent_id Workspace test rewritten to assert the unified path: storeCanonicalFact called with workspace context, resolveAndExecuteAudn receives workspace. Note: claims remain user-scoped (memory_claims/memory_claim_versions have no workspace columns). This closes the workspace lineage asymmetry but does not make scope fully canonical across all tables. 946/946 tests pass. tsc clean. fallow clean.
…cept paths Addresses codex review of 97d2ed4: two AUDN side branches still created user-scoped memories when triggered from workspace ingest. - storeClarification (line 166): needs_clarification memory now carries workspaceId/agentId/visibility from ctx.workspace - tryOpinionIntercept (line 147): opinion-confidence-to-zero clarification memory now carries workspace scope Both paths default to undefined (NULL) for user-scoped ingest, preserving backward compat. 946/946 tests pass. tsc clean. fallow clean.
…e scope Addresses codex review: the two AUDN side branches that were leaking workspace scope (fixed in 2ce7a0f) now have direct test coverage. 1. CLARIFY branch: asserts storeMemory called with workspaceId, agentId, and needs_clarification status when workspace context is present. 2. Opinion-confidence-collapse: asserts the zero-confidence clarification memory carries workspace scope. 948/948 tests pass. tsc clean. fallow clean.
Closes the visibility bypass on public query routes. When workspace_id is present without agent_id, all three routes now return 400 "agent_id is required for workspace queries". The legacy fallback to visibility-unsafe *InWorkspace methods is removed entirely. Service layer cleanup: - MemoryService.listInWorkspace, getInWorkspace, deleteInWorkspace, expandInWorkspace deleted (were @deprecated) - memory-crud workspace helpers now require callerAgentId (was optional with visibility-unsafe fallback) - deleteMemoryInWorkspace visibility check is unconditional Tests: - 3 new 400-fence tests for missing agent_id - Removed backward-compat test in expand-visibility that relied on callerAgentId being optional 950/950 tests pass. tsc clean. fallow clean.
Adds pool: pg.Pool to CoreStores. All previous deps.repo.getPool()
callers now source from deps.stores.pool:
- memory-search.ts: pipelineStores construction
- ingest-post-write.ts: generateLinks call
- memory-audn.ts: deferMemoryForReconciliation
- memory-crud.ts: reconcileUser, reconcileAll, getReconciliationStatus
Runtime-container populates stores.pool from the constructed pool.
MemoryService test shim falls back to a defensive check when repo
lacks getPool (needed for mocks using {} as any).
deps.repo still exists for deprecated field access and for the
three deferred-audn/reconciliation calls that pass repo as second
arg — those migrate in Fix #3b.
950/950 tests pass. tsc clean. fallow clean.
…ps fields Completes the search-seam type enforcement codex flagged: Helper signature migrations (MemoryRepository → SearchStore/MemoryStore, EntityRepository → EntityStore): - iterative-retrieval.ts, agentic-retrieval.ts, keyword-expansion.ts - literal-query-expansion.ts, temporal-query-expansion.ts - subject-aware-ranking.ts, query-expansion.ts (2 params) - deferred-audn.ts (5 functions), consolidation-service.ts (2 functions) Removed all 8 `as any` casts in search-pipeline.ts. The pipeline is now fully compiler-enforced end-to-end. Deprecated fields removed from MemoryServiceDeps: - deps.repo, deps.claims, deps.entities, deps.lessons Only deps.stores.* remains. memory-storage.ts and memory-audn.ts updated to pass deps.stores.memory as the lineage CMO repo (matches structural MutationCanonicalObjectRepo interface). memory-crud.ts consolidation/reconciliation now uses deps.stores.*. memory-search.ts excludeStaleComposites uses deps.stores.memory. 950/950 tests pass. tsc clean. fallow clean.
…l medium) Addresses codex full-branch review: Phase 5 workspace ingest now goes through canonical lineage, but claim slots, entities, and entity relations remain user-scoped — meaning workspace A and workspace B (same user) share entity/claim state. This test pins the current behavior as intentional for Phase 5: - findDeterministicEntity keyed on userId only (no workspace param) - Two sequential ingest calls with same userId resolve against the same entity state regardless of which workspace they came from Phase 6 will introduce workspace_id on entities/memory_claims. When that lands, these assertions will fail and must be updated to assert the new scope isolation. The final test documents the exact Phase 6 surface that flips this. 953/953 tests pass. tsc clean. fallow clean.
Addresses codex review: the previous test only verified the helper signature, not the actual workspace ingest behavior. Replaced with a behavioral test that: 1. Uses a stateful in-memory fake EntityStore (tracks entities by (userId, type, name) — mimics real user-scoped resolution). 2. Runs the actual storeCanonicalFact path twice with two distinct workspace contexts (ws-A, ws-B) for the same user. 3. Asserts workspace B resolves against entities created by workspace A (no new entities created on the second call). 4. Asserts resolveEntity is called with no workspace/agent fields — this is the Phase 6 flip point. Verified by mutation: adding workspaceId to the real resolveEntity call fails the test. 3 tests, all behavioral. 953/953 tests pass. tsc clean. fallow clean.
3 tasks
ethanj
added a commit
that referenced
this pull request
Apr 19, 2026
Cuts over `main` to the Phase 1A–7 rearchitecture (composition root, explicit scope/observability contracts, store-narrowed repository access, public consumption seams, config split, leaf-module config threading, retrieval orchestration polish) plus the OSS-release-prep on top.⚠️ Breaking changes for HTTP / SDK consumers - All API endpoints are now mounted under `/v1` (e.g. `POST /v1/memories/ingest`, `PUT /v1/agents/trust`). The unversioned `/health` liveness probe is unchanged. - Workspace `GET /memories/list`, `GET /memories/:id`, and `DELETE /memories/:id` now require `agent_id` when `workspace_id` is present; missing returns 400 (no visibility-unsafe fallback). - `PUT /memories/config` returns 410 Gone in production. Provider/model fields (embedding_provider, embedding_model, llm_provider, llm_model) are rejected with 400 — they were never honored mid-flight (provider caches are fixed at first use). Set via env at process start. - npm package renamed `@atomicmemory/atomicmemory-engine` → `@atomicmemory/atomicmemory-core`. Tarball now ships `dist/` (built via `tsc`); `main`/`types`/`exports` point at compiled output. - Deep-importers of `services/embedding` and `services/llm` must call `initEmbedding(config)` / `initLlm(config)` before hot-path APIs. Consumers using `createCoreRuntime({ pool })` are auto-initialized. Rearchitecture (Phases 1A–7) - Phase 1A: composition root via `createCoreRuntime` + `createApp` (#8) - Phase 2A: canonical search scope contract (#9) - Phase 2B: explicit retrieval observability contract (#10) - Phase 3: runtime-config seam cleanup to Phase 4 boundary (#11) - Phase 4: ingest pipeline decomposition (475 → 215 lines) (#13) - Post-Phase 4: unify scope contract via `scopedSearch`/`scopedExpand`, document schema scoping gaps as deferred (#15) - Phase 5: narrow repository access behind 8 domain-facing stores; workspace ingest now flows through canonical lineage (#16) - Phase 6: publish stable consumption seams (HTTP, in-process, docker) with two-direction parity contract test (#17) - Phase 7 Steps 3a–3c: split runtime config into supported/internal partitions; deprecate `PUT /memories/config` for production (#18) - Phase 7 Step 3d: thread config through 5 leaf modules (33→28 singleton audit) (#21) - Phase 7 Item 4: retrieval polish — `memory-search.ts` reduced to pure orchestration (374 → 248 lines, -34%) (#22) - Chore: reduce fallow duplication 367 → 234 lines (#12) OSS release prep - `"private": true` removed; package renamed to `@atomicmemory/atomicmemory-core`. `files` field scopes the tarball. - `tsconfig.build.json` + `prepublishOnly` so `npm publish` always ships compiled `dist/`. Bare-import smoke test passes. - `release.yml` publishes to public npm on tag push (NPM_TOKEN secret). - SuperMem codename scrubbed from `src/`, tests, docker-compose, and `.env.example` (DB user/name/password renamed to `atomicmemory`). - Private-research-repo URLs unlinked from public docs. - README links to docs.atomicmemory.ai. - `/v1` API prefix on all routes; mount-coverage test added. - CI workflow: set `CORE_RUNTIME_CONFIG_MUTATION_ENABLED=true` to match `.env.test` (gitignored) and unblock the composed-boot-parity test on `PUT /v1/memories/config`. Verification - 966/966 tests pass (100 files) - npx tsc --noEmit clean - fallow --no-cache: 0 above threshold (maintainability 91.0) - npm publish --dry-run succeeds
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 5 of the rearchitecture plan. Replaces the
MemoryRepositorygod-object facade (40+ methods, 37 importers) with focused domain-facing store interfaces. Additionally closes the workspace lineage asymmetry and fixes a workspace visibility enforcement gap.Visibility enforcement (Pre-PR)
scopedExpandnow enforces agent visibility viagetMemoryInWorkspaceVisibleagent_idquery param for opt-in visibility enforcementagent_idreturns 400; invisible workspace delete returns 404 (not 500)Store layer (Steps 1-3)
src/db/stores.ts: MemoryStore, EpisodeStore, SearchStore, SemanticLinkStore, RepresentationStore, ClaimStore (Pick<>), EntityStore (Pick<>), LessonStore (Pick<>)CoreStoresbundle wired into runtime container alongside existing reposService migration (Steps 4, 7)
deps.repo/deps.claims/deps.entities/deps.lessonstodeps.stores.*SearchPipelineStoresbundledeps.stores.*source of truthDeprecation + schema (Steps 8-9)
MemoryRepositorymarked@deprecatedmemory_atomic_factsandmemory_foresightgainworkspace_id/agent_idcolumns via idempotent ALTER TABLEStoreAtomicFactInput/StoreForesightInputaccept optional workspace fieldsWorkspace canonical lineage (Step 10)
processWorkspaceFactnow routes throughstoreCanonicalFactandresolveAndExecuteAudnstoreWorkspaceMemorydeletedTest plan
npx tsc --noEmit— cleannpm test— 948/948 passfallow --no-cache— 0 above threshold🤖 Generated with Claude Code