Phase 6: publish stable consumption seams for research and SDK consumers

CLAUDE.md

@@ -27,6 +27,9 @@ competitive analysis, and design explorations belong in `atomicmemory-research`.
 If it changes shipped backend behavior, it belongs here. If it only changes
 benchmark outputs or scoring methodology, it belongs in research.
 
+See `docs/consuming-core.md` for the stable seams (HTTP, in-process runtime
+container, docker/E2E compose) that research and SDK consumers should use.
+
 ## Development Guidelines
 
 ### Code Style & Standards

docs/README.md

@@ -3,6 +3,7 @@
 ## Structure
 
 - `api-reference.md` — HTTP API endpoint documentation
+- `consuming-core.md` — how research, extensions, and SDK consumers boot core (HTTP, in-process, docker)
 - `design/` — architecture and design documents
 - `memory-research/architecture-overview.md` — system architecture overview
 

docs/consuming-core.md

@@ -0,0 +1,112 @@
+# Consuming Atomicmemory-core
+
+How research harnesses, extensions, and SDK consumers boot core. Pick the seam
+that matches your use case; do not re-build a parallel runtime.
+
+## Three consumption modes
+
+| Mode | Entry point | Use when |
+| --- | --- | --- |
+| **HTTP** | `POST /memories/ingest`, `POST /memories/search`, etc. | Black-box integration, language-agnostic clients, extension/SDK |
+| **In-process** | `createCoreRuntime({ pool })` | TypeScript/Node harnesses that want no HTTP overhead |
+| **Docker/E2E** | `docker-compose.smoke-isolated.yml` + `scripts/docker-smoke-test.sh` | Release validation, extension E2E, containerized CI |
+
+All three converge on the same composition root (`createCoreRuntime`). Behavior
+cannot diverge between them — `src/app/__tests__/research-consumption-seams.test.ts`
+holds that property.
+
+## HTTP
+
+Boot core as a server (`npm start`) and issue JSON requests. Snake_case on the
+wire.
+
+```ts
+const res = await fetch('http://localhost:3050/memories/ingest', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({
+    user_id: 'alice',
+    conversation: 'user: I ship Go on the backend.',
+    source_site: 'my-app',
+  }),
+});
+const { memoriesStored, memoryIds } = await res.json();
+```
+
+```ts
+const res = await fetch('http://localhost:3050/memories/search', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({ user_id: 'alice', query: 'what stack?' }),
+});
+const { count, injection_text, memories } = await res.json();
+```
+
+See `docs/api-reference.md` for the full endpoint surface and response shapes.
+
+## In-process
+
+Import the composition root and call services directly. Useful when a Node
+harness wants the same runtime without the HTTP hop.
+
+```ts
+import pg from 'pg';
+import { createCoreRuntime } from '@atomicmemory/atomicmemory-engine';
+
+const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
+const runtime = createCoreRuntime({ pool });
+
+const write = await runtime.services.memory.ingest(
+  'alice',
+  'user: I ship Go on the backend.',
+  'my-app',
+);
+
+const read = await runtime.services.memory.search('alice', 'what stack?');
+```
+
+Stable imports from the root export:
+
+- `createCoreRuntime`, `CoreRuntime`, `CoreRuntimeDeps`
+- `createApp` — build the Express app from a runtime
+- `bindEphemeral` — bind the app to an ephemeral port (for tests)
+- `checkEmbeddingDimensions` — startup guard
+- `MemoryService`, `IngestResult`, `RetrievalResult`
+
+**Config caveat.** `runtime.config` still references the module-level config
+singleton. Consumers that need deterministic per-runtime config must set env
+vars before importing core — two runtimes in the same process share config
+today. See `src/app/runtime-container.ts` for the in-progress seam list.
+
+## Docker / E2E
+
+The canonical compose file for isolated end-to-end runs is
+`docker-compose.smoke-isolated.yml`. Driven by `scripts/docker-smoke-test.sh`.
+
+Key env overrides:
+
+- `APP_PORT` (default `3061`) — host port bound to the core container's 3050
+- `POSTGRES_PORT` (default `5444`) — host port for the pgvector container
+- `EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` — already
+  wired to `transformers` / `Xenova/all-MiniLM-L6-v2` / `384` for offline runs
+
+Use this mode for extension E2E, release validation, or any harness that needs
+to treat core exactly as it ships.
+
+## Stability boundary
+
+- **Stable:** the root package export. Types and functions re-exported from
+  `src/index.ts` are the supported consumption surface.
+- **Unstable:** deep-path imports (`@atomicmemory/atomicmemory-engine/services/*`,
+  `@atomicmemory/atomicmemory-engine/db/*`). These exist in `package.json` today for
+  migration convenience and will be narrowed. Research should prefer the
+  root export and raise an issue if something it needs is missing.
+
+## What belongs in research, not core
+
+Research harnesses, benchmarks, eval runners, experimental retrieval
+strategies, and design proposals live in `atomicmemory-research`. Core owns
+runtime truth: canonical API semantics, canonical scope semantics, canonical
+trace fields, canonical schema, canonical write/mutation behavior. If a change
+affects shipped backend behavior, it belongs here. If it only changes
+benchmark outputs or scoring methodology, it belongs in research.

src/__tests__/memory-route-config-seam.test.ts

@@ -10,7 +10,7 @@ import express from 'express';
 import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
 import { createMemoryRouter } from '../routes/memories.js';
 import type { MemoryService } from '../services/memory-service.js';
-import { type BootedApp, bindEphemeral } from './test-helpers.js';
+import { type BootedApp, bindEphemeral } from '../app/bind-ephemeral.js';
 
 interface MutableRouteConfig {
   retrievalProfile: string;

src/app/__tests__/composed-boot-parity.test.ts

@@ -29,7 +29,7 @@ import { MemoryService } from '../../services/memory-service.js';
 import { createMemoryRouter } from '../../routes/memories.js';
 import { createCoreRuntime } from '../runtime-container.js';
 import { createApp } from '../create-app.js';
-import { type BootedApp, bindEphemeral } from '../../__tests__/test-helpers.js';
+import { type BootedApp, bindEphemeral } from '../bind-ephemeral.js';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const TEST_USER = 'composed-boot-parity-user';

src/app/__tests__/research-consumption-seams.test.ts

@@ -0,0 +1,164 @@
+/**
+ * Phase 6 research-consumption contract test.
+ *
+ * Proves the two in-repo consumption seams documented in
+ * `docs/consuming-core.md` both work against a shared runtime and agree
+ * on stored state:
+ *
+ *   - in-process:  `createCoreRuntime({ pool }).services.memory.*`
+ *   - HTTP:         `bindEphemeral(createApp(runtime))` + `fetch`
+ *
+ * The third mode (docker/E2E compose) is exercised by
+ * `scripts/docker-smoke-test.sh` and is out of scope for this test.
+ *
+ * Uses the same mock-hoist pattern as `smoke.test.ts` so no external
+ * LLM/embedding provider is required.
+ */
+
+import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
+
+import { config } from '../../config.js';
+
+function seededEmbedding(text: string): number[] {
+  let seed = 0;
+  for (let i = 0; i < text.length; i++) seed = ((seed << 5) - seed + text.charCodeAt(i)) | 0;
+  return Array.from({ length: config.embeddingDimensions }, (_, i) => Math.sin(seed * (i + 1)) / 10);
+}
+
+const mocks = vi.hoisted(() => ({
+  mockEmbedText: vi.fn(),
+  mockEmbedTexts: vi.fn(),
+  mockConsensusExtractFacts: vi.fn(),
+  mockCachedResolveAUDN: vi.fn(),
+}));
+
+vi.mock('../../services/embedding.js', async (importOriginal) => {
+  const actual = await importOriginal<typeof import('../../services/embedding.js')>();
+  return { ...actual, embedText: mocks.mockEmbedText, embedTexts: mocks.mockEmbedTexts };
+});
+vi.mock('../../services/consensus-extraction.js', () => ({
+  consensusExtractFacts: mocks.mockConsensusExtractFacts,
+}));
+vi.mock('../../services/extraction-cache.js', () => ({
+  cachedResolveAUDN: mocks.mockCachedResolveAUDN,
+}));
+
+import { pool } from '../../db/pool.js';
+import { setupTestSchema } from '../../db/__tests__/test-fixtures.js';
+import { createCoreRuntime } from '../runtime-container.js';
+import { createApp } from '../create-app.js';
+import { bindEphemeral, type BootedApp } from '../bind-ephemeral.js';
+
+const TEST_USER = 'phase6-consumption-user';
+const CONVERSATION =
+  'user: I ship the backend in Go and the frontend in TypeScript with Next.js.';
+
+function stubMocks() {
+  mocks.mockEmbedText.mockImplementation(async (text: string) => seededEmbedding(text));
+  mocks.mockEmbedTexts.mockImplementation(async (texts: string[]) =>
+    texts.map((text) => seededEmbedding(text)),
+  );
+  mocks.mockConsensusExtractFacts.mockImplementation(async () => [
+    {
+      fact: 'User ships Go backend and TypeScript/Next.js frontend.',
+      headline: 'Stack',
+      importance: 0.8,
+      type: 'knowledge',
+      keywords: ['go', 'typescript', 'nextjs'],
+      entities: [],
+      relations: [],
+    },
+  ]);
+  mocks.mockCachedResolveAUDN.mockImplementation(async () => ({
+    action: 'ADD',
+    targetMemoryId: null,
+    updatedContent: null,
+    contradictionConfidence: null,
+    clarificationNote: null,
+  }));
+}
+
+describe('Phase 6 research-consumption seams', () => {
+  const runtime = createCoreRuntime({ pool });
+  const app = createApp(runtime);
+  let server: BootedApp;
+
+  beforeAll(async () => {
+    await setupTestSchema(pool);
+    server = await bindEphemeral(app);
+  });
+
+  afterAll(async () => {
+    await server.close();
+    await pool.end();
+  });
+
+  beforeEach(async () => {
+    stubMocks();
+    await runtime.repos.claims.deleteAll();
+    await runtime.repos.memory.deleteAll();
+  });
+
+  it('in-process seam: ingest + search via runtime.services.memory', async () => {
+    const write = await runtime.services.memory.ingest(TEST_USER, CONVERSATION, 'test-site');
+    expect(write.memoriesStored).toBeGreaterThan(0);
+
+    const read = await runtime.services.memory.search(TEST_USER, 'What stack does the user use?');
+    expect(read.memories.length).toBeGreaterThan(0);
+    expect(read.injectionText.length).toBeGreaterThan(0);
+  });
+
+  it('HTTP seam: POST /memories/ingest + POST /memories/search via bindEphemeral', async () => {
+    const ingestRes = await fetch(`${server.baseUrl}/memories/ingest`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ user_id: TEST_USER, conversation: CONVERSATION, source_site: 'test-site' }),
+    });
+    expect(ingestRes.status).toBe(200);
+    const ingestBody = await ingestRes.json();
+    expect(ingestBody.memoriesStored).toBeGreaterThan(0);
+
+    const searchRes = await fetch(`${server.baseUrl}/memories/search`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ user_id: TEST_USER, query: 'What stack does the user use?' }),
+    });
+    expect(searchRes.status).toBe(200);
+    const searchBody = await searchRes.json();
+    expect(searchBody.count).toBeGreaterThan(0);
+    expect(typeof searchBody.injection_text).toBe('string');
+    expect(searchBody.injection_text.length).toBeGreaterThan(0);
+  });
+
+  it('parity: in-process write is observable through the HTTP seam (shared pool)', async () => {
+    const write = await runtime.services.memory.ingest(TEST_USER, CONVERSATION, 'test-site');
+    expect(write.memoriesStored).toBeGreaterThan(0);
+    const writtenIds = new Set(write.memoryIds);
+
+    const searchRes = await fetch(`${server.baseUrl}/memories/search`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ user_id: TEST_USER, query: 'What stack does the user use?' }),
+    });
+    const body = await searchRes.json();
+    const returnedIds: string[] = body.memories.map((memory: { id: string }) => memory.id);
+    const overlap = returnedIds.filter((id) => writtenIds.has(id));
+
+    expect(overlap.length).toBeGreaterThan(0);
+  });
+
+  it('parity: HTTP write is observable through the in-process seam (shared pool)', async () => {
+    const ingestRes = await fetch(`${server.baseUrl}/memories/ingest`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ user_id: TEST_USER, conversation: CONVERSATION, source_site: 'test-site' }),
+    });
+    const ingestBody = await ingestRes.json();
+    expect(ingestBody.memoriesStored).toBeGreaterThan(0);
+    const writtenIds = new Set<string>(ingestBody.memoryIds);
+
+    const read = await runtime.services.memory.search(TEST_USER, 'What stack does the user use?');
+    const overlap = read.memories.filter((memory) => writtenIds.has(memory.id));
+    expect(overlap.length).toBeGreaterThan(0);
+  });
+});

src/__tests__/test-helpers.ts → src/app/bind-ephemeral.ts

@@ -1,5 +1,13 @@
 /**
- * Shared test utilities for integration tests that spin up Express servers.
+ * Canonical HTTP-boot helper for tests and research harnesses.
+ *
+ * Binds a composed Express app (`createApp(createCoreRuntime({ pool }))`)
+ * to an ephemeral port and returns the base URL plus a close handle.
+ * This is the stable seam for any in-repo test or external research
+ * harness that wants to exercise the HTTP contract against a live core
+ * server without hard-coding port allocation.
+ *
+ * Phase 6 of the rearchitecture — see docs/consuming-core.md.
  */
 
 import type express from 'express';

src/index.ts

@@ -16,3 +16,18 @@ export {
   type RetrievalProfile,
   type RetrievalProfileName,
 } from './services/retrieval-profiles.js';
+export {
+  createCoreRuntime,
+  type CoreRuntime,
+  type CoreRuntimeDeps,
+  type CoreRuntimeConfig,
+  type CoreRuntimeRepos,
+  type CoreRuntimeServices,
+  type CoreRuntimeConfigRouteAdapter,
+} from './app/runtime-container.js';
+export { createApp } from './app/create-app.js';
+export {
+  checkEmbeddingDimensions,
+  type EmbeddingDimensionCheckResult,
+} from './app/startup-checks.js';
+export { bindEphemeral, type BootedApp } from './app/bind-ephemeral.js';