diff --git a/CLAUDE.md b/CLAUDE.md index 77681ab..a8fa76c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,6 +27,9 @@ competitive analysis, and design explorations belong in `atomicmemory-research`. If it changes shipped backend behavior, it belongs here. If it only changes benchmark outputs or scoring methodology, it belongs in research. +See `docs/consuming-core.md` for the stable seams (HTTP, in-process runtime +container, docker/E2E compose) that research and SDK consumers should use. + ## Development Guidelines ### Code Style & Standards diff --git a/docs/README.md b/docs/README.md index 83f5245..2ca816f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,6 +3,7 @@ ## Structure - `api-reference.md` — HTTP API endpoint documentation +- `consuming-core.md` — how research, extensions, and SDK consumers boot core (HTTP, in-process, docker) - `design/` — architecture and design documents - `memory-research/architecture-overview.md` — system architecture overview diff --git a/docs/consuming-core.md b/docs/consuming-core.md new file mode 100644 index 0000000..28b5fe2 --- /dev/null +++ b/docs/consuming-core.md @@ -0,0 +1,112 @@ +# Consuming Atomicmemory-core + +How research harnesses, extensions, and SDK consumers boot core. Pick the seam +that matches your use case; do not re-build a parallel runtime. + +## Three consumption modes + +| Mode | Entry point | Use when | +| --- | --- | --- | +| **HTTP** | `POST /memories/ingest`, `POST /memories/search`, etc. | Black-box integration, language-agnostic clients, extension/SDK | +| **In-process** | `createCoreRuntime({ pool })` | TypeScript/Node harnesses that want no HTTP overhead | +| **Docker/E2E** | `docker-compose.smoke-isolated.yml` + `scripts/docker-smoke-test.sh` | Release validation, extension E2E, containerized CI | + +All three converge on the same composition root (`createCoreRuntime`). Behavior +cannot diverge between them — `src/app/__tests__/research-consumption-seams.test.ts` +holds that property. + +## HTTP + +Boot core as a server (`npm start`) and issue JSON requests. Snake_case on the +wire. + +```ts +const res = await fetch('http://localhost:3050/memories/ingest', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + user_id: 'alice', + conversation: 'user: I ship Go on the backend.', + source_site: 'my-app', + }), +}); +const { memoriesStored, memoryIds } = await res.json(); +``` + +```ts +const res = await fetch('http://localhost:3050/memories/search', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ user_id: 'alice', query: 'what stack?' }), +}); +const { count, injection_text, memories } = await res.json(); +``` + +See `docs/api-reference.md` for the full endpoint surface and response shapes. + +## In-process + +Import the composition root and call services directly. Useful when a Node +harness wants the same runtime without the HTTP hop. + +```ts +import pg from 'pg'; +import { createCoreRuntime } from '@atomicmemory/atomicmemory-engine'; + +const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL }); +const runtime = createCoreRuntime({ pool }); + +const write = await runtime.services.memory.ingest( + 'alice', + 'user: I ship Go on the backend.', + 'my-app', +); + +const read = await runtime.services.memory.search('alice', 'what stack?'); +``` + +Stable imports from the root export: + +- `createCoreRuntime`, `CoreRuntime`, `CoreRuntimeDeps` +- `createApp` — build the Express app from a runtime +- `bindEphemeral` — bind the app to an ephemeral port (for tests) +- `checkEmbeddingDimensions` — startup guard +- `MemoryService`, `IngestResult`, `RetrievalResult` + +**Config caveat.** `runtime.config` still references the module-level config +singleton. Consumers that need deterministic per-runtime config must set env +vars before importing core — two runtimes in the same process share config +today. See `src/app/runtime-container.ts` for the in-progress seam list. + +## Docker / E2E + +The canonical compose file for isolated end-to-end runs is +`docker-compose.smoke-isolated.yml`. Driven by `scripts/docker-smoke-test.sh`. + +Key env overrides: + +- `APP_PORT` (default `3061`) — host port bound to the core container's 3050 +- `POSTGRES_PORT` (default `5444`) — host port for the pgvector container +- `EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` — already + wired to `transformers` / `Xenova/all-MiniLM-L6-v2` / `384` for offline runs + +Use this mode for extension E2E, release validation, or any harness that needs +to treat core exactly as it ships. + +## Stability boundary + +- **Stable:** the root package export. Types and functions re-exported from + `src/index.ts` are the supported consumption surface. +- **Unstable:** deep-path imports (`@atomicmemory/atomicmemory-engine/services/*`, + `@atomicmemory/atomicmemory-engine/db/*`). These exist in `package.json` today for + migration convenience and will be narrowed. Research should prefer the + root export and raise an issue if something it needs is missing. + +## What belongs in research, not core + +Research harnesses, benchmarks, eval runners, experimental retrieval +strategies, and design proposals live in `atomicmemory-research`. Core owns +runtime truth: canonical API semantics, canonical scope semantics, canonical +trace fields, canonical schema, canonical write/mutation behavior. If a change +affects shipped backend behavior, it belongs here. If it only changes +benchmark outputs or scoring methodology, it belongs in research. diff --git a/src/__tests__/memory-route-config-seam.test.ts b/src/__tests__/memory-route-config-seam.test.ts index 7c07d9b..40db6c1 100644 --- a/src/__tests__/memory-route-config-seam.test.ts +++ b/src/__tests__/memory-route-config-seam.test.ts @@ -10,7 +10,7 @@ import express from 'express'; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; import { createMemoryRouter } from '../routes/memories.js'; import type { MemoryService } from '../services/memory-service.js'; -import { type BootedApp, bindEphemeral } from './test-helpers.js'; +import { type BootedApp, bindEphemeral } from '../app/bind-ephemeral.js'; interface MutableRouteConfig { retrievalProfile: string; diff --git a/src/app/__tests__/composed-boot-parity.test.ts b/src/app/__tests__/composed-boot-parity.test.ts index 3ebbeff..b7c5bce 100644 --- a/src/app/__tests__/composed-boot-parity.test.ts +++ b/src/app/__tests__/composed-boot-parity.test.ts @@ -29,7 +29,7 @@ import { MemoryService } from '../../services/memory-service.js'; import { createMemoryRouter } from '../../routes/memories.js'; import { createCoreRuntime } from '../runtime-container.js'; import { createApp } from '../create-app.js'; -import { type BootedApp, bindEphemeral } from '../../__tests__/test-helpers.js'; +import { type BootedApp, bindEphemeral } from '../bind-ephemeral.js'; const __dirname = dirname(fileURLToPath(import.meta.url)); const TEST_USER = 'composed-boot-parity-user'; diff --git a/src/app/__tests__/research-consumption-seams.test.ts b/src/app/__tests__/research-consumption-seams.test.ts new file mode 100644 index 0000000..be976b4 --- /dev/null +++ b/src/app/__tests__/research-consumption-seams.test.ts @@ -0,0 +1,164 @@ +/** + * Phase 6 research-consumption contract test. + * + * Proves the two in-repo consumption seams documented in + * `docs/consuming-core.md` both work against a shared runtime and agree + * on stored state: + * + * - in-process: `createCoreRuntime({ pool }).services.memory.*` + * - HTTP: `bindEphemeral(createApp(runtime))` + `fetch` + * + * The third mode (docker/E2E compose) is exercised by + * `scripts/docker-smoke-test.sh` and is out of scope for this test. + * + * Uses the same mock-hoist pattern as `smoke.test.ts` so no external + * LLM/embedding provider is required. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { config } from '../../config.js'; + +function seededEmbedding(text: string): number[] { + let seed = 0; + for (let i = 0; i < text.length; i++) seed = ((seed << 5) - seed + text.charCodeAt(i)) | 0; + return Array.from({ length: config.embeddingDimensions }, (_, i) => Math.sin(seed * (i + 1)) / 10); +} + +const mocks = vi.hoisted(() => ({ + mockEmbedText: vi.fn(), + mockEmbedTexts: vi.fn(), + mockConsensusExtractFacts: vi.fn(), + mockCachedResolveAUDN: vi.fn(), +})); + +vi.mock('../../services/embedding.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, embedText: mocks.mockEmbedText, embedTexts: mocks.mockEmbedTexts }; +}); +vi.mock('../../services/consensus-extraction.js', () => ({ + consensusExtractFacts: mocks.mockConsensusExtractFacts, +})); +vi.mock('../../services/extraction-cache.js', () => ({ + cachedResolveAUDN: mocks.mockCachedResolveAUDN, +})); + +import { pool } from '../../db/pool.js'; +import { setupTestSchema } from '../../db/__tests__/test-fixtures.js'; +import { createCoreRuntime } from '../runtime-container.js'; +import { createApp } from '../create-app.js'; +import { bindEphemeral, type BootedApp } from '../bind-ephemeral.js'; + +const TEST_USER = 'phase6-consumption-user'; +const CONVERSATION = + 'user: I ship the backend in Go and the frontend in TypeScript with Next.js.'; + +function stubMocks() { + mocks.mockEmbedText.mockImplementation(async (text: string) => seededEmbedding(text)); + mocks.mockEmbedTexts.mockImplementation(async (texts: string[]) => + texts.map((text) => seededEmbedding(text)), + ); + mocks.mockConsensusExtractFacts.mockImplementation(async () => [ + { + fact: 'User ships Go backend and TypeScript/Next.js frontend.', + headline: 'Stack', + importance: 0.8, + type: 'knowledge', + keywords: ['go', 'typescript', 'nextjs'], + entities: [], + relations: [], + }, + ]); + mocks.mockCachedResolveAUDN.mockImplementation(async () => ({ + action: 'ADD', + targetMemoryId: null, + updatedContent: null, + contradictionConfidence: null, + clarificationNote: null, + })); +} + +describe('Phase 6 research-consumption seams', () => { + const runtime = createCoreRuntime({ pool }); + const app = createApp(runtime); + let server: BootedApp; + + beforeAll(async () => { + await setupTestSchema(pool); + server = await bindEphemeral(app); + }); + + afterAll(async () => { + await server.close(); + await pool.end(); + }); + + beforeEach(async () => { + stubMocks(); + await runtime.repos.claims.deleteAll(); + await runtime.repos.memory.deleteAll(); + }); + + it('in-process seam: ingest + search via runtime.services.memory', async () => { + const write = await runtime.services.memory.ingest(TEST_USER, CONVERSATION, 'test-site'); + expect(write.memoriesStored).toBeGreaterThan(0); + + const read = await runtime.services.memory.search(TEST_USER, 'What stack does the user use?'); + expect(read.memories.length).toBeGreaterThan(0); + expect(read.injectionText.length).toBeGreaterThan(0); + }); + + it('HTTP seam: POST /memories/ingest + POST /memories/search via bindEphemeral', async () => { + const ingestRes = await fetch(`${server.baseUrl}/memories/ingest`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ user_id: TEST_USER, conversation: CONVERSATION, source_site: 'test-site' }), + }); + expect(ingestRes.status).toBe(200); + const ingestBody = await ingestRes.json(); + expect(ingestBody.memoriesStored).toBeGreaterThan(0); + + const searchRes = await fetch(`${server.baseUrl}/memories/search`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ user_id: TEST_USER, query: 'What stack does the user use?' }), + }); + expect(searchRes.status).toBe(200); + const searchBody = await searchRes.json(); + expect(searchBody.count).toBeGreaterThan(0); + expect(typeof searchBody.injection_text).toBe('string'); + expect(searchBody.injection_text.length).toBeGreaterThan(0); + }); + + it('parity: in-process write is observable through the HTTP seam (shared pool)', async () => { + const write = await runtime.services.memory.ingest(TEST_USER, CONVERSATION, 'test-site'); + expect(write.memoriesStored).toBeGreaterThan(0); + const writtenIds = new Set(write.memoryIds); + + const searchRes = await fetch(`${server.baseUrl}/memories/search`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ user_id: TEST_USER, query: 'What stack does the user use?' }), + }); + const body = await searchRes.json(); + const returnedIds: string[] = body.memories.map((memory: { id: string }) => memory.id); + const overlap = returnedIds.filter((id) => writtenIds.has(id)); + + expect(overlap.length).toBeGreaterThan(0); + }); + + it('parity: HTTP write is observable through the in-process seam (shared pool)', async () => { + const ingestRes = await fetch(`${server.baseUrl}/memories/ingest`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ user_id: TEST_USER, conversation: CONVERSATION, source_site: 'test-site' }), + }); + const ingestBody = await ingestRes.json(); + expect(ingestBody.memoriesStored).toBeGreaterThan(0); + const writtenIds = new Set(ingestBody.memoryIds); + + const read = await runtime.services.memory.search(TEST_USER, 'What stack does the user use?'); + const overlap = read.memories.filter((memory) => writtenIds.has(memory.id)); + expect(overlap.length).toBeGreaterThan(0); + }); +}); diff --git a/src/__tests__/test-helpers.ts b/src/app/bind-ephemeral.ts similarity index 58% rename from src/__tests__/test-helpers.ts rename to src/app/bind-ephemeral.ts index a85bddb..37e2683 100644 --- a/src/__tests__/test-helpers.ts +++ b/src/app/bind-ephemeral.ts @@ -1,5 +1,13 @@ /** - * Shared test utilities for integration tests that spin up Express servers. + * Canonical HTTP-boot helper for tests and research harnesses. + * + * Binds a composed Express app (`createApp(createCoreRuntime({ pool }))`) + * to an ephemeral port and returns the base URL plus a close handle. + * This is the stable seam for any in-repo test or external research + * harness that wants to exercise the HTTP contract against a live core + * server without hard-coding port allocation. + * + * Phase 6 of the rearchitecture — see docs/consuming-core.md. */ import type express from 'express'; diff --git a/src/index.ts b/src/index.ts index 2edcf0a..e1df81c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -16,3 +16,18 @@ export { type RetrievalProfile, type RetrievalProfileName, } from './services/retrieval-profiles.js'; +export { + createCoreRuntime, + type CoreRuntime, + type CoreRuntimeDeps, + type CoreRuntimeConfig, + type CoreRuntimeRepos, + type CoreRuntimeServices, + type CoreRuntimeConfigRouteAdapter, +} from './app/runtime-container.js'; +export { createApp } from './app/create-app.js'; +export { + checkEmbeddingDimensions, + type EmbeddingDimensionCheckResult, +} from './app/startup-checks.js'; +export { bindEphemeral, type BootedApp } from './app/bind-ephemeral.js';