P9-S37: importers and eval harness

.ai/handoff/CURRENT_STATE.md

@@ -20,8 +20,8 @@
 
 ## Incomplete / At-Risk Areas
 
-- importer coverage is still limited to one shipped adapter path (OpenClaw)
-- broader importer story is not yet public-ready
+- importer coverage is now broader but still file-import scoped (OpenClaw + Markdown + ChatGPT)
+- evaluation harness is local and fixture-backed; hosted/remote benchmarking is still intentionally out of scope
 - OSS license finalization is still open
 
 ## Current Milestone
@@ -30,7 +30,7 @@ Phase 9: Alice Public Core and Agent Interop
 
 ## Latest State Summary
 
-`P9-S33`, `P9-S34`, `P9-S35`, and `P9-S36` are now shipped baselines:
+`P9-S33`, `P9-S34`, `P9-S35`, `P9-S36`, and `P9-S37` are now shipped baselines:
 
 - package boundary is documented around `alice-core`
 - canonical local startup path is documented and script-backed
@@ -64,12 +64,17 @@ Phase 9: Alice Public Core and Agent Interop
   - import -> recall/resumption behavior on imported scope
   - shipped MCP `alice_recall`/`alice_resume` usage over imported data without MCP surface expansion
 - ADR-004 defines the accepted OpenClaw integration boundary and scope constraints.
-
-The next active seam is `P9-S37`:
-
-- expand from single-adapter proof to broader importer coverage and evaluation harness work
-- keep parity strict with existing deterministic continuity semantics
-- avoid widening MCP transport semantics unless parity defects are found
+- Additional importer paths now exist and are shipped:
+  - markdown importer: `markdown_import.py` plus `./scripts/load_markdown_sample_data.sh`
+  - ChatGPT importer: `chatgpt_import.py` plus `./scripts/load_chatgpt_sample_data.sh`
+- Shared importer provenance/dedupe persistence now uses one deterministic policy seam:
+  - `apps/api/src/alicebot_api/importers/common.py`
+  - importer-typed provenance fields with source-specific dedupe keys
+- Local Phase 9 evaluation harness is now shipped:
+  - command: `./scripts/run_phase9_eval.sh`
+  - generated report path: `eval/reports/phase9_eval_latest.json`
+  - committed baseline report path: `eval/baselines/phase9_s37_baseline.json`
+- ADR-005 and ADR-007 now define the accepted importer provenance/dedupe and evaluation-harness boundaries.
 
 ## Critical Constraints
 
@@ -80,11 +85,11 @@ The next active seam is `P9-S37`:
 
 ## Immediate Next Move
 
-Execute `P9-S37` on top of the shipped `P9-S36` boundary:
+Execute `P9-S38` on top of the shipped `P9-S37` boundary:
 
-- broaden importer coverage beyond OpenClaw using the same explicit provenance posture
-- add benchmark/evaluation harness evidence for import quality and correction-aware continuity outcomes
-- preserve startup/sample-data path and avoid MCP contract expansion unless needed for parity fixes
+- convert shipped `P9-S37` evidence and commands into launch-quality public docs
+- keep claims anchored to reproducible local importer/evaluation evidence
+- preserve startup/sample-data/runtime determinism and avoid MCP contract expansion unless parity defects are found
 
 ## Legacy Compatibility Markers
 

ARCHITECTURE.md

@@ -13,12 +13,13 @@ The current implementation already includes:
 
 Phase 9 does not replace that architecture. It packages and exposes it through public-safe boundaries.
 
-Current public packaging baseline now spans `P9-S33` through `P9-S36`:
+Current public packaging baseline now spans `P9-S33` through `P9-S37`:
 
 - `P9-S33`: public-safe core boundary and canonical local startup
 - `P9-S34`: deterministic local CLI continuity contract
 - `P9-S35`: narrow deterministic MCP transport contract
 - `P9-S36`: first OpenClaw adapter/import boundary with provenance + dedupe posture
+- `P9-S37`: additional markdown and ChatGPT importer paths plus reproducible local evaluation harness
 
 ## Technical Stack
 
@@ -28,14 +29,15 @@ Current public packaging baseline now spans `P9-S33` through `P9-S36`:
 - Vector support: `pgvector`
 - Local infrastructure: Docker Compose, Redis, MinIO
 - Testing: pytest, Vitest
-- Shipped packaging/runtime baseline (`P9-S33` to `P9-S36`):
+- Shipped packaging/runtime baseline (`P9-S33` to `P9-S37`):
   - `alice-core` (published package name in `pyproject.toml`)
   - deterministic fixture loader (`scripts/load_sample_data.sh`)
   - deterministic local CLI contract (`python -m alicebot_api ...`)
   - deterministic local MCP transport (`python -m alicebot_api.mcp_server`)
   - OpenClaw import loader and fixture path (`scripts/load_openclaw_sample_data.sh`)
-- Deferred/next packaging targets (`P9-S37+`):
-  - broader importer set and evaluation harness outputs
+  - markdown import loader and fixture path (`scripts/load_markdown_sample_data.sh`)
+  - ChatGPT import loader and fixture path (`scripts/load_chatgpt_sample_data.sh`)
+  - Phase 9 evaluation harness (`scripts/run_phase9_eval.sh`)
 
 ## High-Level Architecture
 
@@ -60,10 +62,12 @@ Current public packaging baseline now spans `P9-S33` through `P9-S36`:
    - terminal access to public core flows
 3. `alice-mcp-server` (shipped baseline in `P9-S35`, narrow tool contract)
    - stable MCP tool surface for external assistants
-4. `alice-importers` (next seam in `P9-S37`)
-   - markdown, chat export, CSV, and adapter-backed imports
-5. `alice-openclaw` (shipped baseline in `P9-S36`, adapter-scoped import path)
-   - OpenClaw-specific ingestion and interop mapping
+4. `alice-importers` (shipped baseline in `P9-S37`)
+   - markdown import path
+   - ChatGPT export import path
+   - shared provenance + dedupe persistence strategy
+5. `alice-openclaw` (shipped baseline in `P9-S36`, integrated with shared importer persistence in `P9-S37`)
+   - OpenClaw-specific normalization with shared deterministic import persistence
 
 ## Module Boundaries
 

BUILD_REPORT.md

@@ -1,73 +1,90 @@
 # BUILD_REPORT.md
 
 ## sprint objective
-Ship `P9-S36` by adding the first OpenClaw adapter/import path so OpenClaw workspace or durable-memory data can be imported into Alice continuity objects with explicit provenance and deterministic dedupe, then queried through shipped recall/resumption semantics and optionally through the shipped MCP tool surface without widening MCP contracts.
+Ship `P9-S37` by expanding importer coverage beyond OpenClaw to at least three production-usable importers total, generalizing deterministic provenance/dedupe posture across importers, and shipping a reproducible local evaluation harness with baseline report evidence for recall precision, resumption usefulness, correction effectiveness, importer success, and duplicate-memory posture.
 
 ## completed work
-- Implemented OpenClaw adapter boundary and input normalization:
-  - `apps/api/src/alicebot_api/openclaw_models.py`
-  - `apps/api/src/alicebot_api/openclaw_adapter.py`
-  - Supported source contract:
-    - JSON file with `durable_memory` / `memories` / `items` / `records`
-    - workspace directory with known JSON files (`workspace.json`, `openclaw_workspace.json`, `durable_memory.json`, `memories.json`, `openclaw_memories.json`)
-- Implemented OpenClaw import-to-continuity mapping:
-  - `apps/api/src/alicebot_api/openclaw_import.py`
-  - deterministic mapping into shipped continuity object types (`Decision`, `NextAction`, `WaitingFor`, `Commitment`, etc.)
-  - explicit provenance tagging on imported material (`source_kind=openclaw_import`, workspace/source metadata)
-  - deterministic dedupe posture via stable workspace+payload fingerprint (`openclaw_dedupe_key`)
-- Hardened importer lifecycle-status handling:
-  - unknown external `status` values are rejected with explicit validation errors
-  - importer no longer silently coerces unknown statuses to `active`
-- Added reproducible fixture and local import path:
-  - `fixtures/openclaw/workspace_v1.json`
-  - `scripts/load_openclaw_sample_data.py`
-  - `scripts/load_openclaw_sample_data.sh`
-- Added verification coverage for adapter/import/interop:
-  - `tests/unit/test_openclaw_adapter.py`
-  - `tests/integration/test_openclaw_import.py`
-  - `tests/integration/test_openclaw_mcp_integration.py`
-- Added adapter boundary ADR:
-  - `docs/adr/ADR-004-openclaw-integration-boundary.md`
-- Synced sprint-scoped docs:
-  - `README.md`
-  - `ROADMAP.md`
-  - `.ai/handoff/CURRENT_STATE.md`
-  - `ARCHITECTURE.md`
-  - `RULES.md`
-  - `.ai/active/SPRINT_PACKET.md` (scope hygiene annotation for archived planning artifacts)
+- Added two new production-usable importer paths:
+  - `markdown_import` (`apps/api/src/alicebot_api/markdown_import.py`)
+  - `chatgpt_import` (`apps/api/src/alicebot_api/chatgpt_import.py`)
+- Generalized importer provenance/dedupe persistence:
+  - shared importer model + normalization helpers in `apps/api/src/alicebot_api/importer_models.py`
+  - shared importer persistence seam in `apps/api/src/alicebot_api/importers/common.py`
+  - `openclaw_import.py` refactored to use shared importer persistence logic
+- Added reproducible importer fixtures:
+  - `fixtures/importers/markdown/workspace_v1.md`
+  - `fixtures/importers/chatgpt/workspace_v1.json`
+- Added importer loader commands:
+  - `scripts/load_markdown_sample_data.py` + `scripts/load_markdown_sample_data.sh`
+  - `scripts/load_chatgpt_sample_data.py` + `scripts/load_chatgpt_sample_data.sh`
+- Added local evaluation harness and report writer:
+  - `scripts/run_phase9_eval.py`
+  - `scripts/run_phase9_eval.sh`
+  - `apps/api/src/alicebot_api/retrieval_evaluation.py` extended with:
+    - `run_phase9_evaluation`
+    - `write_phase9_evaluation_report`
+    - importer/eval metrics for recall, resumption, correction, and dedupe posture
+- Generated and committed baseline evidence:
+  - `eval/reports/phase9_eval_latest.json`
+  - `eval/baselines/phase9_s37_baseline.json`
+- Added tests for importer/eval behavior:
+  - `tests/unit/test_importers.py`
+  - `tests/unit/test_phase9_eval.py`
+  - `tests/integration/test_markdown_import.py`
+  - `tests/integration/test_chatgpt_import.py`
+  - `tests/integration/test_phase9_eval.py`
+  - plus shared-package test import stabilization via `tests/__init__.py`, `tests/unit/__init__.py`, `tests/integration/__init__.py`
+- Synced sprint-scoped docs and ADRs:
+  - `README.md`, `ROADMAP.md`, `ARCHITECTURE.md`, `RULES.md`, `.ai/handoff/CURRENT_STATE.md`, `docs/phase9-sprint-33-38-plan.md`
+  - `docs/adr/ADR-005-import-provenance-and-dedupe-strategy.md`
+  - `docs/adr/ADR-007-public-evaluation-harness-scope.md`
 
 ## incomplete work
-- None inside `P9-S36` scope.
+- None inside `P9-S37` scope.
 - Intentionally deferred (out of scope):
-  - generic multi-source importer framework
+  - Claude importer path
+  - CSV importer path
   - MCP tool-surface expansion
-  - hosted adapter/auth/service work
+  - hosted/remote ingestion or evaluation infrastructure
 
 ## files changed
-- `apps/api/src/alicebot_api/openclaw_models.py`
-- `apps/api/src/alicebot_api/openclaw_adapter.py`
+- `.ai/active/SPRINT_PACKET.md`
+- `apps/api/src/alicebot_api/importer_models.py`
+- `apps/api/src/alicebot_api/importers/__init__.py`
+- `apps/api/src/alicebot_api/importers/common.py`
 - `apps/api/src/alicebot_api/openclaw_import.py`
-- `scripts/load_openclaw_sample_data.py`
-- `scripts/load_openclaw_sample_data.sh`
-- `fixtures/openclaw/workspace_v1.json`
-- `tests/unit/test_openclaw_adapter.py`
-- `tests/integration/test_openclaw_import.py`
-- `tests/integration/test_openclaw_mcp_integration.py`
-- `docs/adr/ADR-004-openclaw-integration-boundary.md`
-- `.ai/archive/planning/2026-04-07-phase9-bootstrap/SPRINT_PACKET.md`
-- `.ai/archive/planning/2026-04-07-phase9-bootstrap/CURRENT_STATE.md`
-- `docs/archive/planning/2026-04-07-phase9-bootstrap/README.md`
-- `docs/archive/planning/2026-04-07-phase9-bootstrap/ROADMAP.md`
-- `docs/archive/planning/2026-04-07-phase9-bootstrap/PRODUCT_BRIEF.md`
-- `docs/archive/planning/2026-04-07-phase9-bootstrap/ARCHITECTURE.md`
-- `docs/archive/planning/2026-04-07-phase9-bootstrap/RULES.md`
-- `ARCHITECTURE.md`
-- `RULES.md`
+- `apps/api/src/alicebot_api/markdown_import.py`
+- `apps/api/src/alicebot_api/chatgpt_import.py`
+- `apps/api/src/alicebot_api/retrieval_evaluation.py`
+- `scripts/load_markdown_sample_data.py`
+- `scripts/load_markdown_sample_data.sh`
+- `scripts/load_chatgpt_sample_data.py`
+- `scripts/load_chatgpt_sample_data.sh`
+- `scripts/run_phase9_eval.py`
+- `scripts/run_phase9_eval.sh`
+- `fixtures/importers/markdown/workspace_v1.md`
+- `fixtures/importers/chatgpt/workspace_v1.json`
+- `eval/reports/phase9_eval_latest.json`
+- `eval/baselines/phase9_s37_baseline.json`
+- `tests/unit/test_importers.py`
+- `tests/unit/test_phase9_eval.py`
+- `tests/integration/test_markdown_import.py`
+- `tests/integration/test_chatgpt_import.py`
+- `tests/integration/test_phase9_eval.py`
+- `tests/__init__.py`
+- `tests/unit/__init__.py`
+- `tests/integration/__init__.py`
+- `apps/web/components/approval-detail.test.tsx`
+- `apps/web/components/continuity-open-loops-panel.test.tsx`
+- `apps/web/components/workflow-memory-writeback-form.tsx`
 - `README.md`
 - `ROADMAP.md`
-- `docs/phase9-sprint-33-38-plan.md`
+- `ARCHITECTURE.md`
+- `RULES.md`
 - `.ai/handoff/CURRENT_STATE.md`
-- `.ai/active/SPRINT_PACKET.md`
+- `docs/phase9-sprint-33-38-plan.md`
+- `docs/adr/ADR-005-import-provenance-and-dedupe-strategy.md`
+- `docs/adr/ADR-007-public-evaluation-harness-scope.md`
 - `BUILD_REPORT.md`
 - `REVIEW_REPORT.md`
 
@@ -78,32 +95,35 @@ Ship `P9-S36` by adding the first OpenClaw adapter/import path so OpenClaw works
   - PASS
 - `./scripts/load_sample_data.sh`
   - PASS (`status=noop`, already loaded)
-- `./scripts/load_openclaw_sample_data.sh --source fixtures/openclaw/workspace_v1.json`
-  - PASS (`status=ok`, `imported_count=4`, `skipped_duplicates=1`)
-- `./scripts/load_openclaw_sample_data.sh --source fixtures/openclaw/workspace_v1.json`
-  - PASS (`status=noop`, `imported_count=0`, `skipped_duplicates=5`)
+- `./scripts/load_openclaw_sample_data.sh --user-id 00000000-0000-0000-0000-000000000038 --user-email openclaw-import-038@example.com --display-name "OpenClaw Import 038" --source fixtures/openclaw/workspace_v1.json`
+  - PASS (`imported_count=4`, `skipped_duplicates=1`)
+- `./scripts/load_markdown_sample_data.sh --user-id 00000000-0000-0000-0000-000000000038 --source fixtures/importers/markdown/workspace_v1.md`
+  - PASS (`imported_count=4`, `skipped_duplicates=1`)
+- `./scripts/load_chatgpt_sample_data.sh --user-id 00000000-0000-0000-0000-000000000038 --source fixtures/importers/chatgpt/workspace_v1.json`
+  - PASS (`imported_count=4`, `skipped_duplicates=1`)
 - `APP_RELOAD=false ./scripts/api_dev.sh`
-  - PASS (started on `http://127.0.0.1:8000`)
+  - PASS (startup observed; server on `127.0.0.1:8000`)
 - `curl -sS http://127.0.0.1:8000/healthz`
-  - PASS (`status":"ok"`)
-- `./.venv/bin/python -m alicebot_api recall --thread-id cccccccc-cccc-4ccc-8ccc-cccccccccccc --project "Alice Public Core" --query "MCP tool surface" --limit 5`
-  - PASS (returned imported OpenClaw `Decision` with `source_kind=openclaw_import` provenance references)
-- `./.venv/bin/python -m alicebot_api resume --thread-id cccccccc-cccc-4ccc-8ccc-cccccccccccc --max-recent-changes 5 --max-open-loops 5`
-  - PASS (`last_decision`, `next_action`, and `recent_changes` include imported OpenClaw data)
-- `./.venv/bin/python -m pytest tests/unit/test_openclaw_adapter.py -q`
-  - PASS (`5 passed`)
-- `./.venv/bin/python -m pytest tests/integration/test_openclaw_import.py tests/integration/test_openclaw_mcp_integration.py -q`
-  - PASS (`2 passed`)
-- `./.venv/bin/python -m pytest tests/unit/test_openclaw_adapter.py tests/integration/test_openclaw_import.py tests/integration/test_openclaw_mcp_integration.py -q`
+  - PASS (`{"status":"ok", ...}`)
+- `./.venv/bin/python -m alicebot_api --user-id 00000000-0000-0000-0000-000000000038 recall --thread-id eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee --project "Markdown Import Project" --query "markdown importer deterministic" --limit 5`
+  - PASS (returned markdown-imported records with explicit provenance)
+- `./.venv/bin/python -m alicebot_api --user-id 00000000-0000-0000-0000-000000000038 resume --thread-id eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee --project "Markdown Import Project" --max-recent-changes 5 --max-open-loops 5`
+  - PASS (`last_decision` + `next_action` from `markdown_import`)
+- `./scripts/run_phase9_eval.sh --user-id 00000000-0000-0000-0000-000000000039 --user-email phase9-eval-039@example.com --display-name "Phase9 Eval 039" --report-path eval/reports/phase9_eval_latest.json`
+  - PASS (`status=pass`, all rates `1.0`)
+- `./.venv/bin/python -m pytest tests/unit/test_importers.py tests/unit/test_phase9_eval.py -q`
   - PASS (`7 passed`)
+- `./.venv/bin/python -m pytest tests/integration/test_openclaw_import.py tests/integration/test_markdown_import.py tests/integration/test_chatgpt_import.py tests/integration/test_phase9_eval.py -q`
+  - PASS (`4 passed`)
 - `./.venv/bin/python -m pytest tests/unit tests/integration`
-  - PASS (`968 passed in 90.94s`)
+  - PASS (`978 passed`)
 - `pnpm --dir apps/web test`
   - PASS (`57 files, 192 tests`)
 
 ## blockers/issues
-- Sandbox restrictions required elevated execution for localhost Postgres/API verification commands.
-- No remaining functional blockers in sprint scope.
+- Localhost database/network checks from sandboxed runs required escalated command execution for several required verification commands.
+- One transient verification failure was encountered and resolved:
+  - `load_openclaw_sample_data.sh` with a new user ID initially failed due global unique email constraint (`users_email_key`) when default import email was reused; rerun with unique `--user-email` succeeded.
 
 ## recommended next step
-Start `P9-S37` by generalizing importer coverage from the now-shipped OpenClaw boundary while preserving the same explicit provenance and dedupe posture, and adding benchmark/evaluation harness evidence for importer quality.
+Execute `P9-S38` by turning the now-shipped importer/evaluation evidence and commands into launch-quality external docs without widening MCP transport or importer semantics.