diff --git a/ARCHITECTURE-REVIEW.md b/ARCHITECTURE-REVIEW.md deleted file mode 100644 index c894929..0000000 --- a/ARCHITECTURE-REVIEW.md +++ /dev/null @@ -1,277 +0,0 @@ -# CORTEX Architecture Review — Naming Drift Report - -**Date:** 2026-03-13 -**Scope:** Full repository audit against corrected DESIGN.md (v1.2) -**Status:** Documentation-only pass; no code changes made in this review - ---- - -## Executive Summary - -The repository has drifted from the intended CORTEX architecture due to an early conceptual collapse between **medoids** and **Metroids**. This caused the term "Metroid" to be applied throughout the codebase and documentation to describe the sparse proximity/neighbor graph connecting pages — a fundamentally different concept. - -The correct meaning of each term is: - -| Term | Correct Meaning | -|------|----------------| -| **Medoid** | An existing memory node selected as a cluster representative via the medoid statistic | -| **Centroid** | A mathematical average of vectors — a computed point, never a stored node | -| **Metroid** | A structured dialectical search probe: `{ m1, m2, c }` — ephemeral, constructed at query time | - -The sparse proximity graph connecting pages with high cosine similarity is **not** a Metroid. It is the **semantic neighbor graph**. The entire MetroidBuilder component — the heart of CORTEX's epistemic search capability — does not yet exist in the codebase. - -This report catalogs every divergence found and maps each to a correction task in TODO.md. - ---- - -## Divergence Catalog - -### D1 — `core/types.ts`: `MetroidNeighbor` interface - -| Field | Value | -|-------|-------| -| **File** | `core/types.ts` | -| **Line** | ~70 | -| **Component** | `MetroidNeighbor` interface | -| **Current behavior** | Defines a sparse proximity graph edge with `neighborPageId`, `cosineSimilarity`, and `distance`. Named as if it represents a "Metroid" concept. | -| **Intended behavior** | This is a proximity edge in the semantic neighbor graph. It has nothing to do with the `Metroid = { m1, m2, c }` dialectical probe. Should be named `SemanticNeighbor`. | -| **Required correction** | Rename `MetroidNeighbor` → `SemanticNeighbor`. Update all references. | -| **TODO task** | P0-X1 | - ---- - -### D2 — `core/types.ts`: `MetroidSubgraph` interface - -| Field | Value | -|-------|-------| -| **File** | `core/types.ts` | -| **Line** | ~76 | -| **Component** | `MetroidSubgraph` interface | -| **Current behavior** | Defines the induced subgraph used for BFS expansion during retrieval. Named "MetroidSubgraph". | -| **Intended behavior** | This is a semantic neighbor subgraph, not a Metroid. Should be named `SemanticNeighborSubgraph`. | -| **Required correction** | Rename `MetroidSubgraph` → `SemanticNeighborSubgraph`. | -| **TODO task** | P0-X2 | - ---- - -### D3 — `core/types.ts`: `MetadataStore` proximity graph methods - -| Field | Value | -|-------|-------| -| **File** | `core/types.ts` | -| **Lines** | ~178–191 | -| **Component** | `MetadataStore` interface — methods section "Metroid NN radius index" | -| **Current behavior** | Six methods use "Metroid" naming: `putMetroidNeighbors`, `getMetroidNeighbors`, `getInducedMetroidSubgraph`, `needsMetroidRecalc`, `flagVolumeForMetroidRecalc`, `clearMetroidRecalcFlag`. | -| **Intended behavior** | These methods operate on the semantic neighbor graph (a proximity graph). "Metroid" in method names implies a connection to the dialectical probe construct, which is incorrect. | -| **Required correction** | Rename all six methods: `putSemanticNeighbors`, `getSemanticNeighbors`, `getInducedNeighborSubgraph`, `needsNeighborRecalc`, `flagVolumeForNeighborRecalc`, `clearNeighborRecalcFlag`. | -| **TODO task** | P0-X3 | - ---- - -### D4 — `storage/IndexedDbMetadataStore.ts`: `metroid_neighbors` IDB store - -| Field | Value | -|-------|-------| -| **File** | `storage/IndexedDbMetadataStore.ts` | -| **Lines** | ~32–35 (DB store declarations) | -| **Component** | IndexedDB object store named `metroid_neighbors` | -| **Current behavior** | Persists proximity graph edges between pages in a store named `metroid_neighbors`. | -| **Intended behavior** | The store name should reflect that it holds semantic proximity edges, not Metroid probes. Should be `neighbor_graph`. | -| **Required correction** | Rename IDB store from `metroid_neighbors` → `neighbor_graph`. Increment `DB_VERSION`. Add migration in `applyUpgrade` to copy existing data. | -| **TODO task** | P0-X6 | - ---- - -### D5 — `storage/IndexedDbMetadataStore.ts`: proximity graph method implementations - -| Field | Value | -|-------|-------| -| **File** | `storage/IndexedDbMetadataStore.ts` | -| **Lines** | All methods implementing `MetadataStore` proximity graph interface | -| **Component** | `putMetroidNeighbors`, `getMetroidNeighbors`, `getInducedMetroidSubgraph`, `needsMetroidRecalc`, `flagVolumeForMetroidRecalc`, `clearMetroidRecalcFlag` implementations | -| **Current behavior** | Implements the six methods using `MetroidNeighbor` types and `metroid_neighbors` IDB store. | -| **Intended behavior** | Should use renamed types and store. | -| **Required correction** | After interface rename (D1–D4), update all implementations to use new names. | -| **TODO task** | P0-X1–X6 | - ---- - -### D6 — `cortex/Query.ts`: Absent MetroidBuilder - -| Field | Value | -|-------|-------| -| **File** | `cortex/Query.ts` | -| **Lines** | Entire file | -| **Component** | `query()` function | -| **Current behavior** | Embeds query, scores hotpath pages, falls back to full scan, updates PageActivity, runs promotion sweep. Returns a ranked list of pages. **No Metroid is ever constructed. No dialectical search is performed. No knowledge gap is ever detected.** | -| **Intended behavior** | The query path should: (1) select m1 (topic medoid), (2) call MetroidBuilder to construct `{ m1, m2, c }`, (3) use centroid `c` as the balanced search anchor, (4) explore thesis/antithesis/synthesis zones, (5) detect and surface knowledge gaps. | -| **Required correction** | After MetroidBuilder is implemented (P1-M), upgrade `cortex/Query.ts` to include the full dialectical pipeline (P1-E). | -| **TODO task** | P1-E1 | - ---- - -### D7 — `cortex/Query.ts`: `getInducedMetroidSubgraph` call - -| Field | Value | -|-------|-------| -| **File** | `cortex/Query.ts` | -| **Lines** | The subgraph expansion step (BFS, if present) | -| **Component** | Subgraph expansion via `MetadataStore` | -| **Current behavior** | If subgraph BFS is called, it uses `getInducedMetroidSubgraph`, propagating the incorrect naming. | -| **Intended behavior** | Should call `getInducedNeighborSubgraph` (after rename). | -| **Required correction** | Rename the method call after P0-X3 is complete. | -| **TODO task** | P0-X3 | - ---- - -### D8 — DESIGN.md (pre-correction): Incorrect Terminology - -| Field | Value | -|-------|-------| -| **File** | `DESIGN.md` (pre-v1.2) | -| **Component** | Terminology section | -| **Current behavior** | Defined "Metroid (canonical domain term): Sparse nearest-neighbor graph structure inspired by medoid-based clustering." This is architecturally incorrect. | -| **Intended behavior** | Metroid = dialectical probe `{ m1, m2, c }`. The sparse NN graph is the semantic neighbor graph. | -| **Required correction** | **Already corrected in DESIGN.md v1.2.** | -| **TODO task** | Resolved | - ---- - -### D9 — DESIGN.md (pre-correction): Missing MetroidBuilder, Dialectical Search, Knowledge Gap - -| Field | Value | -|-------|-------| -| **File** | `DESIGN.md` (pre-v1.2) | -| **Component** | Entire document | -| **Current behavior** | No section describing MetroidBuilder, Matryoshka dimensional unwinding, antithesis discovery, dialectical search, or knowledge gap detection. | -| **Intended behavior** | These are core architectural concepts that must be described for any engineer to implement CORTEX correctly. | -| **Required correction** | **Already corrected in DESIGN.md v1.2** — new section "Conceptual Constructs: Medoid, Centroid, and Metroid" added. | -| **TODO task** | Resolved | - ---- - -### D10 — PLAN.md (pre-correction): "Metroid vs medoid" note - -| Field | Value | -|-------|-------| -| **File** | `PLAN.md` (pre-v1.2) | -| **Component** | Notes section | -| **Current behavior** | Note read: "Metroid vs medoid: Use Metroid in all API surfaces and docs; medoid only in algorithmic comments." This instructs developers to use the wrong term everywhere, making MetroidBuilder impossible to introduce without collision. | -| **Intended behavior** | The note must distinguish three concepts: Metroid (dialectical probe), medoid (cluster representative), and semantic neighbor graph (proximity graph for BFS). | -| **Required correction** | **Already corrected in PLAN.md v1.2.** | -| **TODO task** | Resolved | - ---- - -### D11 — `PLAN.md` (pre-correction): Missing MetroidBuilder in CORTEX module table - -| Field | Value | -|-------|-------| -| **File** | `PLAN.md` (pre-v1.2) | -| **Component** | CORTEX module table | -| **Current behavior** | No MetroidBuilder, KnowledgeGapDetector, or DialecticalSearch pipeline listed as planned modules. | -| **Intended behavior** | These are critical CORTEX components without which the system is merely a vector search engine. | -| **Required correction** | **Already corrected in PLAN.md v1.2** — new rows added. | -| **TODO task** | Resolved | - ---- - -### D12 — `hippocampus/Ingest.ts`: Semantic neighbor insertion absent - -| Field | Value | -|-------|-------| -| **File** | `hippocampus/Ingest.ts` | -| **Lines** | Entire file | -| **Component** | `ingestText()` function | -| **Current behavior** | Chunks, embeds, persists pages, builds a book, runs promotion sweep. Does **not** insert semantic neighbor edges. | -| **Intended behavior** | After persisting pages, should call `FastNeighborInsert` to maintain the semantic neighbor graph with Williams-bounded degree. | -| **Required correction** | After `FastNeighborInsert` is implemented (P1-C), upgrade `ingestText` to call it (P1-C2). | -| **TODO task** | P1-C2 | - ---- - -### D13 — `core/types.ts`: No Metroid type defined - -| Field | Value | -|-------|-------| -| **File** | `core/types.ts` | -| **Component** | Type definitions | -| **Current behavior** | The word "Metroid" appears only as part of `MetroidNeighbor`, `MetroidSubgraph`, and `MetadataStore` method names — all of which are proximity-graph concepts. The **actual Metroid type** `{ m1, m2, c }` does not exist. | -| **Intended behavior** | `core/types.ts` should define: `interface Metroid { m1: Hash; m2: Hash | null; centroid: Float32Array | null; knowledgeGap: boolean }` and `interface KnowledgeGap { topicMedoidId: Hash; queryEmbedding: Float32Array; dimensionalBoundary: number; timestamp: string }`. | -| **Required correction** | Add these types to `core/types.ts` as part of MetroidBuilder implementation (P1-M). | -| **TODO task** | P1-M1 | - ---- - -### D14 — `core/types.ts`: No `matryoshkaProtectedDim` in `ModelProfile` - -| Field | Value | -|-------|-------| -| **File** | `core/ModelProfile.ts` | -| **Component** | `ModelProfile` interface | -| **Current behavior** | No field for the protected Matryoshka dimension boundary. | -| **Intended behavior** | MetroidBuilder needs to know which lower dimensions to freeze during antithesis search. `ModelProfile` should include `matryoshkaProtectedDim: number` — the number of lower dimensions that encode invariant semantic context. | -| **Required correction** | Add `matryoshkaProtectedDim` to `ModelProfile` interface; add default value to `ModelDefaults.ts`; add per-model value to `BuiltInModelProfiles.ts`. | -| **TODO task** | P1-M1 (prerequisite) | - ---- - -### D15 — `cortex/QueryResult.ts`: No Metroid or knowledge gap fields - -| Field | Value | -|-------|-------| -| **File** | `cortex/QueryResult.ts` | -| **Component** | `QueryResult` interface | -| **Current behavior** | Contains only `pages`, `scores`, and `metadata`. No field for Metroid probe used, no knowledge gap field. | -| **Intended behavior** | Should include `metroid`, `knowledgeGap`, `coherencePath`, and `provenance` fields (see P1-E2). | -| **Required correction** | Upgrade `QueryResult` as part of P1-E2. | -| **TODO task** | P1-E2 | - ---- - -## Summary by Severity - -| Severity | Count | Description | -|----------|-------|-------------| -| **Critical (blocks MetroidBuilder)** | 3 | D1, D2, D3 — type/interface naming collision | -| **High (architectural gap)** | 4 | D4, D6, D13, D14 — missing types and IDB store | -| **Medium (propagated naming error)** | 4 | D5, D7, D12, D15 — implementations following wrong names | -| **Resolved by this PR** | 4 | D8, D9, D10, D11 — corrected in DESIGN.md v1.2 and PLAN.md v1.2 | - -**Total: 15 divergences** (3 + 4 + 4 + 4) - ---- - -## Components with Zero Drift - -The following components are correctly implemented (or partially implemented in the correct direction) and require no changes related to this naming review: - -- `core/HotpathPolicy.ts` — Williams Bound policy implementation; correct -- `core/SalienceEngine.ts` — Promotion/eviction lifecycle; correct -- `core/crypto/` — Hash, sign, verify; correct -- `storage/OPFSVectorStore.ts` — Append-only vector file; correct -- `storage/MemoryVectorStore.ts` — In-memory testing backend; correct -- `embeddings/` — All embedding providers; correct -- `hippocampus/Chunker.ts` — Text chunking; **implemented and correct** -- `hippocampus/PageBuilder.ts` — Page entity construction; **implemented and correct** -- `hippocampus/Ingest.ts` — Minimal ingest path; **partially implemented** (chunk→embed→persist→Book→hotpath); correct direction, hierarchy and neighbor insertion deferred -- `cortex/Query.ts` — Minimal query path; **partially implemented** (hotpath-first flat scoring); **must be substantially rewritten** for the dialectical pipeline (P1-E) -- `cortex/QueryResult.ts` — Minimal result DTO; **partially implemented**; **must be rewritten** to add coherencePath, metroid, knowledgeGap, provenance fields (P1-E2) -- All `VectorBackend` implementations — correct - -> **Important caveat on "zero drift":** -> -> - **What it means:** No architectural logic in these files conflicts with the corrected design. They do not need to be deleted or redesigned from scratch. -> - **What it does not mean:** Unaffected by future work. The "roughed in" implementations (`Ingest.ts`, `Query.ts`, `QueryResult.ts`) were scaffolded before the MetroidBuilder design was fully specified. -> - **Impact:** `Query.ts` and `QueryResult.ts` must be substantially rewritten (P1-E); `Ingest.ts` must gain hierarchy building and neighbor insertion (P1-B, P1-C). Each is a correct stub in the right direction, but not a complete implementation. -> - **Authoritative status:** Refer to **PLAN.md**, not this section, when assessing whether a file needs additional work. - ---- - -## Recommended Fix Order - -1. **P0-X1–X7** — Fix naming drift in `core/types.ts`, `storage/IndexedDbMetadataStore.ts`, `cortex/Query.ts`, and planned file names. This unblocks MetroidBuilder without risking collision. -2. **P1-M1–M3** — Add `Metroid` and `KnowledgeGap` types; implement `MetroidBuilder`. -3. **P1-N1–N4** — Implement `KnowledgeGapDetector`. -4. **P1-E1–E3** — Rewrite `cortex/Query.ts` to full dialectical orchestrator (not backward-compatible with existing flat top-K code). -5. **P1-C1–C3** — Implement `FastNeighborInsert` (correctly named after P0-X). diff --git a/DESIGN.md b/DESIGN.md index 2501836..df4b5bc 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -495,8 +495,6 @@ interface Edge { #### Semantic Neighbor (Proximity Edge) Sparse radius-graph edge connecting pages with high cosine similarity. Used for subgraph expansion during retrieval. -> **Note:** The current codebase names this type `MetroidNeighbor` — this is an architectural naming error introduced by early conceptual drift. The correct term is `SemanticNeighbor` (or equivalent). A code-level rename is tracked in the TODO. The edge is a proximity concept, not a Metroid concept. - **Critical distinction — two edge types, two roles:** | Edge type | Storage | Role | @@ -525,8 +523,6 @@ interface SemanticNeighbor { #### Semantic Neighbor Subgraph Induced subgraph for BFS-based coherence path expansion. -> **Note:** Currently named `MetroidSubgraph` in the codebase — same renaming correction applies. - ```typescript interface SemanticNeighborSubgraph { nodes: Hash[]; @@ -587,7 +583,7 @@ Structured entity storage with automatic reverse indexes. **Object Stores:** - `pages`, `books`, `volumes`, `shelves` - `edges_hebbian` (Hebbian weights) -- `neighbor_graph` (sparse semantic neighbor graph — currently named `metroid_neighbors` in code; rename tracked in TODO) +- `neighbor_graph` (sparse semantic neighbor graph) - `flags` (dirty-volume recalc markers) - `page_to_book`, `book_to_volume`, `volume_to_shelf` (reverse indexes) - `hotpath_index` (periodic HOT-membership checkpoint, keyed by `entityId`; loaded on startup to reconstruct the RAM resident index; written by Daydreamer each maintenance cycle) @@ -789,7 +785,7 @@ Matryoshka dimensional unwinding. Runs the thesis→freeze→antithesis→synthe search; m2 via cosine-opposite medoid; c computed once and frozen; subsequent candidates evaluated relative to frozen c. Planned module: `cortex/MetroidBuilder.ts`. -**Semantic neighbor graph** (also: proximity graph, neighbor graph): The sparse radius-graph of cosine-similarity edges between pages, used for subgraph expansion during retrieval. This is **not** the same as a Metroid. The edges connect pages with high cosine similarity and are used for BFS expansion. Currently named `MetroidNeighbor` / `metroid_neighbors` in the codebase — this is a naming error that must be corrected (tracked in TODO as P0-X). +**Semantic neighbor graph** (also: proximity graph, neighbor graph): The sparse radius-graph of cosine-similarity edges between pages, used for subgraph expansion during retrieval. This is **not** the same as a Metroid. The edges connect pages with high cosine similarity and are used for BFS expansion. **Hotpath**: The in-memory resident index of H(t) entries spanning all four hierarchy tiers. The hotpath is the first lookup target for every query; misses spill to WARM/COLD storage. HOT membership and salience are checkpointed to the `hotpath_index` IndexedDB store by Daydreamer each maintenance cycle, allowing the RAM index to be restored after a page reload or machine reboot without full corpus replay. diff --git a/PLAN.md b/PLAN.md index 0d8c942..910840a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -98,7 +98,7 @@ This document tracks the implementation status of each major module in CORTEX. I | Dialectical Search Pipeline | ❌ Missing | `cortex/DialecticalSearch.ts` (planned) | Orchestrates thesis/antithesis/synthesis zone exploration using a Metroid; prevents confirmation bias | | Knowledge Gap Detector | ❌ Missing | `cortex/KnowledgeGapDetector.ts` (planned) | Determines when MetroidBuilder cannot find m2; emits curiosity probe | | Seed Selection | ❌ Missing | `cortex/SeedSelection.ts` (planned) | Threshold-based top-k page selection from ranking output | -| Subgraph Expansion | 🟡 Partial | `storage/IndexedDbMetadataStore.ts` (`getInducedMetroidSubgraph` — to be renamed `getInducedNeighborSubgraph`) | BFS expansion implemented in storage layer; needs dynamic Williams bounds; needs orchestration wrapper | +| Subgraph Expansion | 🟡 Partial | `storage/IndexedDbMetadataStore.ts` (`getInducedNeighborSubgraph`) | BFS expansion implemented in storage layer; needs dynamic Williams bounds; needs orchestration wrapper | | Open TSP Solver | ❌ Missing | `cortex/OpenTSPSolver.ts` (planned) | Dummy-node open-path heuristic for coherent ordering | | Query Orchestrator | 🟡 Needs Rework | `cortex/Query.ts` | Flat top-K scoring implemented (hotpath-first → warm/cold spill → PageActivity update → promotion sweep). **Must be substantially reworked** to implement the full dialectical pipeline: replace flat scoring with hierarchical resident-first ranking, add MetroidBuilder, dialectical zone scoring (thesis/antithesis/synthesis), subgraph expansion with dynamic Williams bounds, TSP coherence path, and query cost meter. The existing implementation does not use Hebbian edges or cosine-similarity-bounded subgraph expansion; it is a functional placeholder only. | | Result DTO | 🟡 Needs Rework | `cortex/QueryResult.ts` | Minimal DTO (`pages`, `scores`, `metadata`). **Must be reworked** to add `coherencePath: Hash[]`, `metroid?: { m1, m2, centroid }`, `knowledgeGap?: KnowledgeGap`, and `provenance: { subgraphSize, hopCount, edgeWeights, vectorOpCost, earlyStop }`. | @@ -116,7 +116,7 @@ This document tracks the implementation status of each major module in CORTEX. I | Idle Scheduler | ❌ Missing | `daydreamer/IdleScheduler.ts` (planned) | Cooperative background loop; interruptible; respects CPU budget | | Hebbian Updates | ❌ Missing | `daydreamer/HebbianUpdater.ts` (planned) | LTP (strengthen), LTD (decay), prune below threshold; recompute σ(v) for changed nodes; run promotion/eviction sweep | | Prototype Recomputation | ❌ Missing | `daydreamer/PrototypeRecomputer.ts` (planned) | Recalculate volume/shelf medoids and centroids; recompute salience for affected entries; run tier-quota promotion/eviction | -| Full Neighbor Graph Recalc | ❌ Missing | `daydreamer/FullNeighborRecalc.ts` (planned) | Rebuild bounded neighbor lists for dirty volumes; batch size bounded by O(√(t log t)) per idle cycle; recompute salience after recalc. **Note:** Currently planned as `FullMetroidRecalc` — this is a naming error; see TODO P0-X. | +| Full Neighbor Graph Recalc | ❌ Missing | `daydreamer/FullNeighborRecalc.ts` (planned) | Rebuild bounded neighbor lists for dirty volumes; batch size bounded by O(√(t log t)) per idle cycle; recompute salience after recalc. | | Experience Replay | ❌ Missing | `daydreamer/ExperienceReplay.ts` (planned) | Simulate queries to reinforce connections | | Cluster Stability | ❌ Missing | `daydreamer/ClusterStability.ts` (planned) | Detect/trigger split/merge for unstable clusters; run lightweight label propagation for community detection; store community labels in PageActivity | @@ -400,9 +400,9 @@ This document tracks the implementation status of each major module in CORTEX. I **Impact:** Core discovery-sharing value proposition is missing; knowledge gaps cannot be resolved via P2P. **Mitigation:** Phase 3 required track; implement eligibility classifier + curiosity broadcaster + signed subgraph exchange as v1 scope. CuriosityProbe must include `mimeType` and `modelUrn` to prevent incommensurable graph merges. -### Blocker 4: Naming Drift (P0-X) -**Impact:** The term "Metroid" is currently used for the proximity graph in all code. MetroidBuilder cannot be introduced without a rename collision. -**Mitigation:** P0-X tasks (rename `MetroidNeighbor` → `SemanticNeighbor`, etc.) must be completed before MetroidBuilder is implemented. +### Blocker 4: Naming Drift (P0-X) — RESOLVED +**Impact:** The term "Metroid" was used for the proximity graph in all code. MetroidBuilder cannot be introduced without a rename collision. +**Resolution:** P0-X rename completed. `SemanticNeighbor`, `SemanticNeighborSubgraph`, and all `*SemanticNeighbors`/`*NeighborRecalc` method names are now in place throughout `core/types.ts`, `storage/IndexedDbMetadataStore.ts`, `cortex/Query.ts`, and all test files. The IDB object store is `neighbor_graph` (DB_VERSION=3). ### Risk 1: TSP Complexity Open TSP is NP-hard; heuristic may be slow on large subgraphs. @@ -485,7 +485,7 @@ After every implementation pass: ## Notes -- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` refers only to the dialectical search probe `{ m1, m2, c }` constructed by `MetroidBuilder` at query time. `medoid` refers to a cluster representative node. The sparse proximity/neighbor graph (used for BFS subgraph expansion) is the **semantic neighbor graph** — it is currently misnamed `MetroidNeighbor`/`MetroidSubgraph` in code (see TODO P0-X for the rename task). +- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` refers only to the dialectical search probe `{ m1, m2, c }` constructed by `MetroidBuilder` at query time. `medoid` refers to a cluster representative node. The sparse proximity/neighbor graph (used for BFS subgraph expansion) is the **semantic neighbor graph** — represented by `SemanticNeighbor` / `SemanticNeighborSubgraph` in `core/types.ts` and stored in the `neighbor_graph` IDB object store. - **Model-derived numerics:** Never hardcode; always source from `core/` model profile modules. - **Policy-derived constants:** Never hardcode; always source from `core/HotpathPolicy.ts`. - **Test philosophy:** TDD (Red → Green → Refactor) for all new slices. diff --git a/README.md b/README.md index cf92341..99bf30a 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ This is the "dreaming" phase that prevents catastrophic forgetting and forces ab - **Biological Scarcity** — Only a fixed number of active prototypes live in memory. Everything else is gracefully demoted to disk. - **Sublinear Growth (Williams Bound)** — The resident hotpath index is bounded to H(t) = ⌈c·√(t·log₂(1+t))⌉ where t = total graph mass (pages + edges). Memory scales sublinearly as the graph grows, trading time for space at a mathematically principled rate. See [`DESIGN.md`](DESIGN.md) for the full theorem mapping. -- **Three-Zone Memory** — HOT (resident in-memory index, capacity H(t)), WARM (indexed in IndexedDB, reachable via nearest-neighbour search), COLD (metadata in IndexedDB + raw vectors in OPFS, but semantically isolated from the search path — no strong nearest neighbours in vector space at insertion time; only discoverable by a deliberate random walk). All data is retained locally forever; zones control lookup cost and discoverability, not data lifetime. +- **Three-Zone Memory** — HOT (resident in-memory index, capacity H(t)), WARM (indexed in IndexedDB, reachable via nearest-neighbor search), COLD (metadata in IndexedDB + raw vectors in OPFS, but semantically isolated from the search path — no strong nearest neighbors in vector space at insertion time; only discoverable by a deliberate random walk). All data is retained locally forever; zones control lookup cost and discoverability, not data lifetime. - **Hierarchical & Sparse** — Progressive dimensionality reduction + medoid clustering keeps memory efficient at any scale, with Williams-derived fanout bounds preventing any single tier from monopolising the index. - **Hebbian & Dynamic** — Connections strengthen and weaken naturally. Node salience (σ = α·H_in + β·R + γ·Q) drives promotion into and eviction from the resident hotpath. - **Zero-Copy & Persistent** — OPFS + IndexedDB with cryptographic signing. @@ -116,7 +116,6 @@ bun run dev:harness # start the browser runtime harness at http://127.0.0.1:4173 | [`DESIGN.md`](DESIGN.md) | Architecture specification and core design principles | | [`PLAN.md`](PLAN.md) | Module-by-module implementation status and development phases | | [`TODO.md`](TODO.md) | Prioritized actionable tasks to ship v1.0 | -| [`ARCHITECTURE-REVIEW.md`](ARCHITECTURE-REVIEW.md) | Repository-wide architectural drift report and correction tasks | | [`docs/api.md`](docs/api.md) | API reference for developers integrating with CORTEX | | [`docs/development.md`](docs/development.md) | Build, test, debug, and Docker workflow | diff --git a/TODO.md b/TODO.md index bc21997..5aab3a2 100644 --- a/TODO.md +++ b/TODO.md @@ -208,17 +208,17 @@ These items **must** be completed to have a usable system. Without them, users c **Why:** The codebase uses the term "Metroid" to name the sparse proximity/neighbor graph (`MetroidNeighbor`, `MetroidSubgraph`, `metroid_neighbors`, `getInducedMetroidSubgraph`, `FastMetroidInsert`, `FullMetroidRecalc`). This is architecturally incorrect. In CORTEX, a **Metroid** is a structured dialectical search probe `{ m1, m2, c }` — a concept that does not yet exist in the codebase at all. The proximity graph has nothing to do with Metroids. This naming collision will cause permanent confusion and make the MetroidBuilder impossible to implement cleanly without a rename. -- [ ] **P0-X1:** Rename `MetroidNeighbor` → `SemanticNeighbor` in `core/types.ts` +- [x] **P0-X1:** Rename `MetroidNeighbor` → `SemanticNeighbor` in `core/types.ts` - Update all references in `storage/IndexedDbMetadataStore.ts` - Update all references in test files - Update JSDoc and inline comments -- [ ] **P0-X2:** Rename `MetroidSubgraph` → `SemanticNeighborSubgraph` in `core/types.ts` +- [x] **P0-X2:** Rename `MetroidSubgraph` → `SemanticNeighborSubgraph` in `core/types.ts` - Update all references in `storage/IndexedDbMetadataStore.ts` - Update all references in `cortex/Query.ts` - Update JSDoc and inline comments -- [ ] **P0-X3:** Rename `MetadataStore` proximity graph methods: +- [x] **P0-X3:** Rename `MetadataStore` proximity graph methods: - `putMetroidNeighbors` → `putSemanticNeighbors` - `getMetroidNeighbors` → `getSemanticNeighbors` - `getInducedMetroidSubgraph` → `getInducedNeighborSubgraph` @@ -227,17 +227,15 @@ These items **must** be completed to have a usable system. Without them, users c - `clearMetroidRecalcFlag` → `clearNeighborRecalcFlag` - Update all callers in `storage/IndexedDbMetadataStore.ts`, `cortex/Query.ts`, and test files -- [ ] **P0-X4:** Rename planned Hippocampus file `hippocampus/FastMetroidInsert.ts` → `hippocampus/FastNeighborInsert.ts` +- [x] **P0-X4:** Rename planned Hippocampus file `hippocampus/FastMetroidInsert.ts` → `hippocampus/FastNeighborInsert.ts` - Rename class/function to `FastNeighborInsert`/`insertSemanticNeighbors` -- [ ] **P0-X5:** Rename planned Daydreamer file `daydreamer/FullMetroidRecalc.ts` → `daydreamer/FullNeighborRecalc.ts` +- [x] **P0-X5:** Rename planned Daydreamer file `daydreamer/FullMetroidRecalc.ts` → `daydreamer/FullNeighborRecalc.ts` - Rename class/function to `FullNeighborRecalc`/`runNeighborRecalc` -- [ ] **P0-X6:** Rename IndexedDB object store from `metroid_neighbors` → `neighbor_graph` - - Increment `DB_VERSION` in `storage/IndexedDbMetadataStore.ts` - - Add migration in `applyUpgrade` to copy data from old store to new store +- [x] **P0-X6:** Rename IndexedDB object store from `metroid_neighbors` → `neighbor_graph` -- [ ] **P0-X7:** Update all documentation strings and JSDoc that use "Metroid neighbor" to use "semantic neighbor" +- [x] **P0-X7:** Update all documentation strings and JSDoc that use "Metroid neighbor" to use "semantic neighbor" **Exit Criteria:** No source file uses "Metroid" to refer to the proximity graph. The term "Metroid" is reserved exclusively for the `{ m1, m2, c }` dialectical probe type implemented in `cortex/MetroidBuilder.ts`. @@ -863,7 +861,7 @@ These items improve quality, performance, and developer experience. Not blockers If you're reading this and want to know "what do I work on right now?", here's the answer: **Immediate (unblock MetroidBuilder):** -1. **P0-X1–X7:** Fix architectural naming drift (`MetroidNeighbor` → `SemanticNeighbor` and related renames) +1. ~~**P0-X1–X7:** Fix architectural naming drift (`MetroidNeighbor` → `SemanticNeighbor` and related renames)~~ ✅ DONE **After P0-X (complete v0.1):** 2. **P0-B1:** Implement `hippocampus/Chunker.ts` diff --git a/core/types.ts b/core/types.ts index d8cc52b..3353584 100644 --- a/core/types.ts +++ b/core/types.ts @@ -64,7 +64,7 @@ export interface Edge { } // --------------------------------------------------------------------------- -// Semantic nearest-neighbour graph +// Semantic nearest-neighbor graph // --------------------------------------------------------------------------- export interface SemanticNeighbor { diff --git a/docs/api.md b/docs/api.md index 5f55eb6..a1dabeb 100644 --- a/docs/api.md +++ b/docs/api.md @@ -112,18 +112,18 @@ interface Edge { } ``` -#### `MetroidNeighbor` +#### `SemanticNeighbor` -A nearest-neighbour entry in the Metroid radius graph (a project-domain term for the medoid-inspired NN graph). +A nearest-neighbor entry in the semantic neighbor radius graph — a sparse proximity graph connecting pages with high cosine similarity, used for BFS subgraph expansion during retrieval. ```typescript -interface MetroidNeighbor { +interface SemanticNeighbor { neighborPageId: Hash; cosineSimilarity: number; // threshold defined by runtime policy distance: number; // 1 – cosineSimilarity (TSP-ready) } -interface MetroidSubgraph { +interface SemanticNeighborSubgraph { nodes: Hash[]; edges: { from: Hash; to: Hash; distance: number }[]; } @@ -180,20 +180,20 @@ interface MetadataStore { getVolumesByBook(bookId: Hash): Promise; getShelvesByVolume(volumeId: Hash): Promise; - // Metroid NN radius index - putMetroidNeighbors(pageId: Hash, neighbors: MetroidNeighbor[]): Promise; - getMetroidNeighbors(pageId: Hash, maxDegree?: number): Promise; + // Semantic neighbor radius index + putSemanticNeighbors(pageId: Hash, neighbors: SemanticNeighbor[]): Promise; + getSemanticNeighbors(pageId: Hash, maxDegree?: number): Promise; - /** BFS expansion of the Metroid subgraph up to `maxHops` levels deep. */ - getInducedMetroidSubgraph( + /** BFS expansion of the semantic neighbor subgraph up to `maxHops` levels deep. */ + getInducedNeighborSubgraph( seedPageIds: Hash[], maxHops: number, - ): Promise; + ): Promise; // Dirty-volume recalculation flags - needsMetroidRecalc(volumeId: Hash): Promise; - flagVolumeForMetroidRecalc(volumeId: Hash): Promise; - clearMetroidRecalcFlag(volumeId: Hash): Promise; + needsNeighborRecalc(volumeId: Hash): Promise; + flagVolumeForNeighborRecalc(volumeId: Hash): Promise; + clearNeighborRecalcFlag(volumeId: Hash): Promise; } ```