Implement semantic memory consolidation and retrieval defragmentation

[epi13]

Summary

Adds the first executable RAVEL memory component: a conservative, provenance-preserving prototype for semantic consolidation and access-pattern-based retrieval layout planning.

This implements the two related ideas behind the original analogy:

• semantic consolidation reorganizes duplicated or fragmented memory into challengeable canonical proposals; and
• retrieval defragmentation derives disposable co-location buckets from memories that are repeatedly selected together.

What changed

• adds immutable typed source-memory and derived-proposal records;
• adds deterministic, scope-bounded near-duplicate clustering;
• preserves explicit contradiction and supersession links;
• adds stable proposal identities, retrieval keys, and clustering confidence;
• adds an append-only SQLite store with identity-reuse protection and deterministic JSONL replay;
• adds a retrieval-layout planner based on observed co-access;
• adds a JSON Schema for consolidation proposals;
• documents the architecture, invariants, validation plan, and implementation path;
• adds seven unit tests and Python 3.11–3.13 CI.

Safety and authority boundaries

This prototype does not delete or rewrite source memories. Consolidations remain derived proposed records and cannot promote principles, alter evaluator results, broaden scope, or convert UNKNOWN into PASS.

The deterministic token/Jaccard baseline is intentionally simple and inspectable. Embeddings can later assist candidate generation, but only behind the same scope, provenance, contradiction, negative-memory, and authority controls.

Validation

Local validation completed:

python -m unittest discover -s tests -v
Ran 7 tests
OK

Also validated editable package installation with no build isolation and compiled all source/test modules.

Follow-on work

1. Bind these records to the versioned RAVEL evidence and experience schemas.
2. Add explicit scope-compatibility contracts.
3. Add full-text retrieval and a preregistered benchmark corpus.
4. Add proposal review/challenge/supersession records.
5. Rebuild graph projections from the append-only stream.
6. Benchmark page, cache, shard, and vector-read layouts under equal workloads.
7. Add embedding-assisted candidate generation only after measuring the deterministic baseline.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.