Optional derived vector index

[valiantone]

Summary

Add an optional derived vector index for HotMem retrieval acceleration.

The vector index may use Chroma or a similar local backend, but it must remain disposable and rebuildable. SQLite/files/bundles/manifests remain canonical.

Related OKF docs

• Architecture: docs/okf/file-aware-architecture.md
• Epic: docs/okf/file-native-epic.md
• Practices: docs/okf/file-native-memory-practices.md
• OKF migration PR: [codex] Adopt OKF docs for file-native memory planning #47

Related vNext issues

• File-backed memories (URI + range + checksum hydration) #38 File-backed memories
• Hydration Profiles (agent/compact/audit/full) #40 Hydration Profiles
• API extensions (/v1/files, /v1/export, hydrate params) #43 API extensions

Goal

Provide optional vector acceleration while preserving HotMem's filesystem-first architecture.

Scope

• Add opt-in index configuration.
• Rebuild index from SQLite and file/bundle references.
• Search falls back to deterministic local retrieval if the index is missing.
• Index metadata records source hashes or rebuild markers.
• Document that the vector index is not canonical storage.

Acceptance criteria

• HotMem runs with no vector backend installed.
• Index can be rebuilt from canonical storage.
• Search fallback works when the index is absent or stale.
• Index loss does not lose memory.
• Tests prove default behavior remains unchanged.

Out of scope

• Mandatory Chroma dependency.
• Canonical vector DB architecture.
• Remote vector services.

Priority

P1 after file-backed memory, bundle discovery, and hydration profile semantics are stable.

TECH-11 execution requirements

Full execution plan: TECH-11.

Additional scope

• Support building, rebuilding, and clearing the index from canonical memory records.
• Make index staleness, missing dependencies, and rebuild state observable.
• Preserve the default response shape and ranking contract when acceleration is enabled or absent.
• Index file-backed content from eligible inline summaries/metadata by default; never eagerly load large backing files.
• File-backed and bundle-backed memories preserve lazy content-read rules.

Required tests

• Core operation without an optional vector package installed.
• Rebuild parity and stale-index handling.
• Search compatibility golden tests from Compatibility golden tests for file-native evolution #54.
• Proof that large referenced files are not eagerly read for indexing.
• Deterministic fallback-search behavior.

[valiantone]

Expanded filesystem-first implementation set:

Core existing vNext spine:

• File-backed memories (URI + range + checksum hydration) #38 File-backed memories
• Snapshot v2 directory format (manifest + jsonl + attachments) #39 Snapshot directory format
• Hydration Profiles (agent/compact/audit/full) #40 Hydration Profiles
• Append-only Event Log + /v1/events #41 Append-only Event Log
• Promotion lifecycle (HOT→READY→PROMOTED→ARCHIVED, signal only) #42 Promotion lifecycle
• API extensions (/v1/files, /v1/export, hydrate params) #43 API extensions

New supporting tickets:

• Compatibility golden tests for file-native evolution #54 Compatibility golden tests for file-native evolution
• Bundle Store: loose local markdown bundle reader #52 Bundle Store: loose local markdown bundle reader
• Bundle indexing and discovery #50 Bundle indexing and discovery
• Lightweight file inspectors for CSV, JSONL, and Parquet metadata #53 Lightweight file inspectors for CSV, JSONL, and Parquet metadata
• Optional derived vector index #49 Optional derived vector index
• Local hygiene and store growth warnings #51 Local hygiene and store growth warnings
• Native helper spike: Rust, C, or WebAssembly fast-path primitives #48 Native helper spike: Rust, C, or WebAssembly fast-path primitives

Planning sources:

• OKF architecture: docs/okf/file-aware-architecture.md
• OKF epic: docs/okf/file-native-epic.md
• OKF practices: docs/okf/file-native-memory-practices.md
• OKF migration PR: [codex] Adopt OKF docs for file-native memory planning #47

Implementation rule: GitHub issues own scope and acceptance criteria; OKF docs own rationale, boundaries, and heuristics.