diff --git a/packages/kits/CorpusKit/docs/AGENT_MAP.md b/packages/kits/CorpusKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..cba0ea4 --- /dev/null +++ b/packages/kits/CorpusKit/docs/AGENT_MAP.md @@ -0,0 +1,243 @@ +--- +doc: AGENT_MAP +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# AGENT_MAP : CorpusKit + +PURPOSE: standalone on-device RAG kit. Text → chunks (content-addressed UUID) → BundleStore (PersistenceKit) + persistent BM25 inverted index + per-provider vectors (VectorKit) → hybrid recall (Hamming kNN + BM25, weighted RRF) → [ScoredChunk]. Ships two targets: CorpusKit (core: stores, engines, protocols) and CorpusKitProviders (concrete embedding providers/tokenizers). Default production ensemble = five honest deterministic signals (RI/PPMI/LSA/NMF/FDC). + +DEPS: CorpusKit imports SubstrateTypes, SubstrateLib (MerkleHash), SubstrateML, EngramLib, EideticLib (sentence segmentation), IntellectusLib (telemetry, off-by-default), PersistenceKit (+InMemory, +SQLite), ConvergenceKit (manifest only), VectorKit, QueueKit, Crypto. CorpusKitProviders additionally imports SubstrateKernel (FloatVecOps), LatticeLib (FDC runtime; FDC math NOT reimplemented). Imported by: GeniusLocusKit (orchestrator tier). Rust ports: `rust/` (core, crate corpus-kit) + `rust-providers/` (crate corpus-kit-providers); shared fixtures `Tests/SharedVectors/*.json` read by BOTH legs gate bit-identity. NL providers are Swift-only (ADR-019, no Rust twin). + +ENTRY POINTS (most callers need only these): +- CorpusKit.swift:702 `Corpus.init(storage:models:)` : open estate corpus; `models[0]` = default signal (:674 single-model convenience) +- CorpusKit.swift:1001 `Corpus.ingest(_ text:sourceID:now:)` : synchronous chunk+index+embed +- CorpusIngestQueue.swift:157 `Corpus.enqueueIngest(_:sourceID:now:)` : async queued ingest (production path) +- CorpusKit.swift:1629 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` : hybrid RRF recall on default signal +- DefaultEnsemble.swift:62 `CorpusEnsemble.defaultEnsemble() -> [EmbeddingModel]` : the five production signals, fresh per estate + +## Symbol Table + +### Facade : CorpusKit.swift +- :48 `enum FloatLaneOutcome` : `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors +- :134 `enum EmbeddingModel` : `.deterministic` (:147, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:157/:167/:177, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:194–:233, trainable), `.fdc(provider:)` (:254, stateless), `.nlEmbedding/.nlContextualEmbedding` (:275/:293, Apple-only); `.default = .deterministic` (:297) +- :338 `EmbeddingModel.isTrainable` : true iff carried provider conforms to TrainableEmbeddingBasis +- :362 `EmbeddingModel.reconstruct(from: Data)` : routes blob to concrete type; throws `.notTrainable` +- :417 `enum EncodeSpeed` : `.foreground` (all cores) / `.background` (cores/4, floor 1) +- :426 `public actor Corpus` : composition root; sealed-vector principle (no VectorKit type in public API except :649 `sharedVectorStore`) +- :491 `setEncodeSpeed(_:)`; :641 `onEncoded` callback (set via CorpusIngestQueue.swift:134 `setOnEncoded`) +- :1001 `ingest(_:sourceID:now:)` : idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence) +- :1181 `ingestBatch(_:)` : identical output to per-item; commit windows 512 items / 4096 rows (:436–:437); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item) +- :1416 `maintainedVocabAnchor() -> Int` : governor's vocab-growth retrain trigger read +- :1488 `reindex(now:)` : THE explicit retrain: reconstruct-fresh → trainOnCorpus(active chunks) → upsert basis → re-embed all under every slot; excludes removed sources +- :1629 `recall(_:limit:now:)`; :1657 `remove(sourceID:)` (BM25 rows + ALL models' vectors + tombstone; chunks kept); :1711 `expunge(sourceID:)` (scrubText FIRST, then remove); :1738 `destroyRecallIndex()` (all derived state; chunks survive) +- :1779 `bm25TopKBySource(query:limit:)` : pure keyword lane, max-chunk-score per source, frontierK ≤ 256 +- :1833 `embed(_:) -> Engram`; :1846 `modelID`; :1863 `embedFloat(_:)` (throws on provider opt-out); :2238 `supportsFloat` +- :1895 `floatNearest(query:limit:) -> FloatLaneOutcome` : never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine +- :2141 `floatNearestPerSignal(query:limit:)` : per-slot dense lanes in slot order, NO fusion (caller fuses); :2214 `floatFarthestPerSignal` : anti-similarity, MIN chunk cosine per source, ascending +- :2249 `count()` (excludes removed sources); :2266 `indexedSourceIDs()`; :2274/:2280 `corpusMerkleRoot(for:)`/`globalCorpusMerkleRoot()` +- :2300 `EmbeddingModel.makeProvider()` (private) : pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m +- :2413 `CorpusDefaultTokenizer` (internal) : FNV-1a fold, duplicated from providers to avoid circular dep; :2445 `CorpusTextProvider` (private) : tokenize→inference→FloatSimHash; :2498 `embedPair` computes pooled vector ONCE for both lanes +- Test seams (never production): :914 init(storage:provider:), :980 `_testForceFloatStoreError`, :656 `_ingestFailureHook` + +### Ingest queue : CorpusIngestQueue.swift (extension Corpus) +- :63 `mountIngestQueue()` : idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :486 `ingestQueueStoreID` (no UUID() nondeterminism) +- :120 `dropIngestQueue()`; :134 `setOnEncoded(_:)` : the ONLY CorpusKit→orchestrator callback +- :157 `enqueueIngest(...)`; :179 `enqueueIngestBatch(...)` : one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size) +- :205 `awaitIngestDrain(timeout: 30s)` : barrier: drained AND vector index republished; throws drainTimeout +- :229 `ingestQueueDepth() -> (pending, inFlight)` : pending IS stream-scoped, inFlight is NOT (all streams) +- :248 `drainIngestQueueOnce()` : claims whole batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw +- :341 `runIngestDrainLoop` (private) : DrainLease single drainer; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²)) +- :430 `ingestOneAndReply` (private) : retry in place ≤ :476 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent +- :480 `encodeStreamID = "encode"` : EVERY queue op must be scoped to it (shared queue.sqlite may carry other streams; unscoped awaitDrain deadlocks) +- :515 `IngestJob` : wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract; :551 `toJob`, :563 `from(job:)` + +### Chunks : Chunk.swift / Chunker.swift +- Chunk.swift:35 `struct Chunk: Sendable, Equatable, Codable` : immutable, content-addressed +- Chunk.swift:68 content-addressed init (normal path); :90 explicit-id init (reconstruction; caller must guarantee id matches content) +- Chunk.swift:129 `Chunk.deriveID(sourceID:startOffset:text:)` : RFC 4122 v5 (SHA-1) over fields joined by \u{1F}; :114 `namespaceBytes` PERMANENT (change re-keys fleet + breaks vector join) +- Chunk.swift:149 `struct ScoredChunk` : chunk + score/vectorScore/keywordScore (per-lane preserved) +- Chunker.swift:28 `ChunkerConfiguration` : targetChars 800 / overlapChars 100 / respectSentences true; init clamps (overlap < target) +- Chunker.swift:49 `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` : EideticLib.sentences segmentation; greedy fill + tail overlap; hlcGenerator `inout`, stamps emission order; offsets are Character counts + +### Tokenization : Tokenizer.swift +- :10 `protocol Tokenizer: Sendable` : vocabID/maxTokens/padTokenID/unknownTokenID; :27 `tokenize(_:) -> [Int32]` (truncation is implementer's job); :33 `keywordTokens(_:)` (default :62) +- :44 `defaultKeywordTokens(_:)` : lowercase + alphabetic/ASCII-digit runs; THE single keyword tokenizer for BM25 + all distributional providers; parity-critical with Rust; overriding keywordTokens breaks the guarantee (convention, not compiler) + +### Errors : CorpusKitError.swift +- :5 `enum CorpusKitError: Error, Sendable, Equatable` : encodingFailure/decodingFailure/tokenizerUnavailable/modelUnavailable/embeddingFailed/storeUnavailable/:18 notTrainable; Equatable on exact message strings + +### Chunk store : BundleStore.swift +- :94 `public actor BundleStore`; :139 `schemaDeclaration` v3 : `chunks` (10 cols incl. content_hash BLOB, ext JSON) + `corpus_metadata`; indices source_id, hlc +- :199 `init(storage:dirtyChainSink:)` : wraps HashingRowStore (MerkleHash.leaf per insert); :70 `ParentChainCache` bridges sync hash callback +- :268 `insert(_:)` : idempotent (duplicateKey = no-op, first write wins); RETURNS ONLY NEWLY-INSERTED chunks : derived-state callers must fold the returned subset, never the input +- :348 `get(id:asOf:)`; :361 `getMany(ids:asOf:)`; :375 `chunksForSource(_:asOf:)` (start_offset ASC); :399 `allSourceIDs` (full scan, maintenance only); :427 `chunkSourcePairs()` (body-free warm-load projection); :457 `count(asOf:)` : asOf accepted but IGNORED; :461 `allChunks(asOf:)` (hlc ASC) +- :500 `scrubText(sourceID:)` : hard-delete text zeroing via direct UPDATE (why schema appendOnly: false); leaves content_hash stale intentionally +- :564 `corpusMerkleRoot(for:)`; :583 `globalCorpusMerkleRoot()` : corpus/root UUIDs derived from fixed SHA-256 namespace strings (cross-port) +- :632 `decodeChunk(_:)` : MUST accept both TypedValue forms (SQLite primitive .text/.int AND InMemory semantic .uuid/.hlc); historical bug: semantic-only decoder dropped all chunks on reopen; InMemory-only tests cannot catch regressions here + +### Tombstones : RemovedSourceStore.swift +- :44 `public actor RemovedSourceStore`; :52 schemaDeclaration (own kitID "CorpusKitRemovedSources") +- :76 `markRemoved(_:now:)` (idempotent upsert; row presence IS the state : no Bool column); :90 `clearRemoved(_:)` (re-ingest = the undo); :99 `removedIDs() -> Set` : EVERY rebuild path (reindex, first-ingest train, count) must subtract this set (unenforced convention; resurrection bug class); :118 `deleteAll()` + +### Provider counts : CorpusProviderCountsStore.swift +- :71 `PersistedCounts` (modelID/modelVersion/counts blob/documentCount/vocabSize/updatedAt); :101 `CountsGrowthAnchor` (cheap pair, no blob) +- :112 `public actor CorpusProviderCountsStore`; :121 schema (kitID "CorpusKitCounts", PK (model_id, model_version), ext slot) +- :156 `upsert(_:)` : full-row replace (caller folds blob first, no atomic increment); :173 `load(...)`; :191 `growthAnchor(...)` : the staleness-check read; :210 `deleteAll()` +- STATUS: "HALF A" : counts persisted/restored, but `Corpus.reindex` still retrains from raw chunk text; counts-backed retrain + vector re-projection (HALF B) not wired + +### Sync : SyncManifest.swift +- :11 `enum CorpusKitSync`; :17 `manifest(zoneIdentifier:)` : chunks table, bidirectional, PK id, conflictPolicy `.appendOnly` (safe because content-addressed); kitID "CorpusKit", schemaVersion 1. Declarative only. Sync-layer appendOnly ≠ BundleStore schema `appendOnly: false` : different systems, same word + +### Sparse engine : Engine/SparseTypes.swift, Engine/BM25Weighting.swift, Engine/InvertedIndex.swift +- SparseTypes.swift:39 `typealias LaneTag = VectorKit.LaneTag` (single owner, avoids ambiguity); :56 `ImpactPosting` (impact Int32, quantized ONCE at build); :94 `SparseHit` (impact Float = int/100); :136 `FusedHit` (fusedScore + perLane raw scores; absent key = no hit in lane) +- BM25Weighting.swift:29 `BM25Parameters` : k1 1.5 / b 0.75 pinned defaults; :50 `quantizeImpact(_:)` : round HALF-TO-EVEN × 100 (Swift default rounding differs at .5 : do not "simplify"); :82 `buildTermIDMap` (sorted term-id assignment); :110 `build(termFreqs:docLengths:parameters:)` : float BM25 math exactly once; IDF = ln((N−df+0.5)/(df+0.5)+1); :164 `queryPairs` : OOV dropped, duplicates deduped, weight = 100 +- InvertedIndex.swift:37 `invertedIndexQuantScale = 100`; :42 `invertedIndexBlockSize = 128` (pinned for conformance traces); :116 `struct InvertedIndex: Sendable` : immutable after init (init sorts postings itemID ASC); :195 `enum Algorithm` .wand/.blockMaxWand (result-identical); :211 `topK(query:k:algorithm:)` : EXACT top-k, integer-only path, tie-break smaller itemID wins; :246 `exhaustiveScan(query:k:)` : DAAT conformance oracle, not production. Item IDs compare as STRINGS (uuidString lexicographic ≠ numeric UUID order, but consistent cross-port) + +### Persistent keyword index : Engine/InvertedIndexStore.swift +- :47 `public actor InvertedIndexStore`; :55 schemaDeclaration (kitID "InvertedIndexStore", tables iix_termfreqs/iix_doclens : RAW statistics only, weighted index derived+cached, so k1/b changes need no migration); :105 init (storage pre-opened/migrated); :114 `open()` (load mirrors, O(terms+docs), no chunk bodies) +- :159 `index(itemID:tokens:now:)` : atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :199 `remove(itemID:)`; :230 `buildIndex(parameters:)` (cached; invalidated per write); :253 `topK(queryTerms:k:parameters:algorithm:)`; :273 `deleteAll()`; :295 `documentCount` +- Rust twin owns a PRIVATE connection with begin/commit/rollback_batch; Swift shares estate storage : hence Corpus-managed transaction windows in ingestBatch + +### Legacy keyword index : BM25Index.swift +- :34 `public actor BM25Index` : in-memory, Chunk/UUID-typed; NO LONGER used by Corpus (kept public for external callers); :49 init(tokenizer:parameters:); :58 `index(_ chunks:)`; :78 `remove(_:)`; :95 `documentCount()`; :107 `topK(_ k:for tokens:)` : pre-tokenized input; tie-break uuidString ASC + +### Fusion : Engine/Fusion.swift +- :48 `enum Fusion`; :74 `fuse(rankedLists:laneScores:weights:rrfK: 60)` : fusedScore = Σ weight·1/(rrfK+rank), rank 1-based; per-lane dedup (best rank only); precondition rrfK > 0; sort fusedScore DESC, itemID ASC; :164 `fuse(scoredLists:weights:rrfK:)` : position = rank, CALLER must pre-sort +- MMR: `mmrLambda` exists only as a HybridRecallConfiguration field : NOT implemented anywhere; do not document MMR as active + +### Hybrid recall : HybridRecall.swift +- :33 `HybridRecallConfiguration` : vectorWeight 0.6 / keywordWeight 0.4 / rrfK 60 / mmrLambda nil (unread); :52 `enum HybridRecall` +- :84 `recall(probe:query:modelID:limit:vectorStore:invertedIndex:bundleStore:configuration:)` : candidateK = max(limit×4, 32) per lane; vector kNN filtered to modelID; UUID hits CANONICALIZED via UUID(uuidString:).uuidString (P3-secfix: lowercase Rust-written ids must fuse with uppercase Swift ids); vectorScore = Hamming (0 = best, kept), keywordScore 0 → nil (BM25 real matches strictly positive); unhydratable ids silently dropped; telemetry post-hoc (corpuskit.recall.*) + +### Trainable-basis seam : TrainableEmbeddingBasis.swift, BasisStore.swift +- TrainableEmbeddingBasis.swift:50 `protocol TrainableEmbeddingBasis: AnyObject, Sendable` : conformers: RI/PPMI/LSA/NMF only +- :70 `trainOnCorpus(texts:)` : ADDITIVE (never retrain a live provider : reconstruct fresh first); :78 `serializeBasis()`; :98 `reconstructBasis(from:)` : INSTANCE method (type-erased witness routes to concrete init(deserializing:)); round-trip law = identical embeddings; throws decodingFailure, never crashes +- :132 `addToCounts(text:)` / :141 `serializeCounts()` / :149 `restoreCounts(from:)` / :154 `countsVocabularySize` : P3 maintained-counts seam, batch-boundary snapshots; infrastructure only (reindex still trains from text) +- No wall-clock reads anywhere in training : pure function of (texts, seeds) +- Rust divergence: EmbeddingProvider is a supertrait there (no trait cross-cast) +- BasisStore.swift:67 `PersistedBasis`; :98 `public actor BasisStore`; :112 schema v2 (`corpus_provider_basis`, PK (model_id, model_version), trained_at TEXT ISO8601, trained_chunk_count anchor, ext slot); :151 `upsert(_:)` (in-place, one row per key); :177 `load(modelID:modelVersion:)`; :195 `deleteAll()`; decoder accepts BOTH TypedValue timestamp forms (same reopen-bug class as BundleStore) + +### Basis codec : BasisCodec.swift (CorpusKitProviders) +- :43 `basisFormatVersion: UInt8 = 1`; :52 `struct BasisWriter` : LE only; f32 as bitPattern (:94); strings u32-len UTF-8 (:99); maps byte-sorted keys (:122/:136 via :148 lhsLess raw-UTF-8 compare : matches Rust str Ord, NOT Swift String <) +- :168 `struct BasisReader` : :192 `expectMagic` / :206 `expectVersion` reject wrong/future blobs with decodingFailure; all reads bounds-checked. Frame = MAGIC(4) | version(1) | payload. No nested-map primitive (PPMI inlines its own) + +### Deterministic tokenizer : DeterministicTokenizer.swift +- :16 `struct DeterministicTokenizer: Tokenizer`; :23 init(vocabID "deterministic-v1", vocabSize 30522, maxTokens 128); :33 `tokenize(_:)` : FNV-1a 32 fold into [2, vocabSize); 0=pad 1=unk reserved; empty input → [pad]. NOT a model vocab : real-model output from these ids is garbage (real WordPiece/SentencePiece = v1.1 model-bundle mission) + +### Shared training inputs : TermDocumentCounts.swift, ReducedVocab.swift +- TermDocumentCounts.swift:56 `struct TermDocumentCounts` : :61 vocab (ENCOUNTER-ORDER indices, cross-port byte contract), :64 tfCounts, :68 dfCounts; :110 `addDocument(_:)` (full); :156 `addDocumentForCountsAnchor(_:)` (vocab+docCount only, no TF/DF : re-tokenize-at-retrain decision); :91 init(restoredVocab:documentCount:) : TF rows exist but EMPTY after restore; NOT thread-safe +- ReducedVocab.swift:33 `defaultReducedVocabCap = 512` (ADR-022); :38 `ReducedVocabulary` (keptTerms/termToColumn/fullIndexToColumn/size); :63 `selectReducedVocabulary(vocab:dfCounts:documentCount:cap:)` : below cap = exact NO-OP (fixture compatibility); above cap: drop df<2, rank df DESC, tie-break raw-UTF-8-byte order (Rust parity); documentCount reserved/unused + +### Honest signals : RandomIndexingProvider.swift, PpmiProvider.swift, LsaProvider.swift, NmfProvider.swift, FdcProvider.swift +- RandomIndexing: :89 riDimension 2048, :93 riNonzeros 10, :98 riWindow 4, :102 riProjectionSeed 0x5249_5F56_315F_4D58 "RI_V1_MX"; :121 `riIndexVector(term:)` : FNV64(lowercased) → SplitMix64, EXACTLY 20 draws (pos %2048 bias-free, sign &1), collision last-wins (constant draw count = cross-port PRNG alignment); :162 `final class RandomIndexingProvider`; :206 `train(terms:window:)` additive; :249 embed / :268 embedFloat / :308 embedPair; :377 serializeBasis "RIB1" (vocab IS the basis, no finalize); :452 serializeCounts "RICT" (same payload, distinct magic on purpose) +- PPMI: :107/:111/:115 same D/K/window as RI (shared index space); :120 ppmiProjectionSeed 0x5050_4D49_5F56_314D "PPMI_V1M"; :153 `final class PpmiProvider`; :226 `train` (counts); :282 `finalize()` : ppmi(t,c)=max(0, ln P(t,c) − ln P(t) − ln P(c)); contextVec = Σ ppmi·riIndexVector(c); idempotent; :346 embed/:365 embedFloat/:397 embedPair; :464 basis "PPB1" (derived vectors only, unnormalized in store); :521 counts "PPMC" (raw additive state incl. nested coCount); NOT plain RI : do not reduce +- LSA: :109 lsaProjectionSeed 0x4C53415F56315F4D "LSA_V1_M"; :114 lsaDefaultRank 64; :165 svdSweeps PINNED 30 (change invalidates all conformance vectors); :148 `final class LsaProvider`; :227 `train(document:)`; :258 `finalize()` : ReducedVocab → tf=ln(1+c), idf=max(0, ln((N+1)/(df+1))) → JacobiSVD (wide matrix transposed+swapped back); query fold-in (1/σ)·Vt·q, σ<1e-9 skipped; :487 `documentEmbedding(at:)` exact U·Σ; :547 basis "LSB1" (reduced vocab + idf + RAW U/σ/Vt, port-neutral); :632 counts "LSAC" (anchors only) +- NMF: :110 nmfProjectionSeed 0x4E4D465F56315F4D "NMF_V1_M"; :114 rank 32; :119 iterations 100; :124 nmfFactorizationSeed 0xDEADBEEFCAFEBABE (pinned); :159 `final class NmfProvider`; :266 `finalize()` : V (terms×docs), log-TF NO idf, ALS with tolerance=0 → FIXED iteration count (bit-identity device); query fold-in dot(W[:,r],q)/(‖W[:,r]‖²+1e-9); :499 basis "NMB1" (W and H raw); :588 counts "NMFC" (anchors only) +- FDC: :106 fdcDimension 256; :111 fdcProjectionSeed 0x4644_435F_5631_5F50 "FDC_V1_P"; :137 `fdcNodeVector(code:)` : FNV64 → ONE SplitMix64 advance → LCG (Knuth 6364136223846793005 / 1442695040888963407) → 256 draws → l2Normalize (pipeline deliberately ≠ RI's); :242 `final class FDCProvider` : STATELESS, no training, no BasisCodec; path = FDC.ancestors + [code], node weight 1/(L+1); UNRESOLVED/empty → opt-out ([] / .zero) : honest, never guess; :284 embed/:299 embedFloat/:310 embedPair; determinism inherited from LatticeLib singletons +- Common float-lane tri-state (four trainable providers): `[]` = structural opt-out (untrained) → .unavailableProviderOptOut; throw VectorKitError.embedFloatVocabMiss = trained-but-all-OOV → .unavailableNoVocabHit; vector = signal. `embedPair` collapses the vocab-miss throw to (.zero, []) on ALL providers. Training thread-contract everywhere: Sendable class, but train/finalize must complete before concurrent embeds; read-only after + +### Ensemble factory : DefaultEnsemble.swift +- :38 `enum CorpusEnsemble`; :62 `defaultEnsemble()` : [.randomIndexing, .ppmi, .lsa, .nmf, .fdc] in pinned order ([0] = default signal); FUNCTION not constant (trainable providers are reference types : sharing one array would alias trained state across estates); returned trainables are UNTRAINED + +### Named neural providers : MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift +- MiniLM: :40 `struct MiniLMTextProvider: EmbeddingProvider`; :48 `inference: @Sendable ([Int32]) async throws -> [Float]` (closure-injected CoreML seam, doctrine §5); :50 init : modelID "minilm-v6", seed 0x4D49_4E4C_4D5F_7631 "MINLM_v1", 384-dim; :64 embed / :80 embedFloat / :95 embedPair (ONE inference pass : separate calls each pay inference) +- MPNet: :30 struct; :38 init : "mpnet-base-v2", seed 0x4D50_4E45_545F_7631 "MPNET_v1", 768-dim; :52/:65/:80 embed/embedFloat/embedPair +- EmbeddingGemma: :32 struct; :40 init : "embedding-gemma-300m", seed 0x454D_4247_4D5F_7631 "EMBGM_v1", 768-dim, stand-in tokenizer vocab 256_000 / maxTokens 2048 (SentencePiece-sized); :58/:71/:86 +- Parity framing: embedding VALUES are a property of the host's model bundle; what is bit-identical cross-port is the kit-owned pipeline (tokens→engram, float lane given a pooled vector) and the full no-host pipeline with DeterministicTokenizer + +### Apple NL providers : NLEmbeddingProvider.swift, NLContextualEmbeddingProvider.swift (Swift-only, ADR-019) +- NLEmbedding: :76 `nlEmbeddingProjectionSeed 0x4150_4E4C_454D_4231` "APNLEMB1" (outside doctrine table, same never-collide rule I-4); :113 `struct NLEmbeddingProvider: EmbeddingProvider, Sendable`; :145 init(language: .english default); :164/:180/:189 embed/embedFloat/embedPair; OS sentence model; no-model-for-language → [] opt-out; l2Normalize via FloatVecOps +- NLContextual: :79 `nlContextualEmbeddingProjectionSeed 0x4150_4E4C_4354_5831` "APNLCTX1"; :121 struct; :152 init; :170/:187/:196; checks `hasAvailableAssets` (free, sync) : NEVER triggers a network fetch; host must prefetch via requestAssets BEFORE constructing; mean-pools token vectors with per-token dimension guard; all failures → [] (asset absence = expected operational state) +- Both `#if canImport(NaturalLanguage)`; no Rust twins; vector dimension comes from the OS model (not pinned) + +## INVARIANTS / GOTCHAS + +- DETERMINISM DISCIPLINE: engines never read `Date()`/`UUID()`; `now` is always caller-supplied. Sanctioned exceptions: RemovedSourceStore audit stamp, telemetry timestamps. Training is a pure function of (texts, pinned seeds). +- UNIVERSAL JOIN KEY: chunk.id.uuidString == VectorKit item_id == inverted-index item_id. Swift canonical uuidString is UPPERCASE; HybridRecall re-canonicalizes vector-lane ids (P3-secfix). Universal tie-break everywhere: score DESC, then id ASC ("smaller id wins"). +- PINNED CONSTANTS (change = new conformance vectors + possible fleet re-key): chunker 800/100; BM25 k1 1.5 / b 0.75; QUANT_SCALE 100 with round-HALF-TO-EVEN; BMW block 128; RRF k 60, weights 0.6/0.4; candidate over-fetch max(limit×4, 32); float-lane sim quantization ×10_000; commit windows 512/4096; ingestMaxAttempts 8; idle drain sleep 15 ms; lease TTL 15 s / standby poll 3 s; RI/PPMI D 2048 K 10 window 4; LSA rank 64 sweeps 30; NMF rank 32 iterations 100 tolerance 0 factorization seed 0xDEADBEEFCAFEBABE; reduced-vocab cap 512; Chunk.namespaceBytes; basisFormatVersion 1. +- PROJECTION SEEDS partition vector storage by model and must be unique + frozen: deterministic 0xC05B_D15C_A15D_1B00, MINLM_v1, MPNET_v1, EMBGM_v1, RI_V1_MX, PPMI_V1M, LSA_V1_M, NMF_V1_M, FDC_V1_P, APNLEMB1, APNLCTX1. All projection goes through SubstrateML.FloatSimHash : ad-hoc projections are banned from the kit graph. +- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. trainOnCorpus is ADDITIVE : always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired : "HALF A"). +- TOMBSTONE CONSULTATION IS UNENFORCED: every chunk-replay path (reindex, first-ingest train, count, any future rebuild) MUST subtract RemovedSourceStore.removedIDs() or removed sources resurrect on the governor's auto-reindex. +- BundleStore.insert returns ONLY newly-inserted chunks; fold derived state over the returned subset. count(asOf:) ignores asOf. +- DUAL TypedValue DECODE (BundleStore, BasisStore): SQLite round-trips UUID/HLC/timestamp as primitives, InMemory keeps semantic forms; decoders must accept both; InMemory-only tests cannot catch the regression (proven bug class: silent total data loss on reopen). +- STREAM SCOPING: every queue op in CorpusIngestQueue uses stream "encode"; the shared queue.sqlite may carry other streams; an unscoped awaitDrain deadlocks. IngestJob JSON field names are a frozen cross-port wire contract. +- appendOnly means two different things: sync conflict policy (SyncManifest, `.appendOnly`, safe via content addressing) vs BundleStore schema flag (`appendOnly: false`, required so scrubText can UPDATE). Do not conflate. +- Chunks are immutable BY CONVENTION (no update API), not by DB trigger; edit = delete + reinsert with new id; metadata is the only legal per-chunk side-data slot (doctrine §2). +- keywordTokens override breaks the single-tokenizer guarantee shared by BM25 + distributional providers (convention, not compiler-enforced). +- MMR (`mmrLambda`) is declared but NOT implemented on any recall path. BM25Index is legacy : Corpus uses InvertedIndexStore. +- Actors everywhere (Corpus, BundleStore, InvertedIndexStore, BasisStore, CorpusProviderCountsStore, RemovedSourceStore, BM25Index): writes serialized per instance; embedding compute deliberately escapes the actor via Sendable providers for parallelism. +- Retry-in-place (8 attempts) is sound ONLY because ingest is idempotent via content-addressed ids : not a general recipe. +- Rust conformance gates: SharedVectors JSON (BM25 impacts, embedding vectors, basis blobs) read by BOTH legs; rust-providers tests pin basis serialization byte-for-byte for RI/PPMI/LSA/NMF and canonical vectors for RI/PPMI/FDC; NL providers exempt (Swift-only). diff --git a/packages/kits/CorpusKit/docs/DETAILS.md b/packages/kits/CorpusKit/docs/DETAILS.md new file mode 100644 index 0000000..696b0df --- /dev/null +++ b/packages/kits/CorpusKit/docs/DETAILS.md @@ -0,0 +1,988 @@ +--- +doc: DETAILS +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# CorpusKit Details + +This document walks through each source file in the package. Read +`OVERVIEW.md` first for the big picture. This document follows pipeline +order. It starts with the content types and the stores. Next comes the +keyword engine, plus fusion and recall. After that comes the +trainable-basis machinery. It ends with the `Corpus` facade, the ingest +queue, and the providers target. + +## Chunk.swift + +This file holds `Chunk`, the basic retrievable unit. It also +holds `ScoredChunk`, the result type recall returns. + +A chunk's id is content-addressed. Its UUID is computed, never +assigned. The function `deriveID(sourceID:startOffset:text:)` builds an +RFC 4122 version 5 UUID. It hashes the source identifier, the start +offset, and the exact text with SHA-1. The three fields join through the +Unicode unit separator. That separator stops different inputs from +colliding through ambiguous concatenation. + +Content addressing is what makes the rest of the kit work. Re-ingesting +the same source reproduces the same identities. A repeat insert becomes +a harmless no-op. Two federated devices that write the same +chunk converge on one row instead of conflicting. The fixed namespace +bytes are permanent. Changing them would re-key each chunk in each +estate. It would also break the join to existing vector rows, since +the chunk id doubles as the VectorKit item identifier. + +Two initializers exist. The content-addressed one computes the id. +Normal ingestion always takes this path. The explicit-id one +reconstructs a chunk whose id is already known, such as a decoded +storage row. The caller must ensure that id truly matches the content. +The `hlc` field is a hybrid logical clock stamp, a timestamp that also +encodes causal order. It is always caller-supplied. `ScoredChunk` pairs +a chunk with its fused `score`. It also carries the raw `vectorScore` +and `keywordScore`. The kit keeps these two scores separate. A caller +can then diagnose which lane produced a hit. + +## Chunker.swift + +This file holds `Chunker`. It splits raw source text into ordered +chunks with sentence-aware boundaries. + +A chunk should never cut a sentence in half. A half sentence embeds +poorly and reads worse. The chunker segments the text into sentences +first, for that reason. It delegates to `EideticLib.sentences(_:)`, so +segmentation logic lives in one shared place. It then fills a buffer +greedily. Sentences accumulate until adding the next one would pass the +target size. At that point the buffer flushes into a chunk. The tail of +the flushed chunk carries over as overlap into the next buffer. Overlap +means a match near a boundary still brings its surrounding context +along. Offsets into the original text are tracked exactly, since the +offset is part of each chunk's content-addressed id. + +`ChunkerConfiguration` holds three settings: `targetChars` (default 800), +`overlapChars` (default 100), and `respectSentences` (default true). Its +initializer clamps nonsense values. Overlap can never reach the target, +since that would loop forever. `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` +is the single entry point. The HLC generator passes in as `inout`. The +chunker is the sole authority for chunk order within one call, and it +stamps each chunk in emission order. Changing the default sizes changes +chunk boundaries. That in turn changes chunk identities for re-ingested +content, so the defaults are pinned to the substrate reference. + +## Tokenizer.swift + +This file holds the `Tokenizer` protocol. It also holds the single +canonical keyword tokenizer. + +A tokenizer serves two different masters. An embedding model needs its +own vocabulary of integer token ids. The protocol requires +`tokenize(_:) -> [Int32]` for that reason, plus id fields: `vocabID`, `maxTokens`, +and pad and unknown ids. Keyword search needs plain words instead. The +protocol also requires `keywordTokens(_:) -> [String]`. The protocol +extension supplies a default for that second method. It delegates to the +free function `defaultKeywordTokens(_:)`, which lowercases the text, then +keeps runs of alphabetic or ASCII-digit characters and splits on +everything else. + +There is just one definition of keyword tokenization in the module, +for a reason. BM25 and each distributional embedding signal must agree +on what a "term" is. Otherwise the keyword lane and the semantic lane +would score different vocabularies. Hybrid recall would then quietly +degrade. The function is also parity-critical. The Rust port implements +the same rules, and committed conformance vectors on both legs break if +the rules change. A provider that overrides `keywordTokens` breaks this +guarantee by convention, not by compiler error. + +## CorpusKitError.swift + +This file holds `CorpusKitError`, the module's single error enum. + +Seven flat cases cover the failure classes: `encodingFailure`, +`decodingFailure`, `tokenizerUnavailable`, `modelUnavailable`, +`embeddingFailed`, `storeUnavailable`, and `notTrainable`. Each case +carries a plain message string. Callers mostly log the message, or +surface it, rather than branch on structured data. The `notTrainable` +case exists for `EmbeddingModel.reconstruct(from:)`. Providers without a +trained basis surface this error instead of silently substituting a +wrong provider. Three provider families fall into this group: the +deterministic provider, the named neural models, and the stateless FDC +provider. The enum is `Equatable` on its message strings. Tests that +compare errors must so construct the exact message. + +## BundleStore.swift + +This file holds `BundleStore`, the persistence layer for chunks. It +is the content half of each content-plus-vector bundle. The vector half +lives in VectorKit, joined by the chunk's UUID string. + +`BundleStore` is an actor. It wraps `any Storage` from PersistenceKit. +The application picks the backend, SQLite or in-memory. The kit itself +does not. It owns schema version 3: a `chunks` table of ten columns, plus a +`corpus_metadata` table, with indices on `source_id` and `hlc`. Inserts +do not go straight to the row store. They pass through a hashing +decorator first. That decorator computes a content hash with +`MerkleHash.leaf` on each write and emits dirty-chain events. A Merkle +hash chain lets the estate prove its content has not drifted. Each chunk +hashes. Each source's chunks combine into a per-corpus root. All corpus +roots combine into one global root. Since the hashing callback is +synchronous, a small lock-guarded `ParentChainCache` pre-stages each +chunk's parent identifiers before the insert. The corpus and root +identifiers derive from fixed namespace strings, through SHA-256, so +both language legs compute the same chains. + +`insert(_:)` is idempotent by design. It attempts a plain insert per +chunk. It treats a duplicate-key error as the documented no-op path. +First write wins, and this is sound only since identities are +content-addressed. The method returns only the chunks that were actually +new, in input order. A caller that maintains derived state never +double-counts a re-ingested chunk this way. Seven methods form the read +surface: `get`, `getMany`, `chunksForSource`, `allChunks`, +`allSourceIDs`, `count`, and `chunkSourcePairs`. All are thin query +wrappers. `chunkSourcePairs()` omits chunk bodies on purpose, so opening +a corpus stays cheap. +`scrubText(sourceID:)` is the hard-delete seam. It zeroes the `text` +column through a direct update. This is why the schema declares the +table `appendOnly: false`, even though the API treats chunks as +immutable. Immutability here is a convention the surface enforces, not a +database trigger. The sync layer has its own separate `appendOnly` +conflict policy, and the two should not be confused. + +One decoding rule deserves emphasis. The SQLite backend round-trips +UUIDs as text and HLCs as packed integers. The in-memory backend +preserves the semantic typed values instead. `decodeChunk` and its +helpers accept both forms. A past semantic-only decoder silently +dropped each persisted chunk on reopen, and in-memory tests never +caught it. Any new decode path must handle both forms. + +## RemovedSourceStore.swift + +This file holds `RemovedSourceStore`, the tombstone table that makes +source removal stick. + +Chunk rows are never deleted. So "removing" a source can only delete its +vectors and its keyword postings. Without a durable marker, any rebuild +that replays `allChunks()` would re-embed the removed source. That +includes an explicit reindex, or the autonomic governor's scheduled one. +The source would then silently resurrect. The store is, for that +reason, the single source of truth each rebuild path must consult. The presence of +a row is the entire state. There is no boolean column, per the +fleet-wide schema rule. Reactivation is symmetric: re-ingesting a source +clears its tombstone, so ingestion itself is the undo. + +`markRemoved(_:now:)` upserts a tombstone with a caller-supplied +timestamp. `clearRemoved(_:)` deletes it. `removedIDs()` returns the full +set that rebuilds must subtract. `deleteAll()` supports index +destruction. Nothing enforces that a new rebuild path remembers to +consult this store. That remains a blast-radius obligation on each +future change. + +## CorpusProviderCountsStore.swift + +This file holds `CorpusProviderCountsStore`. It persists the raw +statistics a trainable embedding signal builds up between retrains. + +The design splits cheap state from heavy state. The `counts` column is +an opaque blob the provider alone serializes. The store never decodes +it, and never imports the providers target. Two small integer columns, +document count and vocabulary size, are lifted out of the blob. A +staleness check can then ask "has the corpus grown enough to retrain?" +with one tiny query, instead of deserializing a large blob. Rows key by +`(model_id, model_version)`. This matches how basis rows and vector rows +are keyed, since counts are only valid for the exact provider version +that accumulated them. `upsert(_:)` replaces the row whole. Additive +merging is the provider's job before it calls in. + +The file is candid that this is half a feature. Counts are persisted and +restored. But `Corpus.reindex` still retrains from raw chunk text rather +than from this table. The counts-backed retrain path, and the matching +vector re-projection, are future work. Documentation should not present +this store as the current retrain mechanism. + +## SyncManifest.swift + +This file holds `CorpusKitSync.manifest(zoneIdentifier:)`, the +declarative sync contract for the `chunks` table. + +The manifest declares one bidirectional synced table. Its primary key is +`id`, and its conflict policy is `appendOnly`. That policy is safe, +since chunks are content-addressed and immutable. Two devices can +never produce conflicting edits to the same id. They can only produce +the same re-derivations, so the sync layer needs no merge strategy. The kit performs no sync itself. The application hands this +manifest, and a storage instance, to ConvergenceKit. When VectorKit sync +is also enabled, both tables should share one zone. That keeps chunks +and their vectors join-compatible on each device. + +## Engine/SparseTypes.swift + +This file holds the value types of the sparse retrieval lane: +`ImpactPosting`, `SparseHit`, `FusedHit`, and the `LaneTag` alias. + +The load-bearing decision is that `ImpactPosting.impact` is an integer. +A float BM25 weight quantizes once, at index build time. From then on +the whole query path runs on integer arithmetic. That is what makes the +Swift and Rust legs bit-identical. `SparseHit` is the consumer surface. +It divides the integer score back by the quantization scale. +`FusedHit` carries the fused score, plus a `perLane` map of raw +per-lane scores. Those raw scores stay available so later selection +stages can read lane signals without recomputing them. `LaneTag` is a +type alias to VectorKit's enum, not a second enum. Two matching Swift +enums would still be distinct types, and that would make case names +ambiguous for a consumer that imports both kits. Posting lists always +sort by item id ascending, the WAND algorithm's pivoting invariant. +Fused results sort by score descending, then by item id ascending. + +## Engine/BM25Weighting.swift + +This file holds BM25 as an impact-weighting scheme that feeds the +inverted index, plus the quantizer and query helpers. + +BM25 scores a document for a term by combining two things: the term's +rarity, called inverse document frequency (IDF), and its frequency in +the document. Document length damps the score. The whole float +computation happens just once, at index build. `build(termFreqs:docLengths:parameters:)` +evaluates the classic formula per term and document. It then quantizes +each contribution with `quantizeImpact(_:)`, which multiplies by 100 and +rounds half to even. The rounding mode is pinned on purpose. Swift's +default rounding differs from banker's rounding at exact halves, and +both legs must agree. Term strings map to dense integer ids in sorted +order, so runs stay reproducible. `BM25Parameters` pins the defaults: +`k1` equals 1.5, and `b` equals 0.75. Both values are tunable per +estate. `queryPairs(queryTerms:termMapping:)` turns query terms into +term-id and weight-100 pairs. It drops unknown terms. It deduplicates +repeats, so each term contributes just once. + +## Engine/InvertedIndex.swift + +This file holds the generic weighted inverted index. It ships two +exact top-k algorithms: WAND and Block-Max WAND. + +An inverted index maps each term to a posting list, the items containing +that term, each carrying a pre-quantized impact. Scoring an item for a +query is an integer dot product over shared terms. The naive approach +scores each candidate. WAND, short for "Weak AND," skips most of them +instead. It keeps one cursor per query term, sorted by current position. +It computes a pivot: the first point where the accumulated best-case +impacts could beat the current k-th best score. Items before the pivot +cannot win, so the algorithm skips them wholesale. Block-Max WAND +refines this idea with per-block maxima, using a block size of 128. If +even the tighter block bound cannot beat the threshold, the whole block +gets skipped. Both algorithms are exact. They return precisely the same +top-k as a full scan. `exhaustiveScan(query:k:)` ships as the reference +oracle for conformance tests. + +The index is immutable after construction. Mutation means rebuilding it, +and serializing rebuilds is the wrapper's job. The internal bounded heap +implements the universal tie-break: equal scores resolve toward the +smaller item id, so results never depend on hash order or insertion +order. Item ids compare as strings. Two pinned constants belong to the +cross-port contract: `invertedIndexQuantScale` (100) and +`invertedIndexBlockSize` (128). + +## Engine/InvertedIndexStore.swift + +This file holds `InvertedIndexStore`, the persistent wrapper that +lets keyword state survive restarts without replaying chunk bodies. + +The store persists only raw statistics: a term-frequency table and a +document-length table, in two small SQLite tables. The weighted index +itself is derived, not stored. On demand, `buildIndex(parameters:)` runs +the BM25 build over the in-memory mirrors and caches the result. Each +write invalidates the cache. Persisting statistics instead of weighted +postings means changing `k1` or `b` never requires a data migration. It +only requires an in-memory rebuild. `open()` loads all rows once, at a +cost proportional to terms plus documents, never to chunk text. + +`index(itemID:tokens:now:)` replaces a document's terms atomically, and +it is idempotent. Empty tokens remove the item. `remove(itemID:)` and +`deleteAll()` complete the mutation surface. `topK(queryTerms:k:parameters:algorithm:)` +is the one-call query path. The actor serializes all mutation. The Rust +twin owns a private database connection with explicit batch methods. +The Swift store instead shares the estate's storage, which is why the +facade manages transaction windows around it during bulk ingest. + +## BM25Index.swift + +This file holds `BM25Index`, the original in-memory keyword index. It +is preserved as a public primitive. + +The `Corpus` facade no longer uses it. Durability required +`InvertedIndexStore` instead. External callers that built on the older +type keep a working, chunk-typed surface. It holds term frequencies +keyed by chunk UUID string. It tokenizes chunk text itself, through an +injected `Tokenizer`. It delegates scoring to the same engine layer: BM25 +weighting, plus Block-Max WAND. The built index caches between writes. +`index(_:)`, `remove(_:)`, `documentCount()`, and `topK(_:for:)` form the +surface. `topK` takes pre-tokenized terms, and the caller must tokenize +with the same vocabulary used at index time. Ties break by UUID string +order. That order differs from numeric UUID order. It stays the same +on both legs, still. + +## Engine/Fusion.swift + +This file holds `Fusion`, the generalized weighted Reciprocal Rank +Fusion engine. + +Reciprocal Rank Fusion, called RRF, merges ranked lists. It never +compares raw scores, since those scores live on incompatible scales. Each +lane contributes `weight × 1 / (rrfK + rank)` for each item it ranked. +The sums decide the final order. The constant `rrfK` defaults to 60, +from the original RRF paper, and it damps the advantage of rank one over +rank two. The function deduplicates within each lane. Only an item's +best rank counts there, since a duplicate would illegally double its +contribution. The function also demands `rrfK > 0`, since zero or +negative values would corrupt the formula. Output sorts by fused score +descending, then by item id ascending. + +Two overloads exist. `fuse(rankedLists:laneScores:weights:rrfK:)` takes +explicit ranks. It takes optional raw scores too, to carry through into +`perLane`. `fuse(scoredLists:weights:rrfK:)` treats array position as +rank instead, so the caller must pre-sort its input. The engine is a +pure function over ranks and weights. It is deterministic and reentrant. +One caution applies here. The configuration type in +`HybridRecall.swift` reserves an MMR field. No diversification logic +exists yet, on this path or anywhere else. + +## HybridRecall.swift + +This file holds `HybridRecall.recall(...)`, the canonical two-lane +retrieval pipeline. + +The pipeline over-fetches a candidate window from each lane. That +window's size is `max(limit × 4, 32)`, since fusion needs headroom. An +item ranked eleventh in both lanes can out-fuse an item ranked first in +only one. The vector lane runs VectorKit's nearest-neighbor search +concurrently, while the query gets tokenized and the keyword lane +queries the inverted index. Both hit lists become ranked lists, keyed by +canonical UUID strings. Canonicalization is a deliberate security-review +fix. A lowercase UUID written by the Rust leg, and an uppercase Swift +keyword hit for the same item, would otherwise never fuse. `Fusion.fuse` +merges the lanes with the configured weights. The result truncates to +the limit. The winners then hydrate from the bundle store, in fused +order. + +`HybridRecallConfiguration` pins four defaults. The vector weight is +0.6. The keyword weight is 0.4. The `rrfK` value is 60. The +`mmrLambda` slot exists, but the pipeline never reads it. Score mapping +is asymmetric on purpose. A vector score of zero is a perfect Hamming +match, and the pipeline keeps it. A keyword score of zero instead means +"did not match," and it maps to nil. Telemetry fires at the operation +boundary, where it cannot affect results. It covers latency and +per-lane counts. With monitoring off it costs one atomic load per +metric. + +## TrainableEmbeddingBasis.swift + +This file holds the `TrainableEmbeddingBasis` protocol. It is the +seam that lets the core drive provider training without importing the +providers target. + +Layering runs one way: providers depend on core, never the reverse. The +`Corpus` holds providers as type-erased values. It needs a protocol to +ask one question: can the provider train, serialize, and reconstruct +itself? Some providers +cannot: the deterministic provider, the named neural models, FDC, and +the Apple NL providers. They simply do not conform, and the facade +surfaces `CorpusKitError.notTrainable`. `reconstructBasis(from:)` is an +instance method rather than an initializer, for exactly this reason. +Invoked on a type-erased witness, it routes to the correct concrete +type's deserializing initializer. + +The protocol has two halves. The basis half covers full training and the +round-trip law: a reconstructed provider embeds byte-identically to the +trained original. It includes `trainOnCorpus(texts:)`, `serializeBasis()`, +and `reconstructBasis(from:)`. The counts half maintains raw additive +statistics incrementally instead. It includes `addToCounts(text:)`, +`serializeCounts()`, `restoreCounts(from:)`, and `countsVocabularySize`. +Snapshots happen at batch boundaries, since per-chunk serialization +would be quadratic over an import. Training must never read the wall +clock. It is a pure function of the texts and fixed seeds. The Rust port +cannot cross-cast trait objects, so there the embedding trait is a +supertrait instead, a documented and sanctioned divergence. + +## BasisStore.swift + +This file holds `BasisStore`, persistence for trained basis blobs, so +a reopened corpus embeds right away instead of retraining. + +One row per `(model_id, model_version)` lives in the +`corpus_provider_basis` table. Each row holds the opaque little-endian +blob, a trained-at timestamp, and a trained-chunk-count anchor reserved +for a future auto-retrain policy. The timestamp is caller-supplied, and +it stores as ISO 8601 text, per the schema rules. The composite key +matters: a blob trained for one provider must never load into another, +and the key matches how each vector row is keyed. Retrain upserts in +place, so just one row exists per provider. There is no history, and +there are no orphans. Schema version 2 adds a nullable JSON `ext` column +as a forward-compatibility slot. Version 1.0 writes that column as null +and never reads it. + +Like `BundleStore`, the decoder tolerates both typed-value forms: the +in-memory backend's semantic timestamps, and SQLite's ISO text. A +semantic-only reader would otherwise silently drop each row on reopen, +and semantic recall would go dark on any restored estate. +`upsert(_:)`, `load(modelID:modelVersion:)`, and `deleteAll()` form the +whole surface. + +## CorpusKit.swift + +This file holds the public entry point: the `Corpus` actor, the +`EmbeddingModel` selection enum, the `FloatLaneOutcome` result type, and +the `EncodeSpeed` quality-of-service knob. It is the largest file in the +package, since it is the composition root. Everything else exists so +this file can wire it together. + +### The Corpus Actor and Its Provider Slots + +A `Corpus` composes the bundle store, the persistent keyword index, and +the vector store. It also composes the basis store, the counts store, +and the tombstone store. It holds one slot per configured embedding +signal, too. It seals VectorKit behind its own surface. No VectorKit type appears in a +public signature, except the deliberate `sharedVectorStore` escape +hatch. That hatch lends the estate's one vector store to the +orchestrator, so no second store gets built over the same table. + +Each provider slot holds three things. First, the serving provider, +which embeds queries and chunks. Second, trainable signals get a +`freshBasisBlob`. This is the serialized untrained basis, captured at +construction. It works as the from-scratch factory. Training is additive, +so retraining a live provider would count the corpus twice. Each +retrain instead reconstructs a fresh provider from this blob. That makes +reindex idempotent and canonical across ports. Third, a separate counts +accumulator, kept apart from the serving provider on purpose. Growing +a vocabulary in place would desync a factorized basis from its frozen +factors. Slot zero is the default signal. Each single-signal entry +point delegates to it, so a one-model corpus behaves just like the old +single-provider design. + +### Opening, Ingesting, Reindexing + +`init(storage:models:)` migrates six schemas. It resolves each slot: a +persisted basis reconstructs a trained provider, and a corrupt blob +throws rather than serving untrained data. It opens the keyword index. +It warm-loads a chunk-to-source map from a body-free projection, so the +whole cold start avoids reading chunk text. `ingest(_:sourceID:now:)` +performs six steps in order. It chunks the text. It clears any +tombstone, since re-ingest reactivates a source. It inserts the chunks +idempotently. It indexes keywords. It folds counts. It embeds the +chunk. Embedding runs in two phases. Any trainable slot with no persisted basis triggers the +one-and-only implicit first-ingest training, over the full corpus +snapshot. Each other slot folds in under its frozen basis instead. +Fold-in embeddings compute concurrently off the actor, since providers +are `Sendable` values, and they land in one batched vector write. +`ingestBatch(_:)` produces output identical to per-item ingest. It +commits in windows of 512 items or 4,096 rows, long enough to amortize +disk syncs and short enough not to starve concurrent captures. It also +fans embedding work out in contiguous slices per core. + +`reindex(now:)` is the explicit retrain trigger. It performs five steps +in order. It reconstructs fresh providers. It trains on all active +chunks, excluding tombstoned sources. It installs the result. It +persists the basis. It re-embeds each active chunk under each slot. +Only two train triggers exist in the whole kit: first ingest, and +explicit reindex. + +### Recall, Removal, Observation + +`recall(_:limit:now:)` embeds the query on the default signal and +delegates to `HybridRecall`. `bm25TopKBySource(query:limit:)` is the +pure keyword lane, aggregated to source granularity. The dense float +lane covers three methods: `floatNearest`, `floatNearestPerSignal`, and +`floatFarthestPerSignal`. All three rank by true cosine similarity. None +of them ever throws. Unavailable states are typed `FloatLaneOutcome` values instead: +`.unavailableProviderOptOut`, `.unavailableNoVocabHit`, +`.unavailableNoFloatRows`, `.emptyQuery`, and `.storeError`. A dark lane +is an expected condition here, not an error. The farthest variant +answers "what is unlike this?" It aggregates by each source's closest +chunk, so a source only counts as unlike when even its best chunk is +far. + +`remove(sourceID:)` suppresses recall. It deletes keyword rows and each +model's vectors, and it writes the tombstone. `expunge(sourceID:)` goes +further: it scrubs the verbatim text first, then removes the source, so +content is destroyed even if a later step fails. `destroyRecallIndex()` +wipes each derived structure while chunk rows survive. Four more +methods round out the observational surface: `count()`, +`indexedSourceIDs()`, `maintainedVocabAnchor()`, and two Merkle-root +readers. + +### EmbeddingModel and the Small Types + +`EmbeddingModel` names each signal the corpus can hold. `.deterministic` +is the permanent federation-grade baseline. It is hash-based and +lexical. It is fully reproducible, with a pinned seed. Three named +neural models follow it: `.miniLM`, `.mpNet`, and `.embeddingGemma`. +Each one takes a host-supplied inference closure. Four trainable statistical +signals carry pre-built providers: `.randomIndexing`, `.ppmi`, `.lsa`, +and `.nmf`. The stateless `.fdc` signal comes next. Two Apple-only +signals close the list: `.nlEmbedding` and `.nlContextualEmbedding`. +`isTrainable` reports whether the carried provider conforms to the +training seam. `reconstruct(from:)` routes a persisted blob to the right +concrete type. `EncodeSpeed` selects the embed-concurrency cap. +`.foreground` uses all cores, and `.background` uses roughly a quarter. +A private `CorpusDefaultTokenizer` duplicates the providers' +deterministic tokenizer, to avoid a circular dependency. A private +`CorpusTextProvider` implements the tokenize-infer-project pipeline for +the named models. It computes the pooled vector once per chunk, for +both the engram and the float row. + +## CorpusIngestQueue.swift + +This file holds the asynchronous ingest pipeline. It arrives as an +extension on `Corpus`. Three pieces make up the pipeline: a durable +queue, a background drain worker, and a single-drainer lease. It exists +so CorpusKit is a complete standalone substrate. Any consumer gets +queued, multi-core encoding with no orchestrator. + +`mountIngestQueue()` picks the backend by estate durability. A SQLite +estate gets a sibling `queue.sqlite` file, derived deterministically +from the estate configuration. That file is encrypted with the same key +as the estate. It replaces an earlier plaintext directory queue, which +was a real security hole beside an encrypted estate. An in-memory estate +gets a transient store instead, under a fixed constant UUID, which +avoids random-id nondeterminism. Since the physical queue can +carry other streams, each operation here is scoped to the `"encode"` +stream. An unscoped wait would deadlock on jobs this drainer never +claims. + +The drain loop coordinates through a `DrainLease`. Only one live drainer +exists per estate, with crash recovery on first acquisition that resets +orphaned in-flight jobs. That recovery is safe precisely since the +lease guarantees no other live drainer holds them. A losing process +becomes a warm standby instead. It re-checks each three seconds, +bounded by the lease's fifteen-second staleness window. Each drain pass +claims the whole available batch. It decodes jobs: undecodable ones are +terminally blocked, and empty ones complete right away. It runs +`ingestBatch` once for the whole batch, and it retires the batch in one +bulk reply. While passes keep draining jobs, the loop spins without +sleeping. It defers the vector-index publish until the burst ends, so +the kit runs one index rebuild per burst instead of one per pass. That +turns a quadratic bulk import linear. When idle, the loop sleeps fifteen +milliseconds, the near-realtime latency floor. A failing item retries in +place, up to eight attempts, before a terminal blocked reply. In-place +retry is sound only since ingest is idempotent. + +`enqueueIngest(_:sourceID:now:)` and `enqueueIngestBatch(_:)` stamp jobs +with caller-supplied instants, never the wall clock. The batch variant +wraps all inserts in one transaction, which removed the last full-core +bottleneck of bulk imports on encrypted SQLite. +`awaitIngestDrain(timeout:)` is the barrier importers use to know writes +are searchable. `setOnEncoded(_:)` installs the one callback CorpusKit +ever makes toward an orchestrator. The `IngestJob` wire format's JSON +field names form a pinned cross-port contract with the Rust twin. + +## BasisCodec.swift + +This file holds the shared binary codec each trainable provider uses +to serialize bases and counts. + +The byte layout is the cross-port contract. The same trained state must +serialize to the same bytes on Swift and on Rust, which rules out JSON. +Float formatting, key order, and whitespace all differ across +ecosystems. The rules stay fixed: everything is little-endian. Floats +write as raw IEEE-754 bit patterns, so negative zero and NaN round-trip +exactly. Strings are length-prefixed UTF-8. Maps write with keys in +ascending raw-byte order. That byte-order sort exists since Swift's +default string comparison is Unicode-canonical, while Rust's is byte +order. The writer compares raw UTF-8 to match Rust. Each blob is framed +with a four-byte magic tag and a format version byte, currently 1. + +`BasisWriter` is an append-only cursor with typed write methods. +`BasisReader` is a bounds-checked sequential reader. Its +`expectMagic(_:)` and `expectVersion(_:)` methods reject wrong-provider +or future-format blobs with `CorpusKitError.decodingFailure`, never a +crash and never a silent misparse. Both types are value types with no +shared state. + +## DeterministicTokenizer.swift + +This file holds `DeterministicTokenizer`, the model-agnostic stand-in +tokenizer that ships as the version 1.0 default for the named neural +providers. + +It is a hash, not a vocabulary. Words split by the canonical keyword +rules first. Each word then folds through FNV-1a into an id in the range +two through the vocabulary size. Ids zero and one are reserved +sentinels, for padding and unknown tokens. Empty input returns a single +pad token, never an empty array. Since both legs fold through the same +hash, conformance harnesses get the same ids for the same input. The +defaults match the BERT family. The vocabulary size is 30522. The token +maximum is 128. One caveat matters most: feeding these ids into a real +embedding model produces garbage, since they carry no relation to the +model's true vocabulary. Real WordPiece and SentencePiece tokenizers +arrive with the version 1.1 model-bundle mission. + +## TermDocumentCounts.swift + +This file holds `TermDocumentCounts`, the shared count builder that +feeds the LSA and NMF providers. + +It owns three things: the vocabulary, per-document term frequencies, and +per-term document frequencies. The vocabulary builds in encounter order. +A term's column index is fixed by the first document that mentions it, +which keeps matrix columns stable for a fixed document sequence. The +builder does not own weighting or factoring, on purpose. Those +belong to the consuming providers, which weight the same counts +differently. `addDocument(_:)` is the full training path. +`addDocumentForCountsAnchor(_:)` is the lightweight incremental path +instead. It grows the vocabulary and the document count, but it keeps no +frequencies, since the heavy inputs get re-derived by re-tokenizing the +corpus at retrain time. That bounds maintained state to the vocabulary's +size. The restored-vocabulary initializer likewise seeds a deserialized +provider with truthful metadata and empty frequency rows. The builder is +not thread-safe. All writes must finish before any reads. + +## ReducedVocab.swift + +This file holds the shared vocabulary-reduction step for the dense +factorizations, LSA and NMF. + +A dense matrix over tens of thousands of terms is unfactorizable on a +device. The code comment estimates ten-to-the-fifteenth operations for +that case. The fix is to keep only the most informative columns. +Below the cap, 512 by default, the function is a strict no-op. Small +estates and each conformance fixture behave just as before reduction +existed. Above the cap, the function drops terms seen in only one +document, since those are pure noise. It ranks the rest by document +frequency descending, since terms that co-occur across many documents +carry the latent structure a factorization can find. It breaks ties by +raw UTF-8 byte order, to match Rust's string ordering, so both legs +select the same vocabularies. The selection is shared rather than +per-provider, since informativeness is a corpus property. It stays the +same for both factorizations. `ReducedVocabulary` freezes the kept +terms, the projection map, and the row-remapping table. + +## RandomIndexingProvider.swift + +This file holds `RandomIndexingProvider`, the first honest +distributional signal in the dense lane. + +Random Indexing gives each term a deterministic sparse "index vector": +2,048 dimensions, with exactly ten nonzero entries of plus or minus one. +The generator seeds a counter-based random stream from the FNV hash of +the lowercased term. It draws exactly twenty values: ten positions, then +ten signs. Collisions resolve last-wins rather than by rejection, so the +draw count stays constant and the Swift and Rust streams stay aligned. A +term's meaning is then learned by addition. A window of four slides over +the training text, and each term builds up the index vectors of its +neighbors into a context vector. Terms that keep similar company +converge. This is genuine co-occurrence semantics at almost no +computational cost, and the accumulation is incremental by construction. + +Embedding text sums the context vectors of its in-vocabulary terms and +normalizes to unit length. The float lane is honest about failure. An +untrained provider opts out with an empty vector. A trained provider +whose query is entirely out of vocabulary throws a typed vocabulary-miss +error instead, and the facade maps that error to the right dark-lane +outcome. The basis blob carries the magic tag `RIB1`. It is the whole +vocabulary map, since Random Indexing has no separate finalize step. +The counts blob carries the same payload, under the distinct magic tag +`RICT`. A counts row can so never be misread as a basis row. The +projection seed spells +`RI_V1_MX`. + +## PpmiProvider.swift + +This file holds `PpmiProvider`, the co-occurrence signal weighted by +positive pointwise mutual information. + +PPMI asks of each word pair whether the words co-occur more than chance +would predict. The score is the logarithm of the observed co-occurrence +probability over the product of the individual probabilities, floored at +zero. Frequent-but-meaningless neighbors, the "the" problem, score near +zero. Genuinely associated pairs keep full weight instead. Training runs +in two phases. Phase one counts: a sliding window of four builds up +pair counts, term counts, and totals, additively across calls. Phase +two, `finalize()`, converts counts into context vectors. Each term's +vector becomes the PPMI-weighted sum of its neighbors' Random Indexing +index vectors. That sum lives in the same 2048-dimension space, with +the same generator. The two signals stay directly comparable, as a +result. The file warns outright that this differs mathematically from +unweighted Random +Indexing, and it must never be "simplified" into it. + +Embedding, opt-out, and vocabulary-miss behavior all mirror the Random +Indexing provider. The basis blob, `PPB1`, persists only the derived +vectors. The count tables are training scratch instead. The counts +blob, `PPMC`, persists the raw additive state. That state includes the +nested pair-count map. The map serializes with byte-sorted keys. A +retrain can then resume counting, without re-tokenizing. Stored vectors stay unnormalized. +Normalization happens at embed time instead, so tests can inspect raw +sums. The projection seed spells `PPMI_V1M`. It must never equal the +Random Indexing seed, since the seed partitions vector storage by +model. + +## LsaProvider.swift + +This file holds `LsaProvider`, the latent semantic analysis signal. + +LSA finds hidden topic structure by factorizing a term-document matrix, +with the singular value decomposition, or SVD. Words that appear in +similar documents land near each other in the latent space, so synonyms +co-locate even when they never co-occur. Training builds on the shared +count builder. It reduces the vocabulary first, the ADR-022 step. It +weights each cell by log term frequency times smoothed IDF, with +add-one smoothing on both sides, so an unseen query term still gets a +positive weight. It decomposes with `JacobiSVD` from SubstrateML, at +rank 64. The Jacobi method is chosen since its sweep count is pinned +at exactly thirty sweeps. That makes the factorization a fixed +computation, rather than a convergence-dependent one. Changing the sweep +count would invalidate each conformance vector. Wide matrices transpose +for the decomposition, and the factors swap back afterward, so +downstream code always sees one orientation. + +New text embeds through the classical fold-in formula. The query's +weighted term vector projects through the factor matrix. It scales by +the inverse singular values, and it skips values below a small floor. +Training documents use the exact projection instead. The dark-lane +contract matches the other trainable providers. The basis blob carries +the magic tag `LSB1`. It persists three things: the reduced vocabulary, +the IDF weights, and the raw factor matrices. This form is port-neutral, +so each leg re-derives its own document vectors. The counts blob +carries the magic tag `LSAC`. It persists only the vocabulary and +document-count anchors, per the re-tokenize-at-retrain decision. The +projection seed spells `LSA_V1_M`. + +## NmfProvider.swift + +This file holds `NmfProvider`, the non-negative matrix factorization +signal. + +NMF factorizes the term-document matrix into two non-negative factors, +so each latent dimension reads as an additive combination of terms: +parts, not contrasts. The matrix orients terms-by-documents, weighted by +log term frequency only. NMF requires non-negative input, and its +update steps are most stable with uniformly scaled entries, so IDF is +left out on purpose. Factorization reuses SubstrateML's alternating +least squares, at rank 32, with two pinned determinism devices. The +convergence tolerance sets to zero, which forces exactly one hundred +iterations on each platform, regardless of floating-point convergence +behavior. Factor initialization seeds with a fixed constant. Document +embeddings are the normalized factor columns, precomputed at finalize. + +Queries fold in through a pseudo-inverse projection onto each factor +column, with a small epsilon guarding the denominator, then normalize. +Serialization follows the family pattern. The basis blob, `NMB1`, +carries configuration, the reduced vocabulary, and both raw factor +matrices. The counts blob, `NMFC`, carries anchors only. The projection +seed spells `NMF_V1_M`. The dark-lane contract matches the other +trainable providers. + +## FdcProvider.swift + +This file holds `FDCProvider`, the taxonomic co-classification +signal. It is the one honest signal that needs no training at all. + +The provider classifies text with LatticeLib's FDC engine. That engine +lives in the moot-semantics repository, and it is not reimplemented +here. `FDC.encode` returns a lattice code, or nothing. `FDC.ancestors` +returns the code's chain up the classification hierarchy. The provider +turns that chain into geometry. Each node in the chain gets a +deterministic 256-dimension unit vector, generated from the hash of its +code string. The vectors sum with weight one over depth-plus-one. Shared +roots give any two texts a similarity floor, and shared deep ancestors +add to it. Texts filed near each other in the subject hierarchy then +embed near each other, regardless of surface wording. The node-vector +generator runs a different pipeline from Random Indexing's, on purpose. +The two input domains cannot collide. + +Honesty governs the edges. Text the classifier cannot resolve returns +UNRESOLVED. The provider then opts out, with an empty float vector and a +zero engram, since unclassifiable text must not contribute false +similarity. The provider is stateless and `Sendable`. Its determinism +rests on LatticeLib's own agreement property. The projection seed spells +`FDC_V1_P`. The free function `fdcNodeVector(code:)` is public, so +conformance tests can pin individual node vectors. + +## DefaultEnsemble.swift + +This file holds `CorpusEnsemble.defaultEnsemble()`, the single source +of truth for the production recall ensemble. + +The factory returns the five honest signals in fixed order: Random +Indexing, PPMI, LSA, NMF, and FDC. Order is load-bearing here, since +the first element becomes the corpus's default signal. It is a function +rather than a shared constant, since the four trainable providers are +reference types holding mutable trained state. A shared array would +alias one provider instance across each estate. Fresh construction +gives each estate its own untrained providers instead. The corpus +lifecycle then trains and persists those providers under their own +model identifiers. The factory lives in the providers target, since +it names concrete types, and the core's `EmbeddingModel` enum never +does. + +## MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift + +These three files hold the named neural embedding providers. +MiniLM-L6 v2 uses 384 dimensions. mpnet-base-v2 uses 768. EmbeddingGemma +300M also uses 768. They share one structure, so this section describes +them together. Each fact below holds for all three, unless the text +says otherwise. + +Each provider runs the same pipeline. It tokenizes the input. It runs +the host-injected inference closure to get a pooled float vector. It +projects that vector to a 256-bit engram with `FloatSimHash` from +SubstrateML. An engram is a fixed-size binary fingerprint. Similar +vectors produce similar engrams, so the vector lane can compare chunks +by fast Hamming distance. The inference closure is the doctrine's model +seam. The provider never holds a CoreML model, which keeps it testable +without a model bundle. Model loading differs on iOS, macOS, and CI. +The host owns that step. It composes the model once, at startup. Each +provider pins its own projection seed. The seed keeps fingerprints +model-tagged, in ASCII: `MINLM_v1`, `MPNET_v1`, and `EMBGM_v1`. Seeds +partition vector storage by model, and they must never change or +collide. + +Each provider exposes three surfaces. `embed` returns the engram. +`embedFloat` returns the raw pooled vector, for the true-cosine float +lane. `embedPair` runs inference once and derives both results, which +halves the cost of ingest, since ingest needs both per chunk. Empty +input short-circuits to a zero engram and an empty vector. All three +default to the `DeterministicTokenizer` stand-in. EmbeddingGemma's copy +is sized for its own SentencePiece vocabulary. That vocabulary holds +256000 terms, with a context of 2048 tokens. Until real tokenizers ship, embedding values stay a +property of the host's model bundle. What the kit itself owns, the +token-to-engram pipeline given a pooled vector, stays bit-identical +across ports. + +## NLEmbeddingProvider.swift + +This file holds `NLEmbeddingProvider`, the Apple sentence-embedding +signal built on the operating system's bundled `NLEmbedding` model. + +It is the cheap, immediate Apple-native lane. It needs no download. It +needs no CoreML seam. It needs no training either, since the OS +framework is the model itself. There is nothing for a host to inject. + +The provider looks up the OS sentence model for its configured +language. It embeds the text. It casts the result to floats. It then +normalizes the vector to unit length, using the substrate's canonical +vector operations. This keeps the float lane's cosine on a unit sphere, +like each other provider. + +Absence stays graceful. Two cases return an empty vector: no model +exists for the language, or the model cannot embed the given text. +Either case counts as a typed lane opt-out, never an error. + +The projection seed spells `APNLEMB1`. It is defined here, rather than +in the doctrine's CoreML seed table, but it still follows the same +never-collide rule. The file compiles only where NaturalLanguage +exists. There is no Rust counterpart here. ADR-019 records this as a +sanctioned Swift-only divergence. Recall fusion simply handles the +absent lane elsewhere. + +## NLContextualEmbeddingProvider.swift + +This file holds `NLContextualEmbeddingProvider`, the higher-quality +Apple lane built on the on-device `NLContextualEmbedding` transformer. + +The transformer needs a per-language downloadable asset that may be +absent. The file's central rule holds that an embed call must never +trigger a network fetch as a side effect. The provider checks asset +availability with a free, synchronous call. It opts out with an empty +vector when the asset is missing. Prefetching assets is the host +application's job, done before the provider gets constructed. When +assets exist, the provider runs the transformer. It mean-pools the +per-token vectors, the conventional strategy for a transformer without a +sentence-pooling head, with a defensive dimension guard that skips +malformed token vectors. Then it normalizes the result. Each failure +mode collapses to the empty-vector opt-out, since a missing asset is +an expected operational state. The projection seed spells `APNLCTX1`, +distinct from the sentence provider's, so the two lanes key to separate +storage partitions. This provider is Apple-only, with no Rust +counterpart, per ADR-019. + +## Rust Port and Conformance + +The `rust/` directory mirrors the core target. The `rust-providers/` +directory mirrors the providers target, matching the two-target split, +so core consumers never depend on provider code. The core crate ships +the same chunk, chunker, tokenizer, store, engine, hybrid-recall, +ingest-queue, and `Corpus` types. Concurrency uses locks where Swift uses +actors. The queue and the inverted-index store own private connections +there, where Swift instead shares the estate's storage. The providers +crate ships the deterministic tokenizer and the shared +count-and-vocabulary-reduction code. It also ships the basis codec, the +five honest signals, and the three named neural providers. All of them +share the same host-injected inference seam. Neither crate bundles +model weights. + +Three fixture families gate the ports. The shared canonical vectors in +`Tests/SharedVectors/` cover BM25 impacts, per-provider embedding +vectors, and serialized basis blobs. Both legs read them, and both must +reproduce them byte for byte. The Rust test suite also pins +basis serialization byte-for-byte for all four trainable providers, and +it round-trips the counts seam. The two Apple NaturalLanguage providers +are the one sanctioned divergence. They exist only in Swift, and the +parity baseline is the deterministic and classical providers. When you +change either leg, run both test suites. The fixtures are the contract. diff --git a/packages/kits/CorpusKit/docs/OVERVIEW.md b/packages/kits/CorpusKit/docs/OVERVIEW.md new file mode 100644 index 0000000..b84e656 --- /dev/null +++ b/packages/kits/CorpusKit/docs/OVERVIEW.md @@ -0,0 +1,223 @@ +--- +doc: OVERVIEW +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# CorpusKit Overview + +## What This Kit Does + +CorpusKit stores text. It finds that text again by meaning, and also by +keyword. It is the retrieval tier of MOOTx01, an on-device AI memory +system. The technique it uses has a name: retrieval-augmented generation, +or RAG. In RAG, an AI looks up stored material first. Then it answers. +The answer rests on real text, not on guesswork. + +A kit is a larger package. It composes libraries into one subsystem. +CorpusKit composes storage, indexing, and embedding libraries. Together +they form one database-like surface. Callers hand it text. The kit +splits the text into chunks. It stores each chunk, then indexes it two +ways. A chunk is a piece of source text with a stable identity. Each +chunk is sized to a few sentences. Later, a caller hands the kit a +query. The kit returns the most relevant chunks, scored and ranked. + +The kit stands alone. A developer can use it as a private RAG database. +No other MOOT component is required. Inside MOOTx01, the GeniusLocusKit +orchestrator uses CorpusKit differently. It treats the kit as the +estate's recall engine. An estate is one user's complete memory store. + +## The Problem It Solves + +Recall must work on the device. It must be deterministic. It must never +leak text. Cloud embedding services fail all three tests. They see +private content. They need a network connection. They change without +notice. If recall depended on such a service, a user's memory would be +neither private nor reproducible. + +Federation raises the stakes further. MOOTx01 estates can share memories +across devices. Shared recall only works when every device computes the +same result from the same input. Call this the agreement property. + +CorpusKit answers with two ranked lanes. Both run entirely on the +device. A lane is one independent way to score how well a chunk matches +a query. The keyword lane uses BM25. BM25 is a standard formula. It +rewards chunks that contain the query's rarer words. The semantic lane +uses embeddings instead. An embedding is a list of numbers, called a +vector, that represents what a text means. Texts with similar meaning +get nearby vectors. A rule called Reciprocal Rank Fusion then merges the +two lanes into one ranking. The rule is simple: it rewards any chunk +ranked high in either lane. + +For the semantic lane, the kit ships an ensemble of five signals. Each +one is "honest." Honest means the signal reflects real word +co-occurrence, or real classification structure. It is never a disguised +hash of the surface text. The five signals are Random Indexing, PPMI, +LSA, NMF, and FDC. All five are classical statistical methods. None +needs a neural network. Each costs little to run. Each runs identically +on every platform. Shared conformance fixtures gate all five: recorded +input and output pairs that the Swift leg and the Rust leg must +reproduce exactly, byte for byte. The Rust leg lives in `rust/` and +`rust-providers/`. Optional neural providers plug into the same seam +when a host supplies a model. Four ship today: MiniLM, mpnet, +EmbeddingGemma, and two Apple NaturalLanguage providers. + +## How It Works + +Ingestion runs as a pipeline. Text enters through an ingest queue backed +by QueueKit. Callers never wait on encoding to finish. A background +drain worker takes batches from the queue. It hands each batch to the +`Corpus` actor, the kit's central type. The `Corpus` splits each text +into chunks with sentence-aware boundaries. Each chunk then receives a +content-addressed identity. Its UUID is computed from its source, its +offset, and its exact text. The same content always produces the same +identity. A repeat ingest of one document is therefore a harmless +no-op. Two federated devices that ingest the same content converge on +identical rows. + +Each stored chunk is indexed twice. The keyword side tokenizes the +chunk. It records term frequencies in a persistent inverted index. An +inverted index is a table that maps each word to the chunks that contain +it. The semantic side runs the chunk through every configured embedding +signal. It writes the resulting vectors to VectorKit, the sibling kit +that owns vector storage and nearest-neighbor search. Content and +vectors join on the chunk's UUID string. A chunk and its meaning never +drift apart. + +Four of the five honest signals are trainable. They learn a basis from +the corpus itself. A basis is the trained reference data a signal needs +to embed new text. One example is the word co-occurrence vectors that +Random Indexing accumulates. Training happens exactly twice: once +automatically on first ingest, and again whenever a caller requests an +explicit reindex. Trained bases serialize to a pinned little-endian byte +format and persist to storage. A reopened corpus embeds text right away. +It never has to retrain first. + +Recall runs the same pipeline in reverse. The query is embedded once. +The vector lane fetches its nearest neighbors. The keyword lane fetches +its best BM25 matches. Reciprocal Rank Fusion merges the two rankings. +It uses pinned weights: 0.6 for the vector lane and 0.4 for the keyword +lane. The winning chunks hydrate from the chunk store, then return to +the caller as scored chunks. Ties always break toward the smaller +identifier. Results stay deterministic down to the last position. + +Deletion is honest about its own limits. Chunk rows are immutable, so +removing a source cannot erase its chunk rows. Instead it deletes the +source's index rows and vectors. It also records a tombstone, which +every rebuild consults afterward. Expunging goes one step further: it +scrubs the stored text itself. + +## How the Pieces Fit + +Figure 1 shows the kit's topology. It shows the major parts, and how +data moves between them. + +![Figure 1. Topology of CorpusKit](topology.svg) + +*Figure 1. Topology of CorpusKit. Ingested text flows through the queue +and the `Corpus` actor into the chunk store, the keyword index, and the +vector store. A query fans out to both index lanes. Reciprocal Rank +Fusion merges them into scored chunks. Dashed regions mark the external +kits and the persisted tables.* + +The `Corpus` actor is the seam that everything passes through. It owns +the chunk store, called `BundleStore`. It owns the persistent keyword +index, called `InvertedIndexStore`. It also owns the basis store, the +counts store, and the tombstone store. It holds one slot per embedding +signal, too. The actor hides VectorKit behind its own surface. Consumers +never touch vector storage directly. + +Beneath the actor sits an engine layer. The layer holds the sparse +types, the BM25 weighting code, the inverted index, and the fusion +function. The inverted index runs two exact algorithms, WAND and +Block-Max WAND. The whole engine layer is pure computation. It holds no +storage of its own. + +The package splits into two targets on purpose. The `CorpusKit` core +target holds the storage layer, the engines, and the protocols. The +`CorpusKitProviders` target holds every concrete embedding provider and +tokenizer. A consumer that needs only storage and BM25 search never has +to pull in provider code or model seams. + +## What Ships in the Package + +The package ships two Swift targets. Together they hold thirty-four +Swift source files. It also ships two mirror Rust crates: `rust/` for +the core and `rust-providers/` for the providers. Shared canonical +vectors in `Tests/SharedVectors/` gate both legs. Three things must +match byte for byte across the legs: BM25 impacts, per-provider +embeddings, and serialized basis blobs. The kit bundles no model weights +of its own. A host that wants neural embeddings must inject an inference +function. Everything the kit itself computes stays deterministic. It +stays reproducible from the sources alone. diff --git a/packages/kits/CorpusKit/docs/topology.svg b/packages/kits/CorpusKit/docs/topology.svg new file mode 100644 index 0000000..228e080 --- /dev/null +++ b/packages/kits/CorpusKit/docs/topology.svg @@ -0,0 +1,126 @@ + + + + + + + + + + + + + CorpusKit: text in, scored chunks out + + + + Text in + enqueueIngest + + + Ingest queue + QueueKit · encode stream + + + Corpus actor + composition root + + + Chunker + 800 chars · 100 overlap + + + + + + + + + Embedding ensemble + RI · PPMI · LSA · NMF · FDC + + + + + + VectorKit (external kit) + + VectorStore + engrams + float rows + + + + + + Persisted tables (PersistenceKit; SQLite or in-memory, chosen by the host) + + + BundleStore + chunks · Merkle roots + + + InvertedIndexStore + BM25 stats · WAND/BMW + + + Basis · counts · tombstones + trained bases · removed sources + + + + + + + + + Query + recall(query) + + + HybridRecall + Fusion + RRF k=60 · weights 0.6/0.4 + + + ScoredChunk results + score DESC, id ASC + + + + + + + + + diff --git a/packages/kits/LocusKit/docs/AGENT_MAP.md b/packages/kits/LocusKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..e1cac03 --- /dev/null +++ b/packages/kits/LocusKit/docs/AGENT_MAP.md @@ -0,0 +1,481 @@ +--- +doc: AGENT_MAP +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# AGENT_MAP : LocusKit + +PURPOSE: storage substrate for one MOOTx01 estate (MemPalace). Nine gated noun tables (drawer/tunnel/diary/kg_fact/proposal/association/learned_reference/source_catalog/node) over PersistenceKit; every state-changing write routes through SubstrateLib's AuditGate (legal-transition + forbidden-combination check) and appends one sealed AuditEvent atomically with the row update. Read side: RecallFrame → [Filter] chain → fingerprint-pruned, four-tier BitmapEvaluator (bitmap/structured/content/order) → paged RecallStream. Estate (actor) is the sole public facade; GeniusLocusKit composes on top. + +DEPS: imports SubstrateLib+SubstrateTypes+SubstrateKernel (AuditGate, BitField, RowStateAutomaton, HLC, SimHash, ORReduce, FNV, MerkleHash : DO NOT REIMPLEMENT), SubstrateML (LexRank-adjacent kernels used indirectly), PersistenceKit (Storage/RowStore/AuditLog/Transaction abstraction : SQLite/Postgres/InMemory backends), IntellectusLib (opt-in telemetry, off-path ~1ns), LatticeLib (QIDClosure : pinned P31/P279 ancestor snapshot, DrawerFingerprint lattice block only). Imported by: GeniusLocusKit (composes on top : never the reverse). Rust port in rust/ (57 files under src/+tests/) mirrors every type + the bitmap/state/fingerprint math; conformance tests in rust/tests/*.rs gate byte-identity per axis. + +ENTRY POINTS (most callers need only these): +- Estate.swift:187 `Estate.open(storage:owner:identityKeyStore:) -> Estate` : open existing estate, validates bitmap_layout_version, mints/loads Ed25519 identity +- Estate.swift:314 `Estate.create(storage:owner:manifest:) -> Estate` : create new estate +- EstateVerbs.swift:69 `Estate.capture(_ frame: CaptureFrame) -> Drawer` : file a drawer +- EstateVerbs.swift:619 `Estate.recall(_ frame: RecallFrame) -> RecallStream` : query drawers +- EstateVerbs.swift:1228 `Estate.mutate(rowID:kind:payload:)` : move state/adjective axis +- EstateVerbs.swift:1030 `Estate.expunge(rowID:reason:confirmation:now:)` : hard delete + audit + +## Symbol Table + +### Module : LocusKit.swift +- :1 doc-only file; no public symbols. Public types live in neighbouring files. + +### Errors : LocusKitError.swift +- :11 `enum LocusKitError: Error, Sendable, Equatable` : every LocusKit failure mode +- :18 `.databaseUnavailable(String)` / :21 `.drawerNotFound(id:)` / :23 `.tunnelNotFound` / :27 `.diaryEntryNotFound` / :32 `.recallTraceItemNotFound` +- :38 `.sqliteError(String)` / :42 `.schemaTooNew(found:expected:)` +- :48 `.invalidContent(String)` : msg IS the contract, tests assert on text +- :59 `.disciplineViolation(from:to:reason:)` : illegal transition / forbidden combo; from/to are raw Int, not State (keeps LocusKitError dependency-free of Adjectives) +- :72 `.corruptStoredValue(table:column:storedText:)` : fail-loud, never fabricate a default +- :81 `.notSupported(String)` +- :90 `description: String` : English render consumed by GLK's gate-rejection text parser (soft contract) + +### Schema : LocusKitSchema.swift +- :80 `kitID = "LocusKit"` / :90 `version = 8` (no migration ladder : no estate data has shipped) +- :97 `static var schema: SchemaDeclaration` : 17 tables + generated columns + indices, ALL declarative (zero raw SQL) +- Every persistent table carries nullable `.json` `ext` column (ADR-012 forward-compat slot) +- Generated columns: `g_state_cluster` (adjectiveBitmap & 0x3F, on drawers/kg_facts/proposals), `g_provenance_source`/`g_provenance_confirmation`/`g_operational_channel` (drawers only) : indexed like ordinary columns +- Header comment: bitmap reservation map (assigned vs FREE bit ranges per column) : consult before claiming a new flag bit + +### Vocabulary : LocusKitVocabulary.swift +- :27 `enum LocusKitVocabulary` : LocusKit's operational+provenance FieldSlot union (adjective basis supplied by SubstrateLib itself) +- :61 `frozen() -> Result` : called once at DrawerStore.init; frozen for estate lifetime + +### Estate types : EstateTypes.swift +- :15 `typealias RowID = String` +- :27 `struct FrameFilteredDrawers` : :30 `admissible: [Drawer]`, :32 `loadedIDs: Set` (loaded-but-filtered vs never-loaded distinction) +- :51 `struct OwnerCredentials` : :55 `ownerIdentifier: String` (non-empty enforced at Estate.open/create) +- :76 `struct LatticeAnchor: Equatable, Codable` : udcCode(required)/udcFacets/wikidataQID/wikidataQidsSecondary; :101 `.udc(_:)` convenience +- :117 `enum EstateError` : :121 substrateUnavailable / :128 manifestMismatch(key:found:expected:) / :133 emptyOwnerIdentifier / :140 keychainError(status:) + +### Estate facade : Estate.swift +- :27 `actor Estate` : owns :45 `store: DrawerStore` (internal, EstateVerbs-only), :50 `containerFP: ContainerFingerprintStore`, :57 `nodeStore: NodeStore` (public) +- :36 `expectedBitmapLayoutVersion = "v1.0"` : Estate.open refuses mismatch +- :187 `open(storage:owner:identityKeyStore:)` : mints Ed25519 keypair on first open (public key → manifest, private key → identityKeyStore); backfills containerFP via rebuildAll +- :314 `create(storage:owner:manifest:)` : no keypair minted (deferred to first open) +- :401 `close()` : no-op; caller's storage owns teardown +- :412/:429/:448 `allDrawers(...)` overloads : full/limit/hydrationLevel variants +- :462 `tombstonedLineageIDs() -> Set` : cluster-C block-set for vault re-import guard +- :474 `roomLevelFingerprints()` : passthrough to containerFP.roomLevelEntries +- :485/:494 `getDrawers(ids:...)` : batch by-id, hydration-level aware +- :527 `getDrawers(ids:matchingFrame:hydrationLevel:) -> FrameFilteredDrawers` : frame-faithful O(candidates) by-id load; forces .full when chain has content predicate +- :566 `hydrateBodies(ids:)` : late body hydration for dense-first pool pattern +- :578 `tunnelsFromWing(_:)`, :587 `recentRecallTraces`, :594/:604 `allTunnels`/`allActiveTunnels`, :612/:621 `retireTunnel`/`unretireTunnel`, :635 `pruneRecallTraces`, :651 `markRecallTracesUsed`, :657/:665 `countRecallTraces`/`countDrawerRows` +- :674/:681/:689/:698/:706/:713 `allProposals`/`allAssociations`/`allLearnedReferences`/`allKGFacts`/`allKGFactsIncludingRetired`/`allDiaryEntries` +- :722 `resolveNodeNames(parentNodeIds:)` +- :737 `manifest: ManifestValues { get async throws }` : re-reads on every access +- :745 `estateUUID: UUID` : stable, parsed once at open +- :759 `retrievePrivateSigningKeyData() -> Data?` +- :780/:789 `meta(key:)`/`setMeta(key:value:)` : public per-estate KV surface (caller must namespace keys) +- :70 `RecallInternalRead` enum + :97 `_testForceInternalReadError` : TEST-ONLY single-use fault-injection seam for recall's internal reads + +### Estate audit : EstateAudit.swift +- :53 `auditTrail(rowID:) -> [AuditEvent]` : full sealed history, HLC order; empty until first mutation (capture = INSERT, not mutation) +- :83 `bitmapState(rowID:asOf:) -> BitmapState` : folds audit log via AuditLogFold.projectStateAt; throws drawerNotFound if no events ≤ asOf + +### Identity keys : EstateIdentityKeyStore.swift +- :23 `protocol EstateIdentityKeyStore: Sendable` : loadPrivateKey/storePrivateKey(forEstateID:) +- :48 `struct KeychainEstateIdentityKeyStore` : kSecClassGenericPassword, service "com.mootx01.estate.identity", account=estateUUID, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly; production +- :130 `final class InMemoryEstateIdentityKeyStore: @unchecked Sendable` : NSLock-guarded dict; test-only; :147 `_storedPrivateKey(forEstateID:)` test peek + +### Verbs : EstateVerbs.swift (extension Estate) +- :69 `capture(_ frame: CaptureFrame) -> Drawer` : validates 5 required fields; assembles 3 bitmaps via BitField.writeField; resolves wing/room to node ids; routes through :432 `addDrawerCovered` (the ONLY sanctioned add chokepoint : bundles store.addDrawer + containerFP.orIn) +- :239 `captureBatch(_ frames: [CaptureFrame]) -> [Drawer]` : bulk path; ONE transaction for all fresh inserts via `store.insertFreshBatch`; Merkle rollup deliberately deferred (NT_R1) +- :487 `capture(_ frame: TunnelCaptureFrame) -> Tunnel` : standalone tunnel capture (2nd of the two nouns capture accepts) +- :573 `recallCandidateCap = 256` : scan bound floor for director-style callers +- :619 `recall(_ frame: RecallFrame) -> RecallStream` : non-throwing; internal-read failures surfaced via `RecallStream.degradedStages`, never silently emptied +- :743 `enum RecallStage` : stable stage-id strings (locus.liveRows.readFailed, locus.roomFingerprints.readFailed, locus.roomDrawerRead.readFailed, locus.bitmapEval.failed, recall.trace_write_failed) +- :800 `liveRows(for:forcedFault:)` (private) : fingerprint-pruning OR bounded-scan dispatch; no-blob fast path when chain has no content predicate +- :960 `withdraw(rowID:reason:)` : state → .withdrawn via mutateState(.retract) (NOT mutateAdjective : must go through automaton) +- :1030 `expunge(rowID:reason:confirmation:now:) -> AuditEvent?` : requires confirmation:true; walks full lineage chain; rolls up Merkle for every affected room +- :1085 `expungeReturningUnsealedEvent(...)` / :1141 `sealExpungeAudit(_:)` / :1155 `sealExpungeOrphanAudit(...)` : GLK's 3-step §B-2a orchestration (storage tombstone → cross-kit delete → seal); sealAudit:false is the ONLY split path +- :1178 `sealExpungeOrphanAuditSynthetic(rowID:now:)` / :1196 `tombstonedRowsWithoutExpungeAudit()` : crash-window recovery sweep support +- :1228 `mutate(rowID:kind:payload:)` : dispatches MutationKind → confirm(provenance)/reject|contest|resolve|accept|supersede(state)/revive(state, full per-source-state legality table incl. living-successor lineage guard)/correctSensitivity|correctTrust|correctExportability(adjective field) +- :1528 `reanchor(rowID:toRoom:toWing:toLattice:)` : placement move, requires ≥1 non-nil arg; toWing rejects empty/whitespace +- :1578 `reanchorAnchor(rowID:toLattice:changedBy:now:)` : deterministic-timestamp variant for Q-ID-completion acceptance path +- :1615 `propose(_ frame: ProposeFrame, now:) -> Proposal` : derives candidateState+latticeAnchor from target drawer +- :1690 `associate(_ frame: AssociateFrame, now:) -> Association` : derives latticeAnchor from endpoint A +- :1774 `learn(_ frame: LearnFrame, now:) -> LearnedReference` : catalogs source once by handle; NEVER fabricates a sentinel anchor +- :1936 `defaultWing() -> String` (private) : returns "Agentic Memory" (ADR-016, fixed, not owner-derived) +- :1971 `seedWing(_:hint:addedBy:embeddingModelID:now:)` : files ordinary recallable hint drawer per default wing at provision + +### Frames : Frames.swift +- :24 `struct CaptureFrame` : every named capture-time axis; nearly all defaults reproduce pre-slot all-zero bitmap byte-identically +- :178 `struct TunnelCaptureFrame` : no content/anchor/embedding slots (edges carry none) +- :237 `struct RecallFrame` : filterChain/hydrationLevel/limit/ordering/asOf/traceLimit (traceLimit nil = ZERO trace writes, the default) +- :287 `enum MutationKind` : .confirm/.reject/.contest/.resolve/.supersede/.revive/.accept/.correctSensitivity(_)/.correctTrust(_)/.correctExportability(_) +- :338 `struct LearnFrame` : source(SourceCatalogEntry)/handle/mode/refreshPolicy +- :382 `struct ProposeFrame` : target/kind/justification/confirmation/generatedBy/confidence (LocusKit substrate-axis kind, NOT GLK's Brain-layer kind) +- :436 `struct AssociateFrame` : a/b/weight +- :455 `enum HydrationLevel` : .structured/.full/.bitmapOnly +- :476 `enum Ordering` : .byCaptureTimeDesc/.byCaptureTimeAsc/.byRoomAsc (NO .byRelevanceDesc : LocusKit has no scoring signal; that's VectorKit/GLK's job) + +### Storage engine : DrawerStore.swift (actor, largest file, ~3850 lines) +- :57 `actor DrawerStore` : owns every table; :90 `init(storage:hlc:)` opens schema idempotently, freezes vocabulary, classifies estate_uuid (absent/present/CORRUPT-throws), derives HLC maker node id by FNV-1a hashing raw stored uuid text +- :291 `addDrawer(_:now:)` (internal : NOT the public add path, use Estate.addDrawerCovered) : dispatches supersession cascade vs plain gatedCapture +- :346 `addDrawerWithCascade(_:priorID:)` (private) : successor+tunnel in ONE transaction (no orphan on tunnel-insert failure), THEN separate gated mutateState(.superseded) on predecessor +- :405 `findActivePredecessor(lineageID:excludingID:)` : indexed g_state_cluster < 3 query +- :439 `livingSuccessorInLineage(lineageID:excludingID:)` : g_state_cluster < RowState.activeClusterUpperBoundRaw(16); wider predicate than findActivePredecessor, used by revive guard +- :468 `lineageChain(for:) -> [String]` : every row sharing lineageID, any state +- :492/:516/:545 `getDrawer(id:)`/`getDrawers(ids:)`/`getDrawers(ids:hydrationLevel:)` : chunked at 900 ids (SQLite 999-var ceiling); :530 `structuredDrawerColumns` = every column except content +- :591/:627/:657 `drawersIn(wing:)`/`drawersIn(wing:room:)`/`drawersBySource(file:)` +- :678/:718 `allDrawers()`/`allDrawers(hydrationLevel:limit:direction:)` : (filedAt,id) compound sort key, portable across SQLite/Postgres/InMemory (NOT rowid); .descending retains newest under a scan cap +- :774/:795/:814 `mutateProvenance`/`mutateAdjective`/`mutateOperational` : all route through :1708 `gatedColumnWrite` (private) which excludes the state field from field-edit writes +- :858 `gatedCaptureBody(_:now:)` (private) : builds @Sendable closure for capture INSERT + AuditGate.admit(verb:.capture, prior:nil); reused by both gatedCapture and addDrawerWithCascade (no nested-transaction support in PersistenceKit v1.0) +- :912 `gatedCapture(_:now:)` (private) : single-drawer non-cascade path +- :932 `insertFreshBatch(_:now:)` (internal) : ALL inserts + audit events in ONE transaction; caller (captureBatch) pre-verifies no active predecessor +- :979 `mutateState(drawerId:to:via:changedBy:reason:now:)` : RMWs prior bitmaps, AuditGate.admit(verb:), writes merged snapshot + audit event atomically +- :1097 `expungeGated(drawerId:...:sealAudit:commitmentKey:commitmentKeyVersion:) -> AuditEvent?` : lineage-wide: gates target via .tombstone, unconditionally zeroes content on EVERY sibling regardless of gate success/failure (destruction contract > state machine), records erasure ledger entries +- :1343/:1378/:1516 `sealExpungeAudit`/`sealExpungeOrphanAudit`/`sealExpungeOrphanForSweep` : deferred-seal + crash-recovery paths +- :1425 `tombstonedRowsWithoutExpungeAudit()` : crash-window orphan enumeration (no "tombstone"/"expungeOrphan" audit event) +- :1469 `tombstonedLineageIDs() -> Set` +- :1576 `reanchorGated(...)` : verb:.mutate (active→active self-loop); anchor delta via priorLatticeAnchor≠afterLatticeAnchor; writes:[] (placement move, no field write) +- :1689 `requireUuid(_:label:) throws -> UUID` (static) : fail-loud on non-UUID row id +- :1783 `declaredSlots(for:)` (static private) : adjective ⇒ Vocabulary.basis (SubstrateLib); operational/provenance ⇒ LocusKitVocabulary.unionSlots +- :1800/:1805 `auditEventsForRow`/`auditEventCountForRow` +- :1832 `addTunnel(_:)` : enforces "one parent per child" for .parent kind (kit-level constraint, not DB unique index) +- :1954/:1963/:1982/:2001 `allActiveTunnels()`/(T13/ADR-021 Phase 7 retirement)/`retireTunnel`/`unretireTunnel` : bit-13 operationalBitmap flag, filtered in-memory (PersistenceKit predicate DSL has no bitmask ops) +- :2019/:2044/:2076 `outlineChildren(of:)`/`outlineAncestors(of:)` (depth cap 256, cycle-safe)/`reparentDrawer(...)` : ADR-017 §11 outline graph over .parent tunnels + order_key +- :2139/:2166 `addKGFact(_:)`/`withdrawKGFact(id:)` : sourceDrawerID may be "" (unanchored-fact sentinel) +- :2228 `addProposal(_:)` : rejects empty latticeAnchor.udcCode +- :2264 `addAssociation(_:)` : rejects empty latticeAnchor.udcCode +- :2328 `addLearnedReference(_:)` / :2372 `addSourceCatalogEntry(_:)` : both reject empty latticeAnchor.udcCode +- :2408/:2428/:2442 `addDiaryEntry`/`readDiary(agentName:lastN:)`/`readDiary(agentName:in:lastN:)` +- :2592/:2614 `insertRecallTrace`/`insertRecallTraces` (batch, one transaction) : :710 in EstateVerbs is the only production write path (opt-in via frame.traceLimit) +- :2713/:2758 `markRecallTraceUsed(id:now:)`/`markRecallTracesUsed(target:since:now:)` : bulk ARIA reward path +- :2692 `pruneRecallTraces(olderThan:)` / :2807 `countRecallTraces()` / :2827 `countDrawerRows()` (raw COUNT(*), bypasses decode : counts corrupt rows too, used as corruption sentinel) +- :2836/:2881/:2932 `listWings()`/`listRooms(in:)`/`taxonomy()` : computed projections, no dedicated wings/rooms table +- :2939/:2948/:2959 `setMeta`/`getMeta`/`readManifest() -> ManifestValues` +- :3095 `resolveNodeNames(parentNodeIds:) -> [String: (wing:room:)]` +- :3143/:3150 `decodeDrawerRows` (throws on any corrupt value : point-lookup path) / `decodeDrawerRowsResilient`→:3419 `decodeDrawerRowsSkipCorrupt` (logs+skips corrupt rows : corpus-scan path; only .corruptStoredValue is caught, everything else rethrows) +- :3682/:3731 `fingerprintsCaptured(in:)`/`fingerprintBitSeries(bit:bucketSeconds:bucketCount:endingAt:)` : feed FFT rhythm-spectrum / moment-summary lenses; effective capture time = eventTime ?? filedAt (ING-01) + +### Drawer value type : Drawer.swift +- :31 `struct Drawer: Equatable, Hashable, Sendable, Codable` : verbatim content, never truncated/normalized +- :37 `id` / :46 `lineageID: UUID` (fresh per-row default) / :49 `content` / :54 `parentNodeId: String` (FK → nodes.id, room-depth node) +- :75 `filedAt: Date` (ingest clock) vs :83 `eventTime: Date` (world-happened clock; ING-01 two-clock ingest; defaults to filedAt) +- :109/:120/:132 `provenance`/`adjectiveBitmap`/`operationalBitmap: Int64` : all default 0 +- :151/:159/:166/:175 `udcCode`(required non-empty, I-5)/`udcFacets`/`wikidataQID`/`wikidataQidsSecondary` +- Custom Codable (:228) backfills missing eventTime → filedAt on decode + +### Drawer operational : DrawerOperational.swift +- :61 `enum CaptureChannel: Int` : typed=0/voiced=1/ocr=2/importedFile=3/sensor=4/actuator=5 (bits 0–5) +- :78 `enum ContentKind: Int` : prose=0…fingerprintOnly=6 (bits 6–11) +- :100 `struct DrawerFeatureFlags: OptionSet` : bits 12–23; hasAttachments/hasVoice/hasImage/hasLinks/isPinned/isKeystone(§7.2)/isLockedZone +- :141/:149/:159/:170/:179/:188 `Drawer.captureChannel`/`.contentKind`/`.featureFlags`/`.hasFeatureFlag(_:)`/`.stateExtensionActive`(bit24)/`.lineageClusteringActive`(bit25) : all fail-safe to neutral default on unrecognised raw + +### State transition bridge : DrawerStateValidator.swift +- :53 `validate(from:to:via:)` : transition-legality only, bridges State↔SubstrateLib.RowState, delegates to RowStateAutomaton.transition +- :107 `validate(from:to:via:targetingFields:)` : legality + field-invariant (S-1 trust≥canonical, S-2) combined; the overload DrawerStore.mutateState actually calls + +### Drawer fingerprint : DrawerFingerprint.swift +- :75 `struct EstateFingerprintFamilies` : 4 hyperplane families derived once per estate from estate UUID via HyperplaneFamily.blockFamilies; :90 `baseSeed(estateUUID:)` +- :106 `fingerprint(of: Drawer) -> Fingerprint256` : 4 SimHash blocks: bitmap-LSH(adj/op/prov) / lattice(udcPrefixHash+qidDirectHash+qidClosureHash via LatticeLib.QIDClosure) / lineage+temporal(lineageHash+captureWeekBucket, ING-01 keys off eventTime) / channel+source(channel/sourceType/captureChannel/sensitivity/estateUUIDByte) +- :161/:171 `captureWeekBucket(_:)` (mod 256, 2020-01-01 epoch) / `udcPrefixHash(_:)` (FNV-16 of first 4 digits) +- I-17: cross-noun null fields (defer pattern, completion bucket, behavioral recency, stream-source bitset) always zero on a Drawer : those belong to AmbientSample only + +### Adjective axes : Adjectives.swift +- :99 `enum State: Int` : 10 cases, 3 clusters at raw boundaries 16/32: A(active/pending/contested/accepted) B(superseded/decayed/withdrawn/expired) C(rejected/tombstoned); :121 `isClusterA` +- :143 `enum Trust: Int, Comparable` : verbatim=0…ambient=6(NEW v0.6) +- :173 `enum AdjectiveSensitivity: Int` : normal=0/elevated=16/restricted=32/secret=48 (scale-gapped); ADR-007 tier predicates :197 `isBulkExportable`/:217 `requiresOwnerKeyForBulk`/:237 `isExcludedFromBulk` +- :256 `enum AdjectiveExportability: Int` : private_=0/public_=32 +- Drawer accessors :270-:301 `state`/`adjectiveSensitivity`/`exportability`/`trust` (bits 0-5/6-11/12-17/18-23); :311/:323/:338 `isCurrentlyBelieved`/`isKnewPast`/`isTerminal` cluster predicates +- :354 `dreamingRecalcRequired` (bit 26, obligation flag, F17) / :370 `sealed` (bit 27, custody trust hint : cached, seal itself is authoritative if they disagree) +- `@guardian-pair` annotations throughout : SwiftSyntax Guardian CI tool cross-checks these raw sets against SubstrateLib's AuditGate.basis; GuardianPairParityTests backstop + +### Provenance axes : Provenance.swift +- :67 `enum SourceType: Int` : user=0…actuator=9 (bits 0-5) +- :102 `enum Channel: Int` : uiTyped=0…dreamMiningResult=8, deviceSensor=15, actuatorOutcome=16 (bits 6-11, gaps reserved) +- CaptureChannel mirrored at bits 12-17 (provenanceCaptureChannel accessor, :224) +- :137 `enum Confirmation: Int` : unconfirmed=0/userConfirmed=1/automatedConfirmed=2/peerConfirmed=3/actuatorConfirmed=4 (bits 18-23); :261 `isUserConfirmed` +- :155 `enum Confidence: Int, Comparable` : null=0/low=16/medium=32/high=48/verified=56 (bits 24-29) +- :177 `enum Sensitivity: Int` : normal/elevated/restricted/secret mirrors AdjectiveSensitivity raws (bits 30-35) +- :188 `enum EnrichmentStatus: Int` : none/qidPending/qidCompleted/closureCached/qidProposed(terminal, NOT re-picked by maintenance scan) (bits 36-41) + +### Bitmap primitives : BitmapOps.swift +- :55 `andMask(_:mask:expected:)` → BitField.maskedEquals +- :94 `thresholdCompare(_:mask:shift:op:value:)` + :69 `enum ThresholdOp` (.lessThan/.lessThanOrEqual/.greaterThanOrEqual) → BitField.extractField+compare +- :124 `shiftExtract(_:shift:mask:)` → BitField.extractField +- NOT math primitives : no conformance gate; real math primitives live in SubstrateLib (XOR/popcount/Hamming/OR-reduce/SimHash/SHA-256) + +### Forbidden combination : ForbiddenCombinationValidator.swift +- :75 `validate(_:) throws` : I-3: sensitivity=secret(raw48,bits6-11) AND exportability=public(raw32,bits12-17) forbidden; hand-derived raw constants (NOT imported enums) so a rename can't silently break the check +- No live call site today (current writes route the equivalent check through AuditGate/SubstrateLib directly) : ready to wire in wherever a write path bypasses the gate + +### Recall filter algebra : Filter.swift +- :8/:12/:15/:20/:25/:30 typealiases `LineageID`(UUID)/`RoomID`(String)/`WingID`(String)/`WikidataQID`(String)/`ProvenanceChannel`(=Channel)/`FeatureFlag`(=DrawerFeatureFlags) +- :48 `enum StateCluster` : knowNow/knewPast/terminal +- :69 `indirect enum Filter` : NO raw bit position ever exposed; state/trust/sensitivity/exportability/provenance/operational/structural/content cases + :166/:168/:170 `.all`/`.any`/`.not` composition +- RecallFrame.filterChain ([Filter]) ≡ implicit Filter.all(chain) + +### Recall evaluator : BitmapEvaluator.swift +- :60 `struct BitmapEvaluator` (internal : DrawerStore param is internal) : 4-stage pipeline +- :62-125 private layout-constant mirror of Adjectives/Provenance/DrawerOperational bit positions (schema-bump-coupled) +- :147 `evaluate(frame:drawers:store:nodeNames:) -> [Drawer]` : bitmap tier (w/ historical reconstruction via AuditLogFold.projectStateAt when frame.asOf set) → structured tier → content tier → sort +- :221 `insertDefaults(_:)` (private) : prepends .currentlyBelieve/.trustworthy/.sensitivityAtMost(.elevated) when caller's chain doesn't already constrain that axis; NO confirmation default +- :290/:304/:314 `chainHasPrunableFilter`/`chainHasContentPredicate`/`chainHasStructuredNameFilter` (static, called by Estate.liveRows/getDrawers(matchingFrame:) before fetch) +- :349 `containerSurvives(chain:fingerprint:) -> Bool` : sound pruning test; only set-bit filters (.hasFeatureFlag) can prove exclusion from an OR fingerprint; :358 `containerProvablyExcludes` +- :384 `evaluateBitmapTier` : tombstone (state==33) excluded UNCONDITIONALLY, independent of caller's chain +- :641 `reconstructAt(rowID:asOf:store:)` (private) : historical bitmap fold +- :654 `sort(_:ordering:nodeNames:)` (private) + +### Tunnel : Tunnel.swift +- :24 `struct Tunnel: Equatable, Hashable, Codable, Sendable` +- :29 `id` (conventionally SHA-256 of canonicalised endpoint pair : NOT enforced at this layer, LOCI-5) +- source/target: wing+room+optional drawerId (nil = "the room itself") +- :63 `kind: TunnelKind` (default .references) / :69/:75/:80 adjective/operational/provenance bitmaps / :99 `orderKey: Double?` (fractional-index sibling order, .parent kind only) + +### Tunnel operational : TunnelOperational.swift +- :38 `enum TunnelKind: Int` : supersedes=0…parent=9 (outline containment, ADR-017 §11; source=child, target=parent, one-parent-per-child kit-level constraint) +- :60/:70/:80/:94 `TunnelDirection`/`TunnelLifecycle`/`TunnelOriginClass`/`TunnelStrength(Comparable, scale-gapped 0/2/4/6)` : bits 0-2/3-5/6-8/9-11 +- :168 `isRetiredBit = 1<<13` / :175 `isRetired` / :184/:202 `withRetired()`/`withUnretired()` : T13/ADR-021 Phase 7 reversible dreaming suspension (NOT tombstone) +- :241 `isDreamedBit = 1<<0` (provenanceBitmap) / :247 `isDreamed` / :256 `withDreamedProvenance()` : OMEGA retires ONLY isDreamed==true tunnels; declared tunnels never retired + +### Diary : DiaryEntry.swift +- :24 `struct DiaryEntry` : agentName/entry/topic/wing/room (all required, unvalidated strings) +- :85/:92 `reward: Double?`/`rewardProvenance: String?` : explicit quality-signal channel (NEURONKIT_SPEC §3.1 step 1a); nil ⇒ dreaming daemon falls back to implicit RecallTraceItem.used (step 1b) + +### Diary operational : DiaryOperational.swift +- :33 `enum DiaryEventClass: Int` : 12 cases (capture=0…auditTombstone=11), bits 0-3 +- :54 `enum DiarySeverity: Int, Comparable` : trace=0/info=2/warning=4/error=6, bits 4-6 +- :67 `enum DiaryActorClass: Int` : user=0…federationPeer=4, bits 7-9 +- :79 `enum DiaryBatchMembership: Int` : standalone=0…batchEnd=3, bits 10-12 +- :129 `requiresFollowup` : bit 13 flag + +### KG fact : KGFact.swift +- :51 `struct KGFact` : subject/predicate/object (free-form strings, no entity vocabulary yet) + :82 `sourceDrawerID` (may be "" = unanchored-fact sentinel) +- Adjective accessors :149/:159/:171/:180 `trust`/`state`/`adjectiveSensitivity`/`exportability` : IDENTICAL bit layout to Drawer's (shared retrieval predicates) + +### KG fact operational : KGFactOperational.swift +- :48 `enum KGExtractorClass: Int` : manual=0…federated=5, bits 0-3 +- :66 `enum KGAssertionKind: Int` : asserted=0/inferred=1/hypothesized=2/contradicted=3, bits 4-6 +- :83 `enum KGSpecificity: Int, Comparable` : general=0…instance=6 (scale-gapped), bits 7-9 +- :106 `enum KGConfidenceBand: Int, Comparable` : unknown=0…certain=6 (scale-gapped), bits 10-12 +- :171 `isCanonical` : bit 13 flag (estate-wide load-bearing promotion) + +### Proposal : Proposal.swift +- :72 `struct Proposal` : targetRowID(empty legal for brand-new-object proposals)/justification/:99 `candidateState: Int64`(adjective set to apply if accepted)/:106 `latticeAnchor`(required, I-16) +- :165 `state: State` : decodes proposal's OWN lifecycle axis (pending→accepted/rejected/withdrawn) from adjectiveBitmap bits 0-5 + +### Proposal operational : ProposalOperational.swift +- :60 `enum ProposalKind: Int` : newTunnel=0…tierAdvisory=8, bits 0-5 (COOKBOOK-SPECIFIED layout : conformance-gated, unlike KGFact's LocusKit-internal layout) +- :81 `enum ProposalTargetObjectType: Int` : drawer=0…systemState=6, bits 6-11 (.noneBrandNew=4 ⇒ targetRowID empty) +- :100 `enum ProposalConfirmationSource: Int` : human=0/agent=1/automatedThreshold=2/actuator=3, bits 12-17 +- :116 `enum ProposalGeneratedByClass: Int` : dreamingDaemon=0…tierAggregator=4, bits 18-23 +- :136 `enum ProposalConfidenceBucket: Int, Comparable` : null=0/low=8/medium=16/high=32/verified=48, bits 24-29 +- :207 `composeOperational(kind:targetObjectType:generatedBy:confidence:) -> Int64` (static) : assembles 4 of 5 axes via BitField.writeField; confirmationSource left at .human default + +### Association : Association.swift +- :66 `struct Association: Equatable, Codable, Sendable` (NOT Hashable : embedded LatticeAnchor isn't) : mirrors Tunnel minus `kind`, plus REQUIRED :103 `latticeAnchor` (I-16, lattice-midpoint of endpoints) + +### Association operational : AssociationOperational.swift +- :47 `struct AssociationSignalSources: OptionSet` : bits 0-11 bitset (coRecall/coConfirmed/dreamPairing/vectorSimilarity/sharedEntity/explicitHuman/fingerprintSimilarity/crossEstate/crossTier/actionOutcome); :75 `mask = 0xFFF` +- :90 `enum AssociationDecayClass: Int, Comparable` : pinned=0/slow=16/normal=32/fast=48, bits 12-17 +- :106 `enum AssociationArity: Int` : binary=0(v1 only, I-23)/nAry=1(reserved), bits 18-19 + +### Learned reference : LearnedReference.swift +- :85 `struct LearnedReference: Equatable, Codable, Sendable` (NOT Hashable) : mirrors Association; :96 `sourceCatalogID`(FK→SourceCatalogEntry.id)/:101 `handle`(reference's own locator)/:107 `latticeAnchor`(required, I-16, inherited from catalog entry : NEVER fabricated) + +### Learned reference operational : LearnedReferenceOperational.swift +- :47 `enum RefreshPolicy: Int, Comparable` : none=0/monthly=16/weekly=24/daily=32/onDemand=48/realtime=56, bits 0-5 +- :68 `enum DriftSeverity: Int, Comparable` : none=0/minor=16/major=32/critical=48, bits 6-11 +- :85 `enum LearnMode: Int` : byReference=0/byIngestion=1, bit 12 +- :96 `enum LearnedReferenceSource: Int` : user=0…pairedEstate=5, bits 13-18 (same vocabulary as SourceKind : acquisition channel == source class) + +### Source catalog : SourceCatalogEntry.swift +- :50 `struct SourceCatalogEntry: Equatable, Codable, Sendable` (NOT Hashable) : id/kind(SourceKind)/handle/:68 `latticeAnchor`(required, I-16, GENUINE anchor every LearnedReference from this source inherits)/firstSeen/addedBy +- :103 `enum SourceKind: Int, CaseIterable` : user=0…pairedEstate=5; :119 `fromRaw(_:)` fail-closed to .user + +### Containment tree node : Node.swift +- :25 `struct Node: Sendable, Equatable, Codable, Hashable` : id/parentId(nil only at depth 0)/:34 `displayName`(first-writer casing)/:38 `lookupName`(NFC+casefold+whitespace-collapse, ALL resolution keys use this)/:41 `depth`(0=estate,1=wing,2=room, write-once)/:44 `lifecycle`(0 active,1 tombstoned)/createdHlc/tombstonedHlc/merkleRoot +- :116 `normalizeLookupName(_:) -> String` (static) : CONFORMANCE-GATED, Swift+Rust must byte-match + +### Node store : NodeStore.swift +- :39 `actor NodeStore` : create-on-demand resolution race-free via actor serialization (no INSERT-OR-IGNORE needed) +- :89 `createNode(displayName:parentId:now:) -> Node` : enforces I-NT-5(parent must exist)/I-NT-2(depth≤2)/I-NT-4(no dup active lookupName under parent); tombstoned nodes invisible to resolution (§5 no-resurrection) +- :165 `createRoot(displayName:now:) -> Node` : I-NT-1 single root, idempotent +- :207/:220/:237 `getNode(id:)`/`rootNode()`/`childNodes(parentId:)` +- :259 `tombstoneNode(id:now:) -> Node?` +- :296 `updateMerkleRoot(nodeId:merkleRoot:now:)` : 32-byte BLOB write + +### Node bundle store : NodeBundleStore.swift +- :27 `actor NodeBundleStore` : :32 `enum BundleKind: String` .activeA("A")/.departedB("B") +- :51/:68 `encodeCounts(_:)`/`decodeCounts(_:n:)` : 256×UInt32 LE = 1024 bytes; decode throws (not traps) on wrong size +- :87/:103/:119 `put`/`get`/`rooms(forWing:kind:)` + +### Bundle materializer : BundleMaterializer.swift +- :34 `struct BundleMaterializer` : first real caller of SubstrateKernel.countFold256 +- :63 `materializeRoom(wing:room:now:) -> CountVector256` : filters to State.isClusterA before fold (Bundle A cannot be incremental : no subtraction) +- :79 `rollUpWing(wing:now:) -> CountVector256` : merges room bundles; associative, so room materialize order doesn't matter + +### Fingerprint256 adapters : Fingerprint256Adapters.swift +- :55 `Fingerprint256.init(int64Column:)` / :63 `.int64Column` : packs/unpacks one Int64 bitmap column into block 0 (blocks 1-3 zero); pure type-shape convention, NOT the cookbook §3.2 Bitmap-LSH SimHash interpretation of block 0 : do not conflate the two uses + +### Container fingerprint pruning : ContainerFingerprintStore.swift +- :39 `struct ContainerFingerprint: Sendable, Equatable` : adjective/operational/provenance Int64 OR-aggregate; :68 `merging(_:)` routes through SubstrateLib.ORReduce at Fingerprint256 width +- :87 `actor ContainerFingerprintStore` : :91 `wingRollupRoom = ""` sentinel for wing-level row +- :105/:123 `get(wing:room:)`/`roomLevelEntries()` : nil result means "must scan", not "empty" (unsound to prune against absence) +- :146 `orIn(wing:room:adjective:operational:provenance:now:)` : incremental maintenance, called on every capture; NO clear-side (stale set bits are harmless over-approximation) +- :167/:182/:200 `rebuildRoom`/`rollUpWing`/`rebuildAll` : periodic tightening; rebuildAll runs once at Estate.open + +### Merkle rollup : MerkleRollup.swift (extension Estate) +- :54 `rollupRoomsForDrawers(_ drawerIds:)` : coalesced per-room rollup for the deferred/off-write-path queue worker +- :71 `rollupMerkleRoots(roomNodeId:now:)` : room→wing→estate, 3-level bottom-up for ONE changed room +- :125 `computeRoomMerkleRoot(roomNodeId:)` : excludes tombstoned AND withdrawn(state=18) drawers from the snapshot (WS2-F1); reads content_hash column when present, else computes leaf on-demand +- :175/:186 `computeEstateOrWingMerkleRoot(parentNodeId:)`/`computeWingMerkleRoot(wingNodeId:)` : same function serves both levels +- :200 `recomputeAllMerkleRoots(now:)` : full bottom-up; :243 `rollupAllMerkleRoots(now:)` alias for batch-capture reindex (NT_R1) +- :265 `createSnapshot(label:now:additionalAttestations:)` : ALWAYS recomputes full tree first (capture defers rollup, so cached roots may be stale) +- :332 `deterministicUUID(from:)` (static) : SHA-256-derived stable UUID for non-UUID drawer ids (version/variant bits stamped) + +### Manifest : Manifest.swift +- :6 `enum ManifestKey: String, CaseIterable` : 18 required + 7 optional keys; :41 `.ed25519PublicKey` (safe, public) / :56 `.ed25519PrivateKeyWrapped` (NEVER written by current code : reserved read-compat seam only, real private key lives in EstateIdentityKeyStore) +- :80 `struct ManifestValues` : typed snapshot from DrawerStore.readManifest() + +### Default wings : DefaultWings.swift +- :18 `defaultWingName = "Agentic Memory"` (ADR-016, fixed : NOT owner-derived) / :26 `hintRoom = "AI_Charter_Hint"` / :32 `hintUDCCode = "001"` / :36 `hintAddedBy = "estate-provision"` +- :44 `struct WingDefinition` : name+hint / :64 `defaultWings: [WingDefinition]` : 7 seeded wings (suggestion, not enforced schema) + +### Recall stream : RecallStream.swift +- :21 `struct RecallStream: AsyncSequence, Sendable` : :26 `defaultPageSize = 50` +- :54 `degradedStages: [String]` : non-throwing failure channel (empty ⇒ genuine result incl. genuine-empty; non-empty ⇒ named internal-read/eval failure) +- :86 `struct RecallPage` : rows/pageIndex/isLast +- :92 `struct AsyncIterator: AsyncIteratorProtocol` : first page synchronous; :129 `hydrate(_:)` (private) rebuilds with content="" ONLY at .bitmapOnly + +### Recall trace : RecallTraceItem.swift +- :28 `flagUsed: Int64 = 1<<0` / :58 `used: Bool` (computed, no stored Bool : bitmap-backed pattern every noun follows) / :47 `score: Double?` (nil = no score, e.g. capture-time-ordered recall) + +### Audit types : AuditTypes.swift +- :26 `struct BitmapState` : rowID/asOf(HLC)/adjectiveBitmap/operationalBitmap/provenanceBitmap; LocusKit's own wrapper over AuditLogFold.projectStateAt's result shape + +### Summaries : Summaries.swift +- :10 `struct WingSummary` : name/drawerCount/roomCount / :33 `struct RoomSummary` : wing/name/drawerCount; BOTH are live computed projections, no dedicated wings/rooms table + +### Telemetry : LocusKitTelemetry.swift +- All functions `@inline(__always)`, `Intellectus.report(_:)` with @autoclosure (never evaluated when monitoring off : ~1ns off-path cost, no lock, no alloc) +- `now:` ALWAYS caller-supplied : never Date() inside this file (determinism contract) +- :50 `emitDrawerCapture` / :90 `emitDrawerQuery` / :123 `emitKGFactAdd` / :149 `emitKGFactQuery` / :175 `emitTunnelAdd` / :199 `emitGateAdmit` / :224 `emitGateReject`(reason truncated to 64 chars) +- Namespace: `locuskit..` + +## INVARIANTS / GOTCHAS + +- SOURCE OF TRUTH IS THE AUDIT LOG, NOT THE LIVE ROW. Every gated write (mutateState/mutateAdjective/mutateOperational/mutateProvenance/expungeGated/reanchorGated) updates the projection column AND appends the sealed AuditEvent inside ONE `storage.transaction(isolation: .serializable)`. Never split these two writes across transactions except via the documented sealAudit:false + sealExpungeAudit(_:) two-step (GLK §B-2a orchestration only). +- The state field (adjectiveBitmap bits 0-5) is VERB-DRIVEN ONLY. `gatedColumnWrite` (whole-column field-edit path) explicitly excludes it. State only ever moves through `mutateState`, which validates `(from, via, to)` against SubstrateLib.RowStateAutomaton BEFORE any write. +- DrawerStore.addDrawer is `internal`, not `public`. The only sanctioned add path is `Estate.addDrawerCovered` (private in EstateVerbs.swift), which bundles the row insert with `containerFP.orIn` : this is what makes "captured a drawer without updating its container fingerprint" structurally impossible. Tests reaching `store.addDrawer` directly via `@testable import` must document why fingerprint coverage is not needed for that path. +- ContainerFingerprintStore has NO clear side by design. Withdrawing/expunging a drawer never clears bits from the room/wing OR-aggregate : a stale set bit only causes an unnecessary scan, never a missed match. `rebuildRoom`/`rollUpWing`/`rebuildAll` periodically tighten; `Estate.open` calls `rebuildAll` once so the aggregate is sound from the first recall. +- Expunge's destruction contract is STRONGER than the state machine. In `expungeGated`, every lineage sibling has its content column unconditionally zeroed even when the gate refuses that sibling's state transition (e.g. an `accepted` row, forbidden from reaching `.tombstoned` by S-3). Leaving verbatim content behind because a state flip was refused would be a privacy violation. +- Tombstone exclusion in `BitmapEvaluator.evaluateBitmapTier` is UNCONDITIONAL and independent of the caller's filter chain : `state == 33` (raw State.tombstoned) never surfaces through recall no matter what filters are supplied. +- `recall(_:)` NEVER THROWS (spec §7.8.1). Internal read/eval failures are surfaced exclusively via `RecallStream.degradedStages` (named stage strings). An empty `rows` with empty `degradedStages` means genuinely no matches; an empty `rows` with a populated `degradedStages` means a failed internal read. +- PersistenceKit v1.0 has NO nested transactions. This is why `gatedCaptureBody` exists as a separable @Sendable closure builder : `addDrawerWithCascade` needs the drawer INSERT and the supersedes-tunnel INSERT in ONE outer transaction, and cannot call `gatedCapture` (which opens its own transaction) from inside another transaction. +- `findActivePredecessor` predicate is `g_state_cluster < 3`; `livingSuccessorInLineage` predicate is `g_state_cluster < RowState.activeClusterUpperBoundRaw (16)`. These are DIFFERENT thresholds for different purposes (supersession-cascade target vs revive-guard living-successor check) : do not conflate them. +- Corpus scans (`allDrawers`, `drawersIn`, etc.) use `decodeDrawerRowsSkipCorrupt`: a `.corruptStoredValue` on one row is logged and skipped, everything else re-thrown. Point lookups (`getDrawer(id:)`, `getDrawers(ids:)`) use the strict `decodeDrawerRows`, which throws on the first corrupt row. Do not swap these two decode paths between call sites. +- `allDrawers(hydrationLevel:limit:direction:)` sorts by compound key `(filedAt, id)`, NOT SQLite `rowid` : `id` is the declared TEXT primary key, present and portable across SQLite/PostgreSQL/InMemory. DESC is the exact byte-for-byte reverse of ASC for any fixed dataset because of this deterministic tie-break. +- `RecallFrame.traceLimit` defaults to `nil`, which writes ZERO recall-trace rows. Only the GLK RecallDirector's primary locus call sets a non-nil traceLimit. Any new caller of `recall(_:)` that leaves traceLimit nil is correct by default; do not add trace writes to a scan-style caller without deliberate reason (write amplification). +- Bitmap layout constants in `BitmapEvaluator.swift` (adjStateMask, provConfirmMask, opChannelMask, etc.) are a PRIVATE MIRROR of the accessor decoders in Adjectives.swift/Provenance.swift/DrawerOperational.swift. A schema bump (`bitmap_layout_version`) that moves a field must update BOTH the accessor and this mirror in the same change. +- `@guardian-pair` comments (Adjectives.swift, DrawerStore.swift mutateState/expungeGated) mark raw-value sets that MUST stay in sync with SubstrateLib's AuditGate.basis legalValues : a build-time Guardian tool checks these; GuardianPairParityTests is the CI backstop. Touching a State/Trust/AdjectiveSensitivity/AdjectiveExportability case requires checking every paired comment. +- Nine nouns share the three-Int64-bitmap pattern (adjective/operational/provenance) EXCEPT: `Tunnel`/`Association`/`LearnedReference` have no `provenance`-axis adjective sharing beyond the bitmap slot layout itself being noun-specific; `Node` has no bitmap columns at all (lifecycle is a plain Int, not a bitmap). `Association`, `Proposal`, `LearnedReference`, `SourceCatalogEntry` all require a non-empty `latticeAnchor.udcCode` at insert (I-16) : `KGFact` and `Tunnel` predate I-16 and carry no anchor requirement. +- `Association`/`Proposal`/`LearnedReference`/`SourceCatalogEntry` are `Equatable, Codable, Sendable` but deliberately NOT `Hashable` (the embedded `LatticeAnchor` isn't Hashable). Do not add `Hashable` conformance without also widening `LatticeAnchor` : nothing today keys a Set/Dictionary on any of these four types. +- `Node.normalizeLookupName(_:)` and `DrawerFingerprint`'s four-block derivation are CONFORMANCE-GATED: the Rust port must produce byte-identical output for identical input. Changing either requires mirroring the change in rust/src/ and passing the corresponding conformance test. +- Pinned/fixed constants : changing any requires a coordinated Swift+Rust update and, where noted, a schema version bump: `LocusKitSchema.version` (8), `Estate.expectedBitmapLayoutVersion` ("v1.0"), `EstateVerbs.recallCandidateCap` (256), `RecallStream.defaultPageSize` (50), `RecallTraceItem.flagUsed`/`Tunnel.isRetiredBit`(bit 13)/`Tunnel.isDreamedBit`(bit 0), `NodeStore` depth ceiling (2), `outlineAncestors` depth ceiling (256), `NodeBundleStore` count-vector width (256 × UInt32 = 1024 bytes), `defaultWingName`("Agentic Memory")/`hintRoom`/`hintUDCCode`/`hintAddedBy`. +- LocusKit ships NO pinned data artifacts (no Resources/ directory of the LatticeLib kind) : its schema is generated fresh from `LocusKitSchema.schema` at every open; the only external pinned artifact it consumes is LatticeLib's Q-ID closure snapshot, reached read-only via `QIDClosure.ancestors(of:)`. diff --git a/packages/kits/LocusKit/docs/DETAILS.md b/packages/kits/LocusKit/docs/DETAILS.md new file mode 100644 index 0000000..d60564c --- /dev/null +++ b/packages/kits/LocusKit/docs/DETAILS.md @@ -0,0 +1,1415 @@ +--- +doc: DETAILS +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# LocusKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in +pipeline order. First come the module surface and the estate +lifecycle. Next come the nine verbs and their frame types. Then comes +the storage engine. After that comes the drawer value type and its +shared bitmap machinery. Next comes the recall filter and its +evaluator. Then come the remaining eight noun types and their bitmap +decoders. After that comes the containment tree and the bundle +algebra. Next comes the fingerprint and integrity machinery. Then +comes the manifest and setup data. Next come the recall-result +and trace types. Telemetry comes last. + +## LocusKit.swift + +This file provides the module surface. It carries a doc comment +describing the package. That comment points to the files holding the +real public types. LocusKit has no module-level namespace enum of its +own, unlike some sibling libs. Every public type lives in its own +file. This file exists only to orient a reader arriving at the +package for the first time. + +## LocusKitError.swift + +This file provides `LocusKitError`, the single error enum every +LocusKit method throws. Centralizing every failure mode in one type +lets a caller recover in exact ways. For example, a caller can treat a +missing drawer as a routine miss. It can still propagate a genuine +SQLite failure. + +The cases split into three groups. The not-found group names a +missing row by id: `drawerNotFound`, `tunnelNotFound`, +`diaryEntryNotFound`, and `recallTraceItemNotFound`. The storage group +describes the storage layer misbehaving: `databaseUnavailable`, +`sqliteError`, `schemaTooNew`, and `corruptStoredValue`. +`corruptStoredValue` names the table, column, and raw stored text. So +a caller can locate and repair the offending row, rather than receive +a fabricated default value. The domain group describes LocusKit's own +validation refusing a write: `invalidContent`, `disciplineViolation`, +and `notSupported`. `invalidContent` carries a human-readable rule +violation. Tests assert on this message text, as a validation-test +contract. `disciplineViolation` names an illegal state transition or +forbidden bitmap combination. It uses the raw integer state values, +rather than the `State` enum. So the error type does not need to +depend on `Adjectives.swift`. + +`LocusKitError.description` renders each case as plain English, +without Swift's `EnumCase(...)` noise. GeniusLocusKit consumes this +description at its boundary. A caller there parses the English text +back out, to decide how to react to a rejected write. So the wording +is a soft contract, not just a log message. + +## LocusKitSchema.swift + +This file provides `LocusKitSchema`, the entire on-disk schema. The +schema is declared as data, not as SQL text. `Storage.open(schema:)`, +a PersistenceKit primitive, reads this declaration. It creates every +table, generated column, and index from it. No file in LocusKit +issues a raw `CREATE TABLE` or `CREATE INDEX` string. A hand-rolled +SQL implementation used to do this. This declarative approach +replaced it. So the schema is portable across SQLite, PostgreSQL, and +an in-memory test backend. It needs no three copies of the same DDL. + +`LocusKitSchema.schema` assembles seventeen tables. Fourteen of them +are: `drawers`, `tunnels`, `diary`, `manifest`, `kg_facts`, +`proposals`, `associations`, `learnedReferences`, `source_catalog`, +`node_bundles`, `container_fingerprints`, `recall_trace`, `keys`, and +`nodes`. Three more tables come from two sibling schema fragments: +`ErasureLedgerSchema` and `SnapshotSchema`. Every persistent noun +table carries a nullable `.json` column named `ext`. This column is a +forward-compatible extension slot. It absorbs an unforeseen future +field, without a schema migration. Version 1.0 writes `NULL` into it +and never reads it. `LocusKitSchema.version` is currently 8. The +file's header comment documents each version bump. Bumps include a +BLOB Merkle root, a nullable `content_hash` column, and the `nodes` +table, among others. The comment states plainly that there is no +incremental migration ladder. No estate data has shipped under an +earlier version. + +The file's most distinctive feature is its generated columns. +`g_state_cluster` on `drawers` is one example. It is computed as +`adjectiveBitmap & 0x3F`. A generated column is a column whose value +the backend derives on its own. The backend computes it from an +expression over other columns in the same row. So a generated column +can be indexed like an ordinary column, without the application +computing and storing it by hand. `LocusKitSchema.indices` then +declares an ordinary index on each generated column. This turns a +hand-written functional index into a declarative, backend-portable +statement. The old form looked like `CREATE INDEX ... ON drawers +(provenance & 0xF)`. The file's header comment also documents a +bitmap reservation map. This map shows which bit ranges of each Int64 +bitmap column are assigned, and which are free headroom. So adding a +future flag means claiming a documented free bit, not running a +schema migration. + +## LocusKitVocabulary.swift + +This file provides `LocusKitVocabulary`, LocusKit's contribution to +the write gate's vocabulary. A vocabulary here is the set of legal +values a bitmap field may take. It is the fixed rulebook the write +gate checks every proposed value against. A value outside the +vocabulary is refused before it ever reaches storage. + +`LocusKitVocabulary.unionSlots` declares one `FieldSlot` per +operational and provenance field LocusKit defines. These fields +include capture channel, content kind, feature flags, and source +type. Each slot names its bitmap column, bit position, and width. An +enumerated field's slot also lists its legal raw values. The +adjective axis's vocabulary covers state, sensitivity, exportability, +and trust. The write gate itself supplies that vocabulary, from +SubstrateLib, not this file. `unionSlots` is purely LocusKit's own +consumer-side addition. `LocusKitVocabulary.frozen()` compiles this +set into a `Vocabulary` value once, at estate open, through +`VocabularyValidator.freeze`. Every later write in the estate's +lifetime is checked against this frozen vocabulary. It never changes +after that first freeze. + +## EstateTypes.swift + +This file provides five small supporting types used across the +estate surface. `RowID` is a plain `String` type alias. It names a +row's stable identifier. `FrameFilteredDrawers` is the result of a +frame-aware by-id load. It pairs the frame-admissible drawers with +the full set of ids that physically loaded. `OwnerCredentials` is the +estate owner's identifier. It is validated non-empty at open or +create time. `LatticeAnchor` is the four-field classification anchor +every drawer carries. It holds a UDC code, plus optional Wikidata +enrichment. `EstateError` is the error enum for `Estate` lifecycle +failures, distinct from `LocusKitError`. + +`FrameFilteredDrawers.loadedIDs` is reported apart from +`admissible`, for a specific reason. A caller needs to tell two cases +apart. The first case is an id that loaded but failed the frame's +filter, a legitimate drop. The second case is an id that never +loaded, a transient or partial read that must be treated as degraded, +not silently dropped. `LatticeAnchor.udc(_:)` is a convenience +constructor. It handles the common case of a bare UDC code with no +Wikidata enrichment, the shape most content has before an enrichment +daemon has run over it. `EstateError` covers three failure shapes. +`substrateUnavailable` means the backing storage failed to open. +`manifestMismatch` means a stored manifest value does not match what +this build expects. Most importantly, the bitmap layout version must +match. `emptyOwnerIdentifier` is raised before any storage call. So a +caller gets a structurally distinct error, rather than a generic one. +`keychainError` covers a failing Keychain call during identity key +handling. + +## Estate.swift + +This file provides `Estate`, the single public entry point to a +LocusKit-backed memory store. An `Estate` is an actor. An actor is a +Swift type that serializes access to its own state. So concurrent +callers can never interleave two operations against the same estate +unsafely. `Estate` owns one `DrawerStore`, the storage engine +described below. It owns one `ContainerFingerprintStore`, the +recall-pruning totals. It owns one `NodeStore`, the containment +tree. It is the only public surface most callers should ever touch. + +`Estate.open(storage:owner:identityKeyStore:)` opens an existing +estate. It refuses to open a database whose manifest carries a +different `bitmap_layout_version` than this build expects. Bitmap bit +positions are part of the durable on-disk contract. So opening a +database written by an incompatible future schema would silently +misread every bitmap field. On first open, it also mints a fresh +Curve25519 Ed25519 keypair for the estate's federation identity. The +public half is written to the manifest, safe because a public key has +no confidentiality requirement. The private half is handed to an +injected `EstateIdentityKeyStore`, the Keychain in production, an +in-memory store in tests. It is cached for the lifetime of the +`Estate` instance. `Estate.create(storage:owner:manifest:)` is the +sibling constructor for a brand-new estate. It does not mint the +identity keypair, because that only happens on the first `open` after +creation. + +The bulk of the file is a set of typed pass-through reads. These +exist so higher kits, chiefly GeniusLocusKit, never need to import or +construct a `DrawerStore`. They include `allDrawers`, +`getDrawers(ids:)`, `hydrateBodies`, `tunnelsFromWing`, +`recentRecallTraces`, `allTunnels`, `retireTunnel`, +`pruneRecallTraces`, `countDrawerRows`, `allProposals`, +`allAssociations`, `allLearnedReferences`, `allKGFacts`, +`allDiaryEntries`, and `resolveNodeNames`. +`getDrawers(ids:matchingFrame:hydrationLevel:)` is the most elaborate +of these. It runs the exact same frame-filtering pipeline `recall` +uses, but over a caller-supplied id set rather than a full corpus +scan. This is the shape a dense-first candidate-pool search needs. +`meta(key:)` and `setMeta(key:value:)` expose the estate manifest as +a general per-estate key-value store. Higher kits can use it for +durable state of their own, on the condition that they namespace +their keys to avoid colliding with LocusKit's own typed manifest +keys. + +## EstateAudit.swift + +This file provides `Estate`'s audit and history API: +`auditTrail(rowID:)` and `bitmapState(rowID:asOf:)`. Both delegate to +the row's sealed audit log, rather than to any separate history +table. The audit log, not the live row, is the estate's source of +truth for what happened and when. + +`auditTrail(rowID:)` returns every sealed `AuditEvent` for a row in +HLC order. An HLC, or hybrid logical clock, is a timestamp format. It +puts events in one order, even across machines with their own +clocks. `bitmapState(rowID:asOf:)` rebuilds what a row's three +bitmaps looked like at an earlier point in time. It does this by +folding the row's audit log forward, from the beginning up to and +including the requested HLC. This folding runs through SubstrateLib's +`AuditLogFold.projectStateAt`. This is what lets a caller ask what a +drawer's state looked like a week ago. LocusKit needs no separate +append-only history table alongside the live row for this. The live +row is redundant with the log by construction. The log is the only +place that needs to remember the past. + +## EstateIdentityKeyStore.swift + +This file provides the protocol and two implementations for +persisting an estate's Ed25519 private signing key outside the +manifest table. The manifest table is ordinary, unencrypted metadata. +Anyone with database or backup access can read it. So the private +half of the estate's federation identity keypair must never be +written there. Only the public half is safe to store in the +manifest. + +`EstateIdentityKeyStore` declares two methods: `loadPrivateKey` and +`storePrivateKey`. Both are keyed by the estate's UUID. +`KeychainEstateIdentityKeyStore` is the production implementation. It +stores the raw key bytes as a `kSecClassGenericPassword` Keychain +item. The item stays accessible after the first device unlock +following a restart. It is never synced to iCloud Keychain. This +matches the same device-bound posture as the estate's own SQLite +file. `InMemoryEstateIdentityKeyStore` is the test double. It is a +plain dictionary guarded by a lock. It carries a `_storedPrivateKey` +accessor. A test can use this accessor to check what was persisted, +without touching the real Keychain. The real Keychain would require +entitlements. It would also pollute state across test runs. + +## EstateVerbs.swift + +This file provides the nine verb methods as an extension on `Estate`: +`capture`, `captureBatch`, `recall`, `withdraw`, `expunge`, `mutate`, +`reanchor`, `propose`, `learn`, and `associate`. It is declared as a +separate file for a specific reason. It needs to reach `Estate.store`, +which is declared `internal` rather than `private` in `Estate.swift` +for exactly this reason. + +`capture(_:CaptureFrame)` is the drawer entry point. It validates +that content and room are both non-empty. It also validates the +lattice-anchor UDC code, the actor, and the embedding-model id. It +assembles the three bitmap +columns from the frame's named fields, through `BitField.writeField`, +never hand-rolled shift-and-mask arithmetic. It resolves the target +wing and room to containment-tree node ids, creating them on demand. +It writes the drawer through `addDrawerCovered`, the one sanctioned +internal chokepoint. That chokepoint bundles the row insert with the +container-fingerprint update. So a drawer can never be captured +without its fingerprint total being updated in the same call. +`captureBatch(_:)` is the bulk-import sibling. It resolves all wing +and room node ids up front, with a per-call cache. It batches every +fresh, non-superseding insert into one storage transaction, through +`DrawerStore.insertFreshBatch`. It defers the Merkle rollup entirely. +This turns a 40,000-drawer import from roughly 34 minutes of per-row +commits into roughly 30 seconds. + +`capture(_:TunnelCaptureFrame)` is the tunnel entry point. `capture` +is legal on exactly two nouns, drawer and tunnel, and this overload +handles the second. `recall(_:RecallFrame)` is the read path. It +reads the live, non-tombstoned, candidate rows through `liveRows`. It +runs them through `BitmapEvaluator.evaluate`. It optionally writes a +bounded number of recall-trace rows, only when the caller opts in +through `frame.traceLimit`. So internal and bulk-export scans never +accumulate trace rows. `recall` returns a `RecallStream`. Both +`liveRows` and the evaluation step surface any internal read failure +as a named degraded stage on the returned stream, rather than +silently returning an empty result. This is the mechanism by which a +caller can tell a genuinely empty estate from a recall that failed +partway through. + +`withdraw(rowID:reason:)` moves a drawer's state to `.withdrawn`, +through `mutateState`. This is the only correct path, because a state +transition must go through the automaton's legality check. +`expunge(rowID:reason:confirmation:now:)` is the destructive delete. +It requires an explicit `confirmation: true`. It walks the drawer's +full lineage chain, every version sharing its `lineageID`, so a hard +delete scrubs every historical version. It always seals a sealed +audit event recording the fact of the expunge, even though the +content itself is gone. `expungeReturningUnsealedEvent` and the +paired `sealExpungeAudit` and `sealExpungeOrphanAudit` methods exist +for GeniusLocusKit's two-step orchestration. There, a cross-kit +vector delete must happen between the storage tombstone and the +audit seal. Splitting the seal out prevents any caller from +suppressing the audit event by accident. + +`mutate(rowID:kind:payload:)` dispatches on `MutationKind` to move a +drawer along one of three axes. The first axis is confirmation, +through `mutateProvenance`. The second is state, through +`mutateState`. The caller-facing guards for `.resolve`, `.accept`, +and `.revive` are implemented here. This happens before the store's +automaton check even runs. So the error message is clearer. The third axis is +an adjective field, sensitivity, trust, or exportability, through +`mutateAdjective`. The `.revive` case is the most elaborate. It +implements the full per-source-state legality table from the +cookbook's revive rule. This includes the living-successor lineage +check. That check refuses to revive a superseded row while a later +version in the same lineage is still alive. +`reanchor(rowID:toRoom:toWing:toLattice:)` moves a drawer's room, +wing, and lattice position, without touching its bitmaps. +`propose(_:now:)` and `associate(_:now:)` build a `Proposal` or +`Association` from their respective frames. Each derives its lattice +anchor from a specific source. For propose, the anchor comes from +the target row. For associate, it comes from the first endpoint. +Neither ever fabricates an anchor. `learn(_:now:)` +catalogs the frame's source, once, keyed by handle. It writes a +`LearnedReference` anchored to that catalog entry's genuine lattice +position, never a sentinel anchor invented from a bare handle. + +`seedWing(_:hint:addedBy:embeddingModelID:now:)` files one ordinary, +fully recallable drawer per default wing, at estate provision time. +This gives a fresh agent a plain-language description of what each +wing is for. + +## Frames.swift + +This file provides the frame structs that carry a verb's arguments: +`CaptureFrame`, `TunnelCaptureFrame`, `RecallFrame`, `LearnFrame`, +`ProposeFrame`, and `AssociateFrame`. It also provides three +supporting enums: `MutationKind`, `HydrationLevel`, and `Ordering`. +Every field on +every frame is named after a domain concern, a capture channel, a +sensitivity tier, a filter chain. No raw bitmap value or bit position +crosses the public verb boundary anywhere in this file. + +`CaptureFrame` carries every named axis a captured drawer's three +bitmaps encode: channel, sensitivity, kind, and the matching +provenance-side channel, source type, sensitivity, confirmation, and +confidence. It also carries lineage, room, lattice anchor, actor, and +embedding model. It carries an optional event time too, for backdated +bulk ingestion. It also carries feature flags, exportability, and an +optional wing. Nearly every field defaults to the value that produced +the same all-zero bitmap a caller got before that field existed. So +adding a new capture-time axis stays source-compatible with every +existing caller. `RecallFrame` carries the filter chain, hydration +level, page-size limit, and ordering. It also carries an optional +past `asOf` HLC and `traceLimit`. +`traceLimit` is the opt-in recall-trace write count. + +`MutationKind` enumerates the mutate verb's seven cases: `.confirm`, +`.reject`, `.contest`, `.resolve`, `.supersede`, `.revive`, and +`.accept`. Three more cases carry a new value: `.correctSensitivity`, +`.correctTrust`, and `.correctExportability`. `HydrationLevel` names +the three read-cost tiers: `.bitmapOnly`, `.structured`, and `.full`. +`Ordering` names the three result orderings recall supports. A +relevance ordering is absent on purpose. LocusKit has no scoring +signal of its own. That signal lives in VectorKit, composed on top by +GeniusLocusKit. The file's comment explains that shipping a +`.byRelevanceDesc` case here, without a real relevance signal, would +be an honesty violation. + +## DrawerStore.swift + +This file provides `DrawerStore`, the actor that owns every table in +the schema. It implements the full CRUD and mutation surface for all +nine nouns. It is the largest file in the package. Almost every write +in the estate lands here in the end. + +`DrawerStore.init(storage:hlc:)` opens the schema. This is +idempotent: a second open on an existing database is a no-op for +tables and indices. It freezes the write-gate vocabulary once. It +populates the v1 manifest defaults, writing each key only if absent, +so a first-open value is never overwritten by a later open. It +classifies the manifest's `estate_uuid` value into one of three +outcomes. The first outcome is absent: a genuinely fresh estate, +resolved to a freshly minted UUID. The second outcome is present and +well-formed, the normal case. The third outcome is present but +unparseable. This third outcome throws `corruptStoredValue`, rather +than silently fabricating a random replacement UUID, because +conflating corruption with a fresh estate would mask real data loss. +The HLC clock's node id is derived from that same classified value, +by hashing the raw stored text. So a standalone estate gets a stable, +estate-specific clock identity. A Rust port of the same estate +derives that identity identically. + +The gated write path is the file's central pattern. It is used by +`addDrawer`, `mutateProvenance`, `mutateAdjective`, +`mutateOperational`, `mutateState`, `expungeGated`, and +`reanchorGated`. Each of these reads the row's current bitmaps inside +a serializable transaction. Each decomposes the proposed new value +into per-field `FieldWrite`s, never a whole-column replacement. This +is what lets the gate validate each field on its own. It also lets +the gate check the forbidden-combination invariant on the merged +result. Each calls `AuditGate.admit`, a SubstrateLib primitive. On +success, each writes both the merged projection column and the +sealed `AuditEvent`, in the same transaction. `gatedColumnWrite` is +the shared helper behind the three whole-column mutators. It excludes +the adjective bitmap's state field from the field list, because +state is verb-driven and must never move through a field edit. +`mutateState` also checks the verb against the caller's requested +target state. Both must agree with what the automaton's transition +table produces. The verb itself is a `RowVerb` value. So an illegal +from-via-to triple is rejected before any row is touched. + +`addDrawer` implements the supersession cascade. Sometimes the new +drawer's `lineageID` matches an active predecessor. +`findActivePredecessor` finds this match, through an indexed query on +the generated state-cluster column. When a match exists, +`addDrawerWithCascade` inserts the successor and a `.supersedes` +tunnel in one transaction. So a failed tunnel insert never leaves an +orphaned successor row. `addDrawerWithCascade` then separately flips +the predecessor's state to `.superseded`, through the normal gated +`mutateState` path. `insertFreshBatch` is the bulk-import sibling used +by `EstateVerbs.captureBatch`. Every drawer in the batch already has +no active predecessor. The caller pre-verifies this before the batch +runs. Each drawer is inserted and gated inside one single +transaction. This eliminates per-row commit overhead. + +`expungeGated` is the lineage-wide hard delete. It tombstones the +target drawer through the gate. It zeroes the drawer's content. It +sets the `dreaming_recalc_required` obligation flag. It records the +event in the erasure ledger. It then repeats a content scrub across +every other member of the lineage chain. Where the state transition +is legal, it also repeats a full gated tombstone. A sibling whose +state cannot legally transition to tombstoned, an audit-grade +`accepted` row forbidden by invariant S-3, still has its content +zeroed no matter what. The destruction contract is stronger than the +state machine. Leaving verbatim content behind, when the gate refuses +the state flip, would be a privacy violation. `sealExpungeAudit`, +`sealExpungeOrphanAudit`, and `sealExpungeOrphanForSweep` support +GeniusLocusKit's two-step orphan recovery. This recovery matters when +a crash separates the storage tombstone from its audit seal. +`tombstonedRowsWithoutExpungeAudit` is the query the recovery sweep +uses to find rows stuck in that crash window. + +The remainder of the file provides CRUD for the other eight nouns. +Tunnel CRUD includes `addTunnel`, `getTunnel`, `tunnelsFrom`, +`tunnelsTo`, and `allTunnels`. It also includes the tunnel-retirement +pair `retireTunnel` and `unretireTunnel`. It includes the outline +helpers `outlineChildren`, `outlineAncestors`, and `reparentDrawer`. +Fact CRUD +includes `addKGFact`, `getKGFact`, `kgFacts(forDrawerID:)`, and +`withdrawKGFact`. Proposal CRUD includes `addProposal`, `getProposal`, +and `proposals(forTargetRowID:)`. Association CRUD includes +`addAssociation`, `getAssociation`, `associationsFrom`, and +`associationsTo`. Learned-reference CRUD includes +`addLearnedReference`, `getLearnedReference`, and +`learnedReferences(forSourceCatalogID:)`. Source-catalog CRUD includes +`addSourceCatalogEntry`, `getSourceCatalogEntry`, and +`sourceCatalogEntry(forHandle:)`. Diary CRUD includes `addDiaryEntry`, +`getDiaryEntry`, and `readDiary`. The file also provides the +recall-trace surface: `insertRecallTrace`, `insertRecallTraces`, +`recentRecallTraces`, `markRecallTraceUsed`, `markRecallTracesUsed`, +`pruneRecallTraces`, and `countRecallTraces`. It provides the +manifest surface: `setMeta`, `getMeta`, and `readManifest`. It +provides the node-tree name-resolution helpers: `roomNodeIdsInWing`, +`roomNodeId`, and `resolveNodeNames`. It provides the summary +surface: `listWings`, `listRooms`, and `taxonomy`. It provides the +temporal-read surface: `fingerprintsCaptured(in:)` and +`fingerprintBitSeries`. These feed the FFT-based rhythm-spectrum and +moment-summary lenses built on top of LocusKit. + +Every row-decode function follows the same resilience pattern. +`drawerFromRow` and its siblings throw `corruptStoredValue` for a +clearly malformed stored value, such as a non-UUID `lineageID` +or an unparseable timestamp. Corpus-scan callers instead route +through `decodeDrawerRowsSkipCorrupt`. This function logs and skips a +corrupt row, rather than aborting an entire estate-wide scan. Point +lookups fail loud. Corpus scans degrade gracefully. + +## Drawer.swift + +This file provides `Drawer`, the value type behind verbatim content. +A `Drawer` is the thing callers file, and the thing recall returns. +Its `content` field is preserved unchanged, with no truncation and no +normalization. MemPalace's verbatim-first principle requires that +retrieval surface exactly what was filed. + +Every drawer references its containing room by `parentNodeId`, a +foreign key into the `nodes` table, rather than storing wing and room +names. Display names are resolved separately, through +`DrawerStore.resolveNodeNames`. `filedAt` and `eventTime` are two +distinct clocks. `filedAt` is the ingest clock. It marks when the row +entered the local store. It is monotonic, and it anchors audit +ordering. `eventTime` is when the content happened, or was +authored, in the world. For streaming capture the two clocks +coincide. A bulk backfill importer instead supplies a real, +possibly much earlier, `eventTime`. The three bitmap fields, +`provenance`, `adjectiveBitmap`, and `operationalBitmap`, all default +to zero. The four lattice fields are `udcCode`, `udcFacets`, +`wikidataQID`, and `wikidataQidsSecondary`. They locate the drawer in +the classification lattice. `udcCode` is required non-empty at every +capture path, per invariant I-5. `Drawer`'s custom `Codable` +conformance backfills a missing `eventTime` to `filedAt` on decode. So +a row encoded before the `eventTime` column existed still decodes to +a sensible value. + +## DrawerFingerprint.swift + +This file provides the derivation of a drawer's 256-bit structural +fingerprint. This fingerprint is the coordinate system LocusKit uses +for structural similarity and for recall pruning. A fingerprint is +built from four 64-bit SimHash blocks. Each block is a projection of +one facet of the row, through its own family of random hyperplanes. +The first block is bitmap: the three bitmap columns. The second is +lattice: UDC prefix, direct Q-ID, and Q-ID closure. The third is +lineage-and-temporal: lineage hash and capture week. The fourth is +channel-and-source: channel, source type, capture channel, +sensitivity, and an estate identity hash. + +`EstateFingerprintFamilies` derives the four hyperplane families for +one estate from its UUID, once, at estate open. Two replicas of the +same estate, started on their own, still share the same UUID. So +they always derive the same families. So they derive the same +fingerprint for the same drawer content. This determinism property +is what recall pruning depends on. `EstateFingerprintFamilies.fingerprint(of:)` +is the per-drawer entry point. It resolves the drawer's Wikidata +Q-ID, if the drawer has one, through `LatticeLib.QIDClosure`. This +gives the drawer's full ancestor chain. It folds that chain into a +16-bit hash for the lattice block's `qidClosureHash` slot. It hands +all four assembled inputs to `SubstrateLib.SimHash`. Some fields do +not apply to a drawer: lineage-clustering, defer-pattern, and +stream-source data. These fields only apply to a different noun type, +`AmbientSample`. For a drawer, they default to a null value of zero, +per invariant I-17. This default keeps Hamming distance well-defined +across every noun type that shares the same fingerprint shape. + +## DrawerOperational.swift + +This file provides the three named axes packed into +`Drawer.operationalBitmap`. The first is `CaptureChannel`: how the +content entered the system. Its values are typed, voiced, OCR, +imported file, sensor, and actuator. The second is `ContentKind`: the +shape of the content. Its values are prose, code, transcript, list, +structured JSON, image caption, and fingerprint-only. The third is +`DrawerFeatureFlags`: a non-exclusive bitset of seven named flags. The +flags are attachments, voice, image, links, pinned, keystone, and +locked-zone. Feature flags form a bitset, not an exclusive choice. So +`DrawerFeatureFlags` is declared as an `OptionSet`. The other two axes +in this file are declared as enums instead. + +Each axis has a computed accessor on `Drawer`: `captureChannel`, +`contentKind`, `featureFlags`, `hasFeatureFlag(_:)`, +`stateExtensionActive`, and `lineageClusteringActive`. Each accessor +decodes the relevant bit range. Each falls back to a safe neutral +default, typed input or prose content, for any raw value this build +does not recognize. So a row written by a future version, with a +case this build has never heard of, degrades gracefully instead of +crashing. + +## DrawerStateValidator.swift + +This file provides `DrawerStateValidator`, a thin bridge from +LocusKit's `State` enum to SubstrateLib's canonical +`RowStateAutomaton`. This automaton is the single implementation of +the row-state legal-transition table. Every LocusKit noun with a +state axis is validated against it. `DrawerStateValidator` used to +carry its own parallel transition table. That table permitted four +transitions the specification actually forbids, including `accepted` +to `tombstoned`. So it was retired, in favor of using +SubstrateLib itself. + +`validate(from:to:via:)` is the transition-legality-only overload, +used by `StateTransitionTests`. It bridges `State` to `RowState`. It +looks up the legal target for the from-and-via pair in the automaton. +It throws `LocusKitError.disciplineViolation` when a check fails, +naming the rule violated. A check fails in either of two ways. The +from-via pair may have no legal transition. Or the caller's requested +target may disagree with what the automaton actually produces. +`validate(from:to:via:targetingFields:)` is the overload +`DrawerStore.mutateState` actually calls. It also checks +field-level invariants against the post-write bitmap fields, in the +same call. One such invariant, S-1, requires that an accepted row +have trust at or above canonical. Other invariants cover further +bitmap-combination rules, such as S-2. + +## Adjectives.swift + +This file provides the four cross-noun adjective axes: `State`, +`Trust`, `AdjectiveSensitivity`, and `AdjectiveExportability`. It also +provides their computed accessors on `Drawer`. These four enums are +the single source of truth for these axes, across every LocusKit +consumer, including GeniusLocusKit and NeuronKit. Layers beneath +LocusKit cannot import them, because the dependency graph points the +other way. So those layers carry the raw integer encoding instead. A +build-time Guardian tool checks that the two sides never silently +drift apart. + +`State` has ten cases. They partition into three clusters, at +raw-value boundaries sixteen and thirty-two. These boundaries let a +state's cluster be computed as one shift-and-mask operation. Cluster +A is "currently believed": active, pending, contested, and accepted. +Cluster B is "knew past": superseded, decayed, withdrawn, and +expired. Cluster C is terminal: rejected and tombstoned. +`Drawer.isCurrentlyBelieved`, `isKnewPast`, and `isTerminal` expose +these three clusters as predicates. + +`Trust` orders seven levels. The lowest is `.verbatim`, unqualified +as-filed content. Levels rise through `.canonical` up to the highest, +`.ambient`. It conforms to +`Comparable`. So a filter like "trust at least canonical" reads as an +ordinary comparison, rather than raw-value arithmetic. +`AdjectiveSensitivity` and `AdjectiveExportability` are both +scale-gapped. Their case raw values skip numbers, leaving room for +future intermediate tiers without disturbing any existing bitmask. +`AdjectiveSensitivity` carries three `ADR-007`-driven predicates: +`isBulkExportable`, `requiresOwnerKeyForBulk`, and +`isExcludedFromBulk`. These are the enforcement hooks VaultKit's +bulk-export path consults, to decide whether a drawer may ride a bulk +channel without additional friction. `Drawer.dreamingRecalcRequired` +and `Drawer.sealed` decode two single-bit obligation and trust-hint +flags, living above the four main axes. + +## Provenance.swift + +This file provides the seven named axes packed into +`Drawer.provenance`: `SourceType`, `Channel`, a mirrored +`CaptureChannel` reference, `Confirmation`, `Confidence`, +`Sensitivity`, and `EnrichmentStatus`. It also provides their +computed accessors on `Drawer`. The adjective bitmap records a row's +current standing. The operational bitmap records mechanical facts +about the content. The provenance bitmap records something +different: how the row came into being, and how it has been reviewed +since. + +`SourceType` names ten origins: user, observed, imported, canonical, +derived, federation-aggregate, tier-aggregate, paired-estate, +ambient, and actuator. `Channel` names the system surface content +arrived through: UI typed input, UI voiced input, an MCP agent, a +file import, federation, or a dreaming-daemon channel. `Confirmation` +is the review axis. Its levels run from unconfirmed through +user-confirmed, automated-confirmed, peer-confirmed, and +actuator-confirmed. `isUserConfirmed` is the +convenience predicate retrieval layers use, to surface only +user-vetted content. `Confidence` and `Sensitivity` are both +scale-gapped. `Confidence` orders from `.null` to `.verified`. +`Sensitivity` mirrors the raw values of +`AdjectiveSensitivity` on purpose. So the two axes can be compared directly: +sensitivity at capture, versus the estate's current access posture, +which may since have been mutated. `EnrichmentStatus` tracks a Q-ID +enrichment daemon's lifecycle for a row. It usually ends at +`.qidCompleted`. When deterministic re-inference fails instead, it +ends at the terminal in-workflow state `.qidProposed`. + +## BitmapOps.swift + +This file provides three small, `@inlinable` field-extraction +helpers: `andMask`, `thresholdCompare`, and `shiftExtract`. +`BitmapEvaluator` uses them to translate a `Filter` case into a check +against a packed Int64 bitmap. These helpers are not math primitives. +They carry no algorithm to prove. They need no platform-specific +optimization to gate. The substrate primitives are different: SimHash, +Hamming distance, and OR-reduction live in SubstrateLib. Those +primitives participate in the cross-platform conformance gate. Each +function here delegates its actual bit manipulation to a SubstrateLib +primitive: `BitField.maskedEquals`, `BitField.extractField`, or +`BitField.popcount`. So LocusKit never hand-rolls a shift-and-mask +sequence that could silently diverge from the Rust port's +equivalent. + +`andMask(_:mask:expected:)` tests whether a field equals an expected, +already-shifted value, in one masked comparison. +`thresholdCompare(_:mask:shift:op:value:)` extracts a field. It +compares the field against a threshold. The comparison uses one of +three orderings: `.lessThan`, `.lessThanOrEqual`, and +`.greaterThanOrEqual`. This is +the mechanism behind cluster-boundary filters, such as "trust below +the action threshold." `shiftExtract(_:shift:mask:)` returns a +field's raw integer value. It is used wherever the evaluator needs +the value itself, rather than a yes-or-no comparison. + +## ForbiddenCombinationValidator.swift + +This file provides `ForbiddenCombinationValidator`. It documents one +core rule: a forbidden bitmap combination. Where a live +call site exists, it also enforces that rule. The rule: a row must +never carry adjective sensitivity `.secret` together with adjective +exportability `.public_`. Storage can physically represent that +combination. Nothing stops the bits from being set. The verb layer +still must refuse to produce it. Current write paths route this +check through `AuditGate` and SubstrateLib. So this +validator itself has no live call site today. It exists as an +explicit, reviewable statement of the rule. It stays ready to be +wired in, wherever a future write path bypasses the gate. + +`validate(_:)` extracts two fields from a full `adjectiveBitmap` +value. The sensitivity field spans bits six through eleven. The +exportability field spans bits twelve through seventeen. It throws +`LocusKitError.disciplineViolation` if both match their forbidden raw +values, 48 and 32. These raw values are hand-derived from the enum +cases' shipped values, rather than imported. This is deliberate, so a +future rename of an enum case cannot silently change what this +validator checks. + +## Filter.swift + +This file provides `Filter`, the named recall-filter algebra. It also +provides five small supporting type aliases: `LineageID`, `RoomID`, +`WingID`, `WikidataQID`, and `ProvenanceChannel`, `FeatureFlag`, plus +the `StateCluster` enum. No `Filter` case takes a raw bit position, +mask, or threshold integer. Every case names a domain concern, +such as `.trustworthy`, `.inRoom(_:)`, or +`.contentMatches(_:)`. `BitmapEvaluator` is the sole place that +translates a `Filter` into the bitmap primitives internally. + +The cases group by concern. State queries include +`.currentlyBelieve`, `.usedToBelieve`, `.knewOnceAndErased`, +`.state(_:)`, and `.stateInCluster(_:)`. Trust queries include +`.trustworthy`, `.requiresConfirmation`, `.trust(_:)`, and +`.trustAtMost(_:)`. Other groups exist too. Sensitivity and +exportability queries form one group. Provenance queries form +another: confirmation, source type, channel, and confidence. +Operational queries form a third: capture channel, content kind, and +feature flags. Structural queries cover room, wing, lineage, time +bounds, lattice anchor and prefix, and Wikidata concept. One content +query, `.contentMatches(_:)`, searches text. Three +composition cases let filters nest arbitrarily: `.all`, `.any`, and +`.not`. A `RecallFrame.filterChain` is a plain array of `Filter` +values. It is interpreted as an implicit `.all`. Every filter in the +chain must pass. + +## BitmapEvaluator.swift + +This file provides `BitmapEvaluator`, the compiler and interpreter +for a `RecallFrame.filterChain`. This is the heart of the recall +pipeline. It is declared `internal` rather than `public`, because it +takes a `DrawerStore` argument, itself internal. Public callers reach +it only indirectly, through `Estate.recall`. + +`evaluate(frame:drawers:store:nodeNames:)` runs four stages. These +stages run against the candidate row set its caller has already +narrowed down, through `Estate.liveRows`, which applies fingerprint +pruning ahead of this call. The first stage is default insertion. It +prepends implicit filters for any concern the caller left +unconstrained. State defaults to `.currentlyBelieve`. Trust defaults +to `.trustworthy`. Sensitivity defaults to +`.sensitivityAtMost(.elevated)`, the Normal-tier ceiling per ADR-007. +So recall with an empty filter chain returns only content by default: +currently believed, trustworthy, and normal-or-elevated in +sensitivity. A caller need not spell any of this out. The second +stage is bitmap-tier evaluation. It compiles each `Filter` case to a +check over the row's three raw bitmaps, using the `BitmapOps.swift` +primitives. Tombstone exclusion is always enforced here, apart from +the caller's chain. A tombstoned row can never surface +through recall, no matter what the chain says. When `frame.asOf` is +set, each row's bitmaps are first rebuilt as they stood at that +past HLC. This rebuild runs through +`AuditLogFold.projectStateAt`, before the bitmap tier runs. The third +stage is structured-tier evaluation. It applies filters that need the +drawer's non-bitmap fields: room, wing, lineage, time bounds, and +lattice. For room and wing filters, it also needs a resolved +node-name lookup. The fourth stage is content-tier evaluation. It +applies `.contentMatches`, through a case-insensitive substring +search over the verbatim content. A final ordering pass then sorts +the survivors. + +`chainHasPrunableFilter`, `chainHasContentPredicate`, and +`chainHasStructuredNameFilter` are classifiers. `Estate.liveRows` and +`Estate.getDrawers(ids:matchingFrame:hydrationLevel:)` consult them +before even fetching rows. The first classifier decides whether +fingerprint pruning can apply at all. The second decides whether the +fetch needs to pay for the content blob. The third decides whether a +node-name lookup is needed. `containerSurvives(chain:fingerprint:)` +is the actual pruning predicate. It returns false only when the chain +provably cannot be satisfied by any row in a container. This is sound +only for set-bit filters, such as `.hasFeatureFlag`. A threshold or +exact-value filter can never be decided from an OR-reduced fingerprint +alone. So those filters never prune a container. They only ever get +evaluated per row. + +## Tunnel.swift + +This file provides `Tunnel`, the typed cross-reference between two +locations in the estate. A tunnel is stored directionally: from a +source endpoint to a target endpoint. Each endpoint carries a wing and a +room. Each endpoint also carries an optional specific drawer id. A +`nil` id means the room itself is the endpoint. So a query asking +what this side knows about never needs to scan both directions. +`kind` is a `TunnelKind`. This is the closed, indexed relationship +vocabulary the retrieval layer dispatches on. `label` is +a free-form, unvalidated human annotation. `orderKey` is a +fractional-index sibling ordering value. It is used only by +`.parent` tunnels in the outline graph, per ADR-017. Siblings under +the same parent sort by ascending `orderKey`. So no sibling needs to +be rewritten when a new one is inserted between two existing +siblings. + +## TunnelOperational.swift + +This file provides `TunnelKind`, the ten-case closed relationship +vocabulary. Its cases are: `.supersedes`, `.references`, `.blocks`, +`.validates`, `.contradicts`, `.derivesFrom`, `.covers`, +`.elaborates`, `.respondsTo`, and `.parent`, the outline-containment +edge. This file also provides the four operational axes packed into +`Tunnel.operationalBitmap`: `TunnelDirection`, `TunnelLifecycle`, +`TunnelOriginClass`, and `TunnelStrength`, plus their accessors. + +Two bits above those four axes carry standalone boolean flags, +rather than enumerated fields. `hasInverse` records whether a paired +reverse tunnel exists. `isRetired` is bit 13. The T13 mechanism, per +ADR-021, added this bit for dreaming retirement. `withRetired()` and +`withUnretired()` return a copy of a tunnel with that bit flipped. +This is the reversible mechanism by which the dreaming pipeline's +OMEGA cycle suspends a tunnel from active reads, without tombstoning +it. So a later co-recall can re-propose the same pairing. A separate +provenance-bitmap section adds `isDreamed`, bit 0 of +`provenanceBitmap`. This flag is set only for tunnels the dreaming +pipeline itself proposed and later accepted. It is never set for a +user-explicit, imported, or federated tunnel. OMEGA's retirement +predicate requires `isDreamed == true`. So a declared tunnel is never +retired, no matter how much or little it is recalled. + +## DiaryEntry.swift + +This file provides `DiaryEntry`. This is the first-person record of +what an agent thought, did, or learned at a point in time. A diary +entry is stored alongside drawers. It is queried separately, keyed by +`agentName` and an unvalidated `topic` tag. So one agent's diary +never leaks into another agent's wing-filtered recall by accident. +This is a convention, not an enforced constraint. `wing` and `room` +are plain required strings the caller supplies. `reward` and +`rewardProvenance` are the explicit quality-signal channel. When a +caller has a direct quality score for an entry, such as a user rating +or a model confidence value, it is recorded here. Otherwise +it is left for the dreaming daemon's implicit recall-based reward +inference to guess at later. + +## DiaryOperational.swift + +This file provides the four axes packed into +`DiaryEntry.operationalBitmap`, plus a single flag bit and their +computed accessors. `DiaryEventClass` has twelve cases. It records +what kind of substrate event this entry describes: capture, mutation, +withdraw, expunge, propose, associate, learn, signal emission, +maintenance, migration, training, or audit-tombstone. `DiarySeverity` +is scale-gapped, with four levels: trace, info, warning, and error. +`DiaryActorClass` records who produced the entry: user, substrate +daemon, MCP agent, migration tool, or federation peer. +`DiaryBatchMembership` has four values: standalone, batch-start, +batch-member, and batch-end. The single flag bit is +`requiresFollowup`. + +## KGFact.swift + +This file provides `KGFact`, a subject-predicate-object triple +distilled from a verbatim drawer. It retains `sourceDrawerID` as a +backreference, so a fact's provenance is always recoverable. +`subject`, `predicate`, and `object` are all free-form strings at +this rung. The value type itself enforces no entity vocabulary. That +enforcement, when it arrives, belongs to a later federated +knowledge-graph layer. `sourceDrawerID` may legitimately be an empty +string. An empty string is an unanchored-fact sentinel. A caller uses +it when asserting a freestanding triple, one not extracted from any +specific drawer. + +`KGFact` reuses the same three-bitmap pattern every other +bitmap-backed noun uses. Its adjective accessors are `state`, +`adjectiveSensitivity`, `exportability`, and `trust`. These decode +the same bit layout `Drawer` uses. So a fact and its source drawer +can be filtered by the same retrieval predicates. They share the same +encoding. + +## KGFactOperational.swift + +This file provides the four axes, plus one flag, packed into +`KGFact.operationalBitmap`. `KGExtractorClass` has six cases. They are ordered +roughly by rigor: manual, foundation-model, specialized-model, +rules-based, imported-KG, and federated. `KGAssertionKind` has four +cases: asserted, inferred, hypothesized, and contradicted. The last +case records that another fact disputes this one, without either +fact being retracted. Resolution is deferred to retrieval time. +`KGSpecificity` and `KGConfidenceBand` are both scale-gapped and +`Comparable`. So a filter can read as `fact.specificity >= +.specific`. The `isCanonical` flag marks a fact promoted to +estate-wide, load-bearing status. + +## Proposal.swift + +This file provides `Proposal`, a suggested change awaiting +confirmation. It is the durable record produced by the substrate's +only autonomous write surface. `Proposal` structurally mirrors +`KGFact`, with one addition `KGFact` predates: a required +`latticeAnchor`. `KGFact` was written before the cookbook +universalized the every-row-has-an-anchor invariant. `candidateState` +is the adjective set the proposal would apply to its target, if +accepted. The accept path reads this value, to know what to write. +`Proposal.state` decodes the proposal's own lifecycle position, from +its adjective bitmap's state field, the same field `Drawer.state` +decodes. This position starts at pending, while the proposal awaits +confirmation. It then moves to accepted, rejected, or withdrawn. + +## ProposalOperational.swift + +This file provides the five typed axes packed into +`Proposal.operationalBitmap`. `ProposalKind` has nine cases. It +records what kind of write this proposal proposes: `.newTunnel`, +`.mutateDrawer`, and seven more, including the newer +`.actionProposal`, `.recordObservation`, and `.tierAdvisory`. +`ProposalTargetObjectType` records which kind of row this proposal +targets. This includes the `.noneBrandNew` sentinel, for a +not-yet-existing target. It also includes `.systemState`, for a +proposal about the estate itself. `ProposalConfirmationSource` has +four values: human, +agent, automated threshold, and actuator. `ProposalGeneratedByClass` +has five values: dreaming daemon, MCP agent, federation sync, manual, +and tier aggregator. `ProposalConfidenceBucket` is scale-gapped and +`Comparable`. + +This file notes that the operational layout for `ProposalKind` and +its siblings comes straight from the engineering cookbook, rather +than being LocusKit-internal. So a conformance test pins every field +position and raw value against the cookbook table. +`composeOperational(kind:targetObjectType:generatedBy:confidence:)` +assembles a full operational bitmap from four of the five axes, in +one call. Confirmation source is left at its default, until a +confirmation step actually runs. Autonomic daemon sinks use this +function. They need to stamp genuine provenance on the proposals they +emit. + +## Association.swift + +This file provides `Association`, the graph edge recording that two +rows belong together. This is a statistical or dreaming-derived +pairing, distinct from a `Tunnel`'s typed semantic claim. +`Association` structurally mirrors `Tunnel`: a source and target +endpoint, three bitmap columns, and the Rev 1.0 soft-delete +reservation. Two differences set it apart. It carries no `kind` +column. An association's semantics live entirely in its operational +bitmap. It also carries a required `latticeAnchor`, anchored to the +lattice midpoint of its two endpoints. `Tunnel` predates this +requirement, so it lacks the field. + +## AssociationOperational.swift + +This file provides the three axes packed into +`Association.operationalBitmap`. `AssociationSignalSources` is a +twelve-bit bitset, an `OptionSet` rather than an enum, because more +than one signal can support the same pairing on its own. Its +signals include a co-recall signal, a co-confirmation signal, a +dream-pairing signal, a vector-similarity signal, a shared-entity +signal, an explicit-human signal, a fingerprint-similarity signal, +and three v0.36 additions: cross-estate, cross-tier, and +action-outcome. `AssociationDecayClass` is scale-gapped and +`Comparable`, with four levels: pinned, slow, normal, and fast. This +axis records how quickly an association ages out of relevance. +`AssociationArity` is binary today. An n-ary form is reserved for a +future version, per invariant I-23's current binary-only limit. + +## LearnedReference.swift + +This file provides `LearnedReference`, the durable record of an +external reference the grounding-driven `learn` verb brought into +the estate. It structurally mirrors `Association`: an id, content +columns, a required `latticeAnchor`, three bitmap columns, and the +soft-delete reservation. `Association` was chosen as the template +because it is the freshest content-bearing noun that already honors +the anchor requirement. `sourceCatalogID` is a +foreign-key reference to the `SourceCatalogEntry` this reference was +learned from. It is stored as an identifier string, the same way +`KGFact` stores `sourceDrawerID`, rather than embedding the full +catalog entry inline. `handle` is the reference's own locator. It is +distinct from the source's own locator, on the catalog entry it +points into. + +## LearnedReferenceOperational.swift + +This file provides the four axes packed into +`LearnedReference.operationalBitmap`. `RefreshPolicy` is scale-gapped +and `Comparable`. Its order runs from none through monthly, weekly, +daily, on-demand, and realtime. `DriftSeverity` is scale-gapped and +`Comparable`, with four levels: none, minor, major, and critical. +This axis records how far a reference has drifted from its source, +since it was last re-grounded. `LearnMode` is a single bit. It +records how the reference was acquired. `.byReference` means the +reference is held by pointer. `.byIngestion` means its content was +ingested wholesale. `LearnedReferenceSource` records where the reference was acquired +from: user, federation, household pairing, fleet pairing, tier +inheritance, or paired estate. This is distinct from +`sourceCatalogID`, which names which catalog entry the reference came +from, not how the reference arrived. + +## SourceCatalogEntry.swift + +This file provides `SourceCatalogEntry`, the durable, queryable +record of an external source from which references are learned. It +is the substrate behind the `source` slot of the `learn` verb. Its +reason for existing is grounding integrity. Every `LearnedReference` +must carry a genuine lattice anchor, never a fabricated sentinel. An +anchor is a property of the source: a web domain, a document corpus, +or a paired estate. It is not a property of each individual handle +learned from it. Cataloging a source once lets every reference from +it inherit that catalog entry's anchor. This is what makes a genuine +anchor available, without inventing one per reference. `SourceKind` +names six source classes: user, federation, household pairing, fleet +pairing, tier inheritance, and paired estate. This is the same +vocabulary `LearnedReferenceSource` uses, on purpose. The acquisition +channel a reference was learned through is the same vocabulary as the +kind of source it came from. + +## Node.swift + +This file provides `Node`, a container node in the estate's +containment tree. The estate root sits at depth zero. A wing sits at +depth one. A room sits at depth two. Every node carries two name +fields. `displayName` preserves whatever casing the first writer +used. `lookupName` is a normalized form: Unicode NFC, trimmed, with +internal whitespace collapsed and casefolded. This normalized form is +used for lookup and to keep names unique. So "Personal " and +"personal" resolve to the same node. The node's display name still +shows the casing it was first written in. `lifecycle` has two +values, active or tombstoned. A pair of HLC-typed timestamps, +`createdHlc` and `tombstonedHlc`, support this. They let the +containment tree read its own past state, the same way drawers do. +Neither needs a wall-clock timestamp to double as the +ordering key. `Node.normalizeLookupName(_:)` is the single, +conformance-relevant normalization function. Both the Swift and Rust +ports must produce byte-identical results from it. + +## NodeStore.swift + +This file provides `NodeStore`, the actor that owns the containment +tree's storage. It also owns the tree's create-on-demand lookup. +Given a display name and a parent id, it finds the active node with +that normalized lookup name under that parent, or creates one if +none exists. `NodeStore` is an actor for a specific reason. This +find-then-insert sequence stays race-free without needing an `INSERT +OR IGNORE` clause or a conflict-column mechanism. Two concurrent +requests to create the same wing under the same estate serialize +through the actor. So they produce exactly one node. The second +request's `findActiveNode` call cannot run until the first request's +insert has completed. + +`createNode(displayName:parentId:now:)` enforces two structural +invariants before writing. The parent must already exist, per +invariant I-NT-5. The new node's depth is the parent's depth plus +one. This depth must not exceed two, per invariant I-NT-2. A room cannot have a +child, because there is no fourth tier. `createRoot(displayName:now:)` +is the once-per-estate seed for the depth-zero root. It enforces that +exactly one root exists, per invariant I-NT-1, by returning the +existing root unchanged if one is already present. Tombstoned nodes +are invisible to lookup. A tombstoned wing does not block a later +request for a wing with the same name. It also cannot be reused by +mistake, for that later request. This is the no-resurrection guard, per +section five. + +## NodeBundleStore.swift + +This file provides `NodeBundleStore`, persistence for the +bundle-algebra count-vector totals. One row exists per node, room +or wing, per bundle kind. Each row is stored in the `node_bundles` +table. A count-vector here is a 256-element array of counts, one per +fingerprint bit, encoded as 1024 bytes of little-endian `UInt32` +values. `BundleKind.activeA` is the active centroid, a fold of a +node's currently active members. `BundleKind.departedB` is the +departed accumulator, eager-folded at departure time. The per-row +drawer fingerprints that feed these totals are never themselves +stored. Only the folded total is stored. This keeps the table +small, no matter the estate size. + +`put`, `get`, and `rooms(forWing:kind:)` are the read-and-write +surface. `encodeCounts` and `decodeCounts` handle the wire format. +`decodeCounts` throws `LocusKitError.invalidContent`, rather than +trapping, when a stored blob is not exactly 1024 bytes. So a corrupt +row surfaces as a recoverable error, instead of crashing the process. + +## BundleMaterializer.swift + +This file provides `BundleMaterializer`, the operation that computes +a Bundle A total from live drawers. It is the first +real consumer of `SubstrateKernel.countFold256`. Bundle A cannot be +maintained bit by bit. Active membership changes over time, and +the fold operation has no subtraction. So Bundle A is fully +recomputed on demand, typically by a periodic dreaming tick. + +`materializeRoom(wing:room:now:)` fetches every non-tombstoned drawer +in a room. It filters to `State.isClusterA` rows, the currently +believed ones. It derives each survivor's `Fingerprint256`, through +`EstateFingerprintFamilies`. It folds the set with +`kernel.countFold256`. It stores the result. `rollUpWing(wing:now:)` +merges a wing's already-materialized room bundles into one +wing-level bundle. The count-vector fold is associative. So this +merge produces the same result as folding every active drawer +in the wing. Rooms may be materialized in any order before +the wing roll-up runs. + +## Fingerprint256Adapters.swift + +This file provides a small marshaling convention. It packs a single +Int64 bitmap column into block zero of a `Fingerprint256`, with +blocks one through three zeroed. This lets a per-column bit +operation route through a SubstrateLib primitive instead: an OR, an +XOR, or a popcount. That primitive operates at the substrate's +canonical 256-bit width. LocusKit avoids reimplementing the same math at Int64 width this way. +This file is explicit that the convention is purely a type-shape +choice, with no algorithm of its own. The file also warns that this +packing differs from the cookbook's Bitmap-LSH SimHash interpretation +of block zero. A caller must not mix the two uses of the same 64-bit +lane. + +`init(int64Column:)` and the `int64Column` computed property are the +pack and unpack directions. They are used throughout +`ContainerFingerprintStore`. + +## ContainerFingerprintStore.swift + +This file provides `ContainerFingerprintStore`, the per-container +OR-reduction totals. These totals make fingerprint-based +recall pruning possible. For every room, the store keeps the bitwise +OR of the three bitmap fields, across every active drawer in that +container. It also rolls this total up to every wing. OR is +monotone. So a container's total can only ever gain bits as content +is added. So a bit absent from the total is provably absent from +every row in the container. This is exactly the soundness property +recall pruning needs. A filter requiring that bit can safely skip the +whole container, without fetching a single row. + +`orIn(wing:room:adjective:operational:provenance:now:)` is the +incremental maintenance path. It runs on every drawer capture. It +folds the new drawer's bitmaps into both the room-level and the +wing-level row, through `ContainerFingerprint.merging(_:)`. That +function itself routes through `SubstrateLib.ORReduce.reduce`, at +canonical 256-bit width, using the `Fingerprint256` packing +convention above. The clear side is absent on purpose. Withdrawing +or expunging a drawer never clears bits from the total. A stale +set bit is a harmless over-approximation. It can only cause an +unnecessary scan, never a missed match. Meanwhile, three other +functions exist for a different purpose. They are `rebuildRoom`, +`rollUpWing`, and `rebuildAll`. They tighten the total back down, +now and then, from a fresh full scan. `rebuildAll` is what +`Estate.open` calls once at startup. So an existing estate's +total is guaranteed complete, and thus sound, before any +recall runs against it. + +## MerkleRollup.swift + +This file provides the Merkle content-integrity rollup, an extension +on `Estate`. It computes a hash tree from the bottom up: room, then +wing, then estate. This hash tree covers the same containment tree +`NodeStore` maintains. A Merkle tree is a hash tree where each +parent's hash is computed from its children's hashes. So a single +top-level root hash can attest to the exact contents of an entire +subtree. Changing any leaf changes every hash on the path up to the +root. A caller must invoke the rollup by hand. It does not run on +its own after every write. Computing it inline on every drawer +capture would be +costly. The cost would be proportional to room size, on every write. +This would peg the CPU during a bulk import. + +`computeRoomMerkleRoot(roomNodeId:)` hashes a room's live drawers. It +reads each row's stored `content_hash` column when present, written +by a separate hash-on-write hook. It computes a leaf hash on demand +from the drawer's content when the column is absent. It always +excludes two kinds of rows from the snapshot. The first kind is +tombstoned rows. These rows are irreversibly deleted. The second kind +is withdrawn rows, at state raw value 18. A withdrawn drawer's +content has been retracted by the user. So it must not remain +retrievable through a content-attesting snapshot. +`computeEstateOrWingMerkleRoot(parentNodeId:)` hashes a set of +children's already-computed roots. The same function serves two +levels. At the wing level, it hashes room roots. At the estate level, +it hashes wing roots. `rollupMerkleRoots(roomNodeId:now:)` runs all +three levels for one changed room. `recomputeAllMerkleRoots(now:)` is +the full bottom-up recompute, used after bulk import, migration, or +corruption recovery. `rollupAllMerkleRoots(now:)` is a documented +alias of the full recompute, named for the batch-capture reindex +path. `createSnapshot(label:now:additionalAttestations:)` recomputes +the full tree, so a snapshot is never taken against stale roots. It +then writes an attestation row for the estate root and every wing, +plus any attestations a higher composition-layer kit supplies. + +## Manifest.swift + +This file provides `ManifestKey`, the eighteen required and seven +optional typed keys of the v1 manifest key-value table. It also +provides `ManifestValues`, the typed, read-only snapshot +`DrawerStore.readManifest()` produces. The manifest table itself is +a plain key-value store. This file turns a table of plain strings +into a fully typed record a caller can rely on. +`ManifestKey.ed25519PublicKey` and `ed25519PrivateKeyWrapped` are the +two identity-related keys. `ed25519PublicKey` is the estate's +federation public key. It is safe to store here, because a public +key has no confidentiality requirement. `ed25519PrivateKeyWrapped` is +different: a reserved, never-written seam. It exists only for +backward read-compatibility, with an estate opened before the +Keychain migration. The private key itself lives exclusively in +`EstateIdentityKeyStore`. + +## DefaultWings.swift + +This file provides the seven default wing definitions. These are +seeded into a fresh estate at provision time. It also provides three +related constants. `defaultWingName` is "Agentic Memory," the wing +`capture` uses when a caller supplies no explicit wing. `hintRoom` is +"AI_Charter_Hint," the room each seeded wing's hint memory lives in. +`hintUDCCode` is "001," the UDC Knowledge-class code stamped on hint +drawers. Each `WingDefinition` pairs a wing name with a +plain-language hint describing the wing's role. The seven wings are: +Agentic Memory, User Canon, Source Corpus, Personal, Professional, +Projects, and Temp. Each hint is seeded as an ordinary, fully +recallable, user-deletable drawer. So a fresh agent has a working +orientation to its own memory structure, from its first session. The +set is only a suggestion, not a fixed schema. An agent may +create any extra wing it needs. + +## RecallStream.swift + +This file provides `RecallStream`, the paged async sequence +`Estate.recall` returns. Iterating a `RecallStream` produces one +`RecallPage` at a time. The first page arrives synchronously, on the +first `next()` call. Later pages arrive lazily. `isLast` is true only +on the final page. So a caller can drive a uniform `for await` loop, +without special-casing an empty result. A genuinely empty corpus +still emits exactly one page, with zero rows and `isLast == true`. + +`degradedStages` is the channel by which a failed internal read +becomes observable to a caller. A failed read might come from several places: the bounded corpus +scan, the room-fingerprint enumeration, a room's drawer read, or the +bitmap evaluator itself. `recall` never throws. +Section 7.8.1 of the specification requires it stay non-throwing. So +an internal failure would otherwise be indistinguishable from a +genuinely empty result. A non-empty `degradedStages` array is what +lets a caller, chiefly GeniusLocusKit's `RecallDirector`, tell the +two apart. `AsyncIterator.hydrate(_:)` applies the requested +`HydrationLevel` per page. At `.bitmapOnly`, it rebuilds each drawer +with `content` blanked to the empty string, while preserving every +bitmap and metadata field. At `.structured` and `.full`, rows pass +through unchanged. The content-stripping decision for `.structured` +was already made upstream, at the storage-fetch layer, not here. + +## RecallTraceItem.swift + +This file provides `RecallTraceItem`, one record of a single drawer +returned by a recall operation. This is the substrate for the "later +two-source reward" mechanism. A dreaming daemon uses this mechanism +to distinguish rows a user acted on from rows that were +returned but ignored. The `used` flag is bit 0 of `operationalBitmap`, +decoded through a computed property rather than stored as a separate +`Bool` column. So the type carries no stored boolean fields at all. +Every bitmap-backed LocusKit noun follows this convention. `score`, +when present, is the recall's own similarity score for the row. A +`nil` score means the recall that produced this trace carried no +score, for example an ordinary ordered-by-capture-time query. + +## AuditTypes.swift + +This file provides `BitmapState`, the snapshot of a row's three +bitmap columns at a specific past HLC. `Estate.bitmapState(rowID:asOf:)` +returns this snapshot. `BitmapState` exists as a small, focused +return type, for a precise reason. The reconstruction that produces +it, `AuditLogFold.projectStateAt`, is a SubstrateLib primitive. It +folds a row's sealed audit events forward in HLC order. Yet it +returns its result in a different shape. `BitmapState` is LocusKit's +own public-facing wrapper. It names the three bitmaps by hand, +rather than exposing the substrate's internal projection type +directly. + +## Summaries.swift + +This file provides `WingSummary` and `RoomSummary`, the two small +summary types `DrawerStore.listWings` and `listRooms` produce. Both +are computed projections over the current drawer set. LocusKit has +no separate `wings` or `rooms` table. Wing and room identity lives +entirely in the `nodes` table. A summary's drawer and room counts are +simply whatever the live query finds, at the moment it runs. They are +not a maintained running total. + +## LocusKitTelemetry.swift + +This file provides LocusKit's opt-in telemetry emission functions. +They are wired through `IntellectusLib.Intellectus.report(_:)`. Every +emit function here follows the same three rules, stated in the +file's header. First, the reported value is always an +`@autoclosure`. It is never evaluated when monitoring is disabled. +The off-path cost is one atomic boolean load and a branch: no lock, +no allocation. Second, the `now` timestamp is always caller-supplied, +never read from a clock inside the function. This preserves +IntellectusLib's own determinism contract. Third, the metric +namespace follows the fleet-wide `..` convention. +Examples include `locuskit.drawer.capture_latency_ms`, +`locuskit.kgfact.add_count`, and `locuskit.gate.reject_count`. + +`emitDrawerCapture`, `emitDrawerQuery`, `emitKGFactAdd`, +`emitKGFactQuery`, `emitTunnelAdd`, `emitGateAdmit`, and +`emitGateReject` share one pattern. Each is called from its +corresponding `DrawerStore` method. Each call happens at the exact +moment its operation completes. For the gate functions, the call +happens when `AuditGate.admit` returns instead. Telemetry is purely additive. It is never +consulted by any control flow that decides what a method returns. So +every store method's functional behavior is byte-identical, whether +monitoring is enabled or not. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the kit. It holds +fifty-seven source and test files, under `rust/src/` and +`rust/tests/`. These files mirror the Swift implementation +file-for-file: `drawer.rs`, `drawer_store.rs`, `bitmap_evaluator.rs`, +`merkle_rollup.rs`, `estate.rs`, and more. The Rust side also carries +three concrete `DrawerStore` backends the Swift side does not +carry: `drawer_store_inmemory.rs`, `drawer_store_sqlite.rs`, and +`drawer_store_postgres.rs`. All three build over a shared, +storage-agnostic `DrawerStoreCore`. + +Conformance tests under `rust/tests/` gate byte-for-byte agreement +with the Swift implementation. These tests cover adjective and +operational bitmap conformance, provenance bitmap conformance, +corrupt readback, Merkle rollup, outline proofs, recall pruning, +temporal reads, and the LP0 fixed-vector suite. They gate agreement +on every bitmap layout, every fingerprint derivation, and every +state-transition legality decision. Suppose you change a bitmap layout, a fingerprint derivation, or a +transition rule on either leg. Mirror that change on the other leg. +Then run both test suites. The fixtures +and conformance tests are the contract, not a convenience. diff --git a/packages/kits/LocusKit/docs/OVERVIEW.md b/packages/kits/LocusKit/docs/OVERVIEW.md new file mode 100644 index 0000000..063b643 --- /dev/null +++ b/packages/kits/LocusKit/docs/OVERVIEW.md @@ -0,0 +1,344 @@ +--- +doc: OVERVIEW +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# LocusKit Overview + +## What This Kit Does + +LocusKit is the storage substrate for a MOOTx01 estate. An estate is +one user's complete memory store. MOOTx01 is an on-device AI memory +system. It stores what an AI observes over time. It helps the AI +recall that memory later. LocusKit is the layer that holds memories on +disk. It lets a caller file a memory, find it, and change its standing +over time. + +LocusKit is a kit, not a library. A library, or lib, is a folder of +related files that does one job well. A kit is a larger package. A kit +composes several libraries into one subsystem. Kits may depend on +libs. Libs never depend back on kits. + +LocusKit depends on several libs. `PersistenceKit` handles storage. +`SubstrateLib` and `SubstrateTypes` supply the write-gate math. +`SubstrateML` supplies classification math. `LatticeLib` supplies a +taxonomic lookup. LocusKit composes these libs into the estate's +spatial memory surface. A higher kit, `GeniusLocusKit`, builds on top +of LocusKit. This document does not cover `GeniusLocusKit`. + +The word `MemPalace` names the spatial metaphor LocusKit implements. +Content lives in drawers. Drawers sit in rooms. Rooms sit in wings. +Wings sit in one estate. This document calls that three-level +structure the containment tree. + +## The Problem It Solves + +An AI's memory needs more than a place to store text. It must know if +a memory is still believed. It must know how sensitive a memory is. It +must know if a memory can leave the device. It must know how much to +trust a memory and where the memory came from. It needs to change its +mind about a memory. It must mark a memory as superseded, contested, +confirmed, or gone. It must do this without losing the history of that +change. It needs to find memories fast, by many criteria. It must do +this without scanning every row each time. It needs all of this to +survive a crash partway through a write. + +LocusKit answers each of these needs with one mechanism per need. + +- **Standing.** Every memory carries three packed 64-bit numbers. Each + number is called a bitmap. One bitmap records the memory's adjective + state: active, superseded, contested, accepted, rejected, or gone. A + second bitmap records operational facts. It records the capture + method and the content kind. A third bitmap records provenance: + where the memory came from and how much the system trusts it. A + bitmap is a fixed-size number. Different ranges of bits inside it + are called fields. Each field holds one small, independent value. + Packing many fields into one number keeps a memory's standing in a + single machine word. Otherwise the record would need a dozen + separate columns. +- **Safe change over time.** A memory's state does not move freely. It + can move only along a fixed set of legal paths. For example, an + active memory can become contested. A rejected memory cannot become + accepted. LocusKit sends every state-changing write through a write + gate, called `AuditGate`. `AuditGate` is owned by SubstrateLib. The + gate checks that the move is legal. It then writes one sealed, + tamper-evident audit event. That event records exactly what changed. + The audit event, not the live row, is the source of truth. The live + row is just a cached copy of the latest audit event, kept for fast + reads. Row and log update together inside one transaction. So every + value read back from a row matches a real sealed event in its audit + log. The row is never observed one step ahead of its own history. +- **Fast, layered search.** A caller expresses a query as a chain of + named filters. Filters name a concern in plain language, such as + currently believed, in this room, or captured after this date. + LocusKit compiles that chain through four tiers, from cheap to + costly. The first tier is the bitmap tier. It runs near-free integer + checks. The second tier is the structured tier. It checks room, + wing, and lattice fields. The third tier is the content tier. It + searches the verbatim text for a substring. The fourth tier is the + ordering pass. It sorts the surviving rows. Before any of that, a + per-container fingerprint check can rule out whole rooms or wings. + It does this without loading a single row. +- **Crash safety.** Every state-changing write happens inside one + database transaction. That transaction updates the live row and + appends the audit event in one step. If the process dies partway + through, the transaction commits both changes or neither. The live + row and the audit log can never disagree about what happened. + +## How It Works + +### Nine kinds of rows, one set of patterns + +LocusKit stores nine kinds of rows. This document calls each kind a +noun. The nine nouns are: `Drawer`, `Tunnel`, `DiaryEntry`, `KGFact`, +`Proposal`, `Association`, `LearnedReference`, `SourceCatalogEntry`, +and `Node`. `Drawer` holds verbatim content. `Tunnel` is a typed link +between two locations. `DiaryEntry` is a first-person agent record. +`KGFact` is a subject-predicate-object triple pulled from a drawer. +`Proposal` is a suggested change waiting for confirmation. +`Association` is a graph edge that records two rows belonging +together. `LearnedReference` is an external reference brought in by +the `learn` verb. `SourceCatalogEntry` is the durable record of where +a learned reference came from. `Node` is an entry in the containment +tree: the estate root, a wing, or a room. + +Most of these nouns share the same three-bitmap pattern described +above. Each noun has its own file that decodes its own bitmap layout. +For example, `DrawerOperational.swift` decodes +`Drawer.operationalBitmap`. `KGFactOperational.swift` decodes +`KGFact.operationalBitmap`. + +### The containment tree + +Estates used to store a drawer's wing and room as two plain text +columns. LocusKit now stores them as a three-level tree of `Node` +rows. The estate root sits at depth zero. Wings sit at depth one. +Rooms sit at depth two. A drawer references its room through a foreign +key, `parentNodeId`, rather than by name. + +`NodeStore` resolves a wing-and-room name pair to a node id. It +creates the node on first use. It returns the existing node on every +later use. So filing a drawer into "Personal / Health" always lands in +the same room node, no matter how many times the name is used. A +Merkle rollup, described below, walks this tree from bottom to top. +Renaming a room never requires touching every drawer inside it. + +### The write gate + +`DrawerStore` is the actor that owns every table. Nearly every write +that changes a row's standing is a gated write. A gated write happens +when a caller captures a drawer, moves its state, or edits an +adjective field. + +For a gated write, `DrawerStore` reads the row's current bitmaps. It +hands those bitmaps to `AuditGate.admit`, along with the proposed +change. `AuditGate.admit` returns one of two things: a sealed +`AuditEvent` to store, or a rejection naming the broken rule. A broken +rule might be an illegal state transition. It might also be a +forbidden bitmap combination, such as sensitivity `secret` paired with +exportability `public`. + +`DrawerStateValidator` and `ForbiddenCombinationValidator` mainly +document these same rules. The live enforcement now happens inside the +gate itself. + +### The recall pipeline + +`Estate.recall` is the read side of LocusKit. A caller builds a +`RecallFrame`. The frame carries a chain of `Filter` values. This +chain is the recall filter algebra. No `Filter` case exposes a raw bit +position. Each `Filter` case names a domain concern instead, such as +trustworthy, in this wing, or captured after a date. + +`Estate.recall` first asks `ContainerFingerprintStore` whether a room +or wing can be ruled out. It checks a cached per-container +fingerprint, a bitwise OR of every active drawer's three bitmaps in +that container. This check happens before LocusKit fetches a single +row. + +Rows that survive this check then pass through `BitmapEvaluator`. +`BitmapEvaluator` runs the filter chain in four tiers: bitmap, +structured, content, and ordering. `BitmapEvaluator` can also rebuild +a row's bitmap state at a past point in time. It does this by folding +the row's audit log forward. The folding function is +`AuditLogFold.projectStateAt`, a SubstrateLib primitive. + +### Structural similarity and integrity + +Two more subsystems sit alongside the write-and-read core. +`DrawerFingerprint` derives a 256-bit structural fingerprint for each +drawer. It builds this fingerprint from the drawer's bitmaps, lattice +anchor, lineage, and timing. `DrawerFingerprint` uses SubstrateLib's +SimHash machinery to do this. These fingerprints feed two systems. The +first is `ContainerFingerprintStore`'s pruning aggregates. The second +is a bundle-algebra subsystem, made of `NodeBundleStore` and +`BundleMaterializer`. This subsystem folds many fingerprints into one +compact count vector per room or wing. + +`MerkleRollup` computes a content-integrity hash tree from the bottom +up. It walks the same containment tree: room, then wing, then estate. +This lets a snapshot attest that its content has not silently changed. + +### The nine verbs + +`EstateVerbs.swift` adds nine verb methods to `Estate`. The nine verbs +are: `capture`, `recall`, `mutate`, `withdraw`, `expunge`, `reanchor`, +`learn`, `propose`, and `associate`. + +`capture` files a new drawer or tunnel. `recall` runs the pipeline +described above. `mutate` moves a row's standing along a named axis. +`withdraw` retracts a drawer. `expunge` hard-deletes a drawer's +content but keeps its audit trail. `reanchor` moves a drawer to a new +room or lattice position. `learn` brings in an external reference, +grounded to its source's own lattice anchor. `propose` records a +suggested change awaiting confirmation. `associate` records a graph +edge between two rows. + +Every verb takes a named frame struct as its argument, such as +`CaptureFrame` or `RecallFrame`. So no raw bitmap value ever crosses +the public boundary. + +## How the Pieces Fit + +Figure 1 shows the kit's topology. It shows the major parts and how a +write and a read move through each one. + +![Figure 1. Topology of LocusKit](topology.svg) + +*Figure 1. Topology of LocusKit. On the left, a capture flows through +the write gate into the drawers table. Off the write path, it also +updates the container-fingerprint and Merkle-rollup subsystems. On the +right, a recall flows through fingerprint pruning and the four-tier +bitmap evaluator. Dashed boxes mark external kits and libs that +LocusKit depends on but does not own.* + +`Estate` is the single public entry point to LocusKit. It owns one +`DrawerStore`, the actor holding every table. It owns one +`ContainerFingerprintStore`, the pruning aggregates. It owns one +`NodeStore`, the containment tree. `Estate` exposes the nine verbs. It +also exposes read-only pass-throughs, such as `allDrawers`, +`allKGFacts`, and `tunnelsFromWing`. GLK and other higher-level +consumers use these pass-throughs instead of reaching into +`DrawerStore` directly. + +`DrawerStore` itself keeps its add path internal, not public. Its add +method, `addDrawer`, stays internal. So the only sanctioned way to add +a drawer from outside the file is through `Estate.addDrawerCovered`. +That method bundles the row insert with the container-fingerprint +update. This is a structural guarantee: a drawer can never be captured +without also updating its container's pruning aggregate. + +## What Ships in the Package + +The package ships the Swift sources under `Sources/LocusKit/`. It also +ships a parallel Rust port under `rust/`. The Swift package depends on +five sibling libs. Four of them supply write-gate and fingerprint +math: `SubstrateLib`, `SubstrateTypes`, `SubstrateKernel`, and +`SubstrateML`. `PersistenceKit` supplies the storage abstraction. +`IntellectusLib` supplies opt-in telemetry. `LatticeLib` supplies the +pinned Q-ID ancestor closure that `DrawerFingerprint` hashes into its +lattice block. + +LocusKit ships no pinned data artifacts of its own. Its schema, +`LocusKitSchema`, is declared entirely in `PersistenceKit` primitives. +The schema is created fresh each time an estate opens. diff --git a/packages/kits/LocusKit/docs/topology.svg b/packages/kits/LocusKit/docs/topology.svg new file mode 100644 index 0000000..b2048d7 --- /dev/null +++ b/packages/kits/LocusKit/docs/topology.svg @@ -0,0 +1,129 @@ + + + + + + + + + + + + LocusKit: capture and recall through the estate facade + + + + Capture + Estate.capture + + + Recall + Estate.recall + + + + Estate + verb facade + (actor) + + + + NodeStore + containment tree + + + + DrawerStore + tables + write gate + + + + ContainerFingerprint + Store (pruning OR) + + + + BitmapEvaluator + 4-tier filter pipeline + + + + MerkleRollup + room → wing → estate + + + + + + + + + + + + + + + + + + + + External kits and libs (owned outside LocusKit) + + + AuditGate + SubstrateLib : legal-transition + + + PersistenceKit + Storage / RowStore / AuditLog + + + LatticeLib QIDClosure + pinned P31/P279 ancestors + + + + + + + every gated write validated here + every table's backend + read by DrawerFingerprint's lattice block + + Figure reads left to right: an input verb enters through Estate, which resolves + containment-tree nodes, writes through DrawerStore's audit-gated path (top), + or prunes and evaluates candidates for a read (bottom). Dashed edges cross a + kit/lib boundary LocusKit depends on but does not own. + diff --git a/packages/kits/VectorKit/docs/AGENT_MAP.md b/packages/kits/VectorKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..8ff7873 --- /dev/null +++ b/packages/kits/VectorKit/docs/AGENT_MAP.md @@ -0,0 +1,205 @@ +--- +doc: AGENT_MAP +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# AGENT_MAP | VectorKit + +PURPOSE: on-device embedding generation (`EmbeddingProvider` seam) + model-tagged vector storage (`VectorStore`, PersistenceKit-backed) + dual-lane nearest-neighbour search: binary Hamming (Lane A `BruteForceIndex` oracle / Lane B `MIHIndex` sub-linear exact, promoted at `mihThreshold`) and float cosine/l2/dot (`FloatBruteForceIndex`, one index per modelID) + ColBERT MaxSim late-interaction scorer (`MaxSimScorer`, standalone, not wired into VectorStore). + +DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session | I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file | conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap). + +ENTRY POINTS (most callers need only these): +- VectorStore.swift:451 `VectorStore.addVector(itemID:engram:modelID:modelVersion:filedAt:)` | write one binary vector +- VectorStore.swift:1134 `VectorStore.findNearest(probe:modelID:limit:) -> [VectorMatch]` | binary Hamming k-NN +- VectorStore.swift:1223 `VectorStore.findNearestFloat(probe:modelID:limit:) -> [VectorMatch]` | float cosine k-NN +- FloatSimHashEmbeddingProvider.swift:63 `FloatSimHashEmbeddingProvider.embed(_:) -> Engram` | text → fingerprint via injected inference + FloatSimHash + +## Symbol Table + +### Module surface +- VectorKit.swift:1 | namespace/header only; no types. Consumers `import EngramLib` separately for `Engram` (not re-exported). + +### Errors | VectorKitError.swift +- :5 `enum VectorKitError: Error, Sendable, Equatable` | concrete cases, never optional+log +- :9 `.embeddingFailed(String)` / :13 `.modelUnavailable(String)` / :17 `.storeUnavailable(String)` / :21 `.notFound` (reserved, unused by current API) / :27 `.invalidPayload(String)` / :31 `.decodingFailure(String)` +- :41 `.int8QuantizationPolicyUndefined(String)` | thrown on every int8 write; policy unratified (VECTORKIT_SPEC §I-4a); remove guard+case only when ratified +- :56 `.embedFloatVocabMiss(String)` | distributional-provider OOV signal, distinct from embeddingFailed + +### Embedding seam | EmbeddingProvider.swift +- :15 `protocol EmbeddingProvider: Sendable` | modelID/modelVersion + embed/embedFloat/embedPair/embedBatch +- :37 `embed(_:) -> Engram` | MUST return `Engram.zero` for empty string (cross-provider contract, mirrored in Rust trait) +- :64 `embedFloat(_:) -> [Float]` | opt-in; default impl (:105) throws embeddingFailed; empty input → `[]` never zero-vector +- :81 `embedPair(_:) -> (engram, floats)` | default impl (:115): two-pass (embed then embedFloat), float opt-out swallowed to `[]` +- :93 `embedBatch(_:) -> [Engram]` | default impl (:123): sequential; override for batched inference + +### Concrete provider | FloatSimHashEmbeddingProvider.swift +- :35 `struct FloatSimHashEmbeddingProvider: EmbeddingProvider` | Swift mirror of Rust vectorkit::FloatSimHashEmbeddingProvider +- :44 `projectionSeed: UInt64` | distinct seeds ⇒ distinct fingerprints for same float vector (I-4 enforced at projection layer) +- :49 `inference: @Sendable (String) async throws -> [Float]` | host-injected; kit owns no tokenizer/model +- :63 `embed(_:)` | empty-string short-circuit BEFORE inference call, then `FloatSimHash.project(vector:seed:)` +- :87 `embedFloat(_:)` | returns the SAME vector embed() projects; no double inference + +### Engine foundation types (Lane F | additive-only, no local field additions) +- VectorRecordKey.swift:33 `struct VectorRecordKey: Sendable, Equatable, Hashable, Comparable` | (itemID, vectorIndex, modelID, modelVersion); ordering IS the partition/tie-break order +- VectorRecordKey.swift:87 `< (lhs:rhs:)` | lexicographic (itemID, vectorIndex, modelID, modelVersion); DO NOT reorder fields +- VectorPayload.swift:76 `enum VectorKind: UInt8` | .binary=0/.float32=1/.int8=2; ON-DISK raw values, never reorder +- VectorPayload.swift:101 `struct VectorPayload` | kind+dim+bytes+scale(int8 only, unused in prod) +- VectorPayload.swift:146 `init(engram:)` | binary, zero-copy wire bytes +- VectorPayload.swift:163 `init(floats:)` | float32, explicit little-endian serialization (byte-order portability, not native-order) +- VectorPayload.swift:187/:200 `asEngram()` / `asFloats()` | throw invalidPayload on kind/size mismatch +- VectorPayload.swift:38 `struct VectorPayloadInput` | bulk-write row bundle (itemID, vectorIndex, payload, modelID, modelVersion, filedAt) +- DenseHit.swift:45 `struct DenseHit: Sendable, Equatable` | key + rawDistance(Int32, dual-purpose: Hamming int OR Float bit pattern) + metric +- DenseHit.swift:101/:121 `hammingDistance` / `floatDistance` | typed accessors reinterpreting rawDistance +- DenseHit.swift:132 `enum LaneTag` | .binaryDense/.floatDense/.sparse/.lateInteraction (fusion/cross-package use) +- DenseMetric.swift:41 `enum FloatMetric` | .cosine/.l2/.dot; VectorKit-owned (ADR-008) +- DenseMetric.swift:58 `enum BinaryMetric` | .hamming/.jaccard (jaccard reserved, BruteForceIndex rejects it) +- DenseMetric.swift:81 `enum DenseMetric` | .binary(BinaryMetric)/.float(FloatMetric) umbrella; :91-:103 shorthand statics +- DenseIndex.swift:67 `enum SearchDirection` | .nearest/.farthest; farthest is bottom-K scan, NOT negated top-K +- DenseIndex.swift:85 `struct MetadataFilter` | modelID/modelVersion wildcard-if-nil; :109 `accepts(_:)` +- DenseIndex.swift:35 `enum IndexKind` | .bruteForce/.mih tag (nominal dispatch, not type-casting) +- DenseIndex.swift:131 `protocol DenseIndex: Sendable` | build/search/add/remove seam; BruteForceIndex is the binary oracle + +### Binary engines +- BruteForceIndex.swift:47 `actor BruteForceIndex: DenseIndex` | Lane A; conformance oracle; ZERO Hamming math in file (I-7) +- BruteForceIndex.swift:101 `search(probe:metric:k:filter:)` | only .binary(.hamming); model-partition slice via O(log m) lookup; sorts (distance ASC, FULL key ASC) | NOT EngramLib.findNearest (different tie-break) +- BruteForceIndex.swift:227 `add(key:vector:)` | tombstone-then-append upsert; :276 `remove(key:)` | tombstone only, no reclaim +- BruteForceIndex.swift:305 `currentSnapshot()` | value-copy for cross-actor read (VectorStore tombstone scans) +- BruteForceIndex.swift:326/:342 `setTombstoneBit` / `buildPartitions` | shared bit-layout + partition-rebuild helpers, mirrored in ResidentArrayStore +- MIHIndex.swift:251 `actor MIHIndex: DenseIndex` | Lane B; sub-linear EXACT Hamming k-NN via Multi-Index Hashing; output MUST equal BruteForceIndex bit-for-bit (BLOCKER conformance gate, MIHIndexTests.swift) +- MIHIndex.swift:72 `enum MIHBandCount: UInt32` | {.m4,.m8,.m16,.m32} ONLY (§1.7: keeps sub_bits∈{64,32,16,8}, no word-straddle) +- MIHIndex.swift:303 `init(bandCount:maskBudget:)` | maskBudget nil ⇒ dynamic max(n, 2^20) per query +- MIHIndex.swift:351 `search(...)` → :450 `knn(...)` | progressive-radius pigeonhole expansion; stop when heap full AND worstDist ≤ r +- MIHIndex.swift:504 enumeration-budget guard | projected flip-mask count vs budget; over-budget + heap not exact ⇒ :575 `bruteScan` fallback (still exact, just O(n)); fires `.notice` log + `vectorkit.mih.enumeration_fallback` metric +- MIHIndex.swift:711 `cumulativeChoose(subBits:rho:)` | Σ C(subBits,d); saturates to Int.max on overflow; MUST match Rust saturating_add bit-for-bit +- MIHIndex.swift:764 `colexFlipMasks(subBits:maxHamming:body:)` | Gosper's-hack colex enumeration, ascending subset size then ascending mask value; canonical order, internal (test-visible) +- MIHIndex.swift:654 `extractBand(from:bandIndex:)` | canonical bit numbering (bit i → word i/64, LSB=0); word-straddle branch unreachable for allowed m + +### Float engine +- FloatBruteForceIndex.swift:60 `actor FloatBruteForceIndex: DenseIndex` | Lane C/D (file header says "Lane C", VectorStore.swift comments say "Lane D" | same type, inconsistent lane label in source, not a functional issue); float32 ONLY; NOT four-way bit-identical (documented, do not "fix") +- FloatBruteForceIndex.swift:84 `build(from:)` | O(1) reference store; array IS the index +- FloatBruteForceIndex.swift:102 `search(...)` | validates probe.kind/.dim vs array stride; cosine treats zero-vector as distance 1.0 (no div-by-zero) +- FloatBruteForceIndex.swift:156 `searchFarthest(...)` | identical scan+distance as search(); only sort direction flips (:233 `rank(...direction:)`) +- FloatBruteForceIndex.swift:273 `add(key:vector:)` | FIRST add establishes stride; later mismatched byte count throws invalidPayload (prevents storage corruption) +- FloatBruteForceIndex.swift:323 `remove(key:)` | tombstone; compaction only on next build() + +### Late-interaction scorer (standalone, not a DenseIndex) +- MaxSimScorer.swift:97 `struct MaxSimScorer: Sendable` | Lane E1, Exact-A exhaustive ColBERT MaxSim; conformance reference for future pruned variants +- MaxSimScorer.swift:148 `score(queryTokens:documents:k:) -> [MaxSimHit]` | Σ(256−min hamming) per query token; documents iterated in SORTED itemID order (dict order is undefined); sort (score DESC, itemID ASC); truncate to k AFTER full sort +- MaxSimScorer.swift:55 `struct MaxSimHit` | itemID + integer score [0, 256×|Q|] +- All distances via `EngramLib.Session.distances` (I-7); session built once per scorer, reused across the whole score() call + +### Resident array (shared data contract, Lane F) +- ResidentVectorArray.swift:65 `struct ResidentVectorArray: Sendable` | packed fixed-stride array; kind/stride/count/storage/keys/modelPartitions/tombstones; measured 87% of pre-resident latency was fetch+decode, 0.4% kernel | this type removes the fetch+decode cost +- ResidentVectorArray.swift:135 `liveCount` | O(count/64) tombstone-bitmap walk; used for sidecar staleness (live-vs-live compare) +- ResidentVectorArray.swift:190 `partitionRange(for:)` | binary search, O(log m) +- ResidentVectorArray.swift:214/:228 `isTombstoned(_:)` / `vectorBytes(at:)` | per-slot accessors every engine scan loop uses +- ResidentVectorArray.swift:44 `struct ModelPartitionEntry` | modelID + half-open Range + +### Resident array persistence | ResidentArrayStore.swift +- :116 `actor ResidentArrayStore` | owns optional `.vec` sidecar; vectors TABLE remains sole durable source; sidecar is regenerable cache only +- :97 `kVecVersion = 0x0002` | format version; adds live_count field after count (discarded on load, recomputed from tombstone bitmap | stale header value cannot corrupt results) +- :102 `kDefaultTombstoneCompactionThreshold = 0.25` +- :183 `load()` | missing/invalid sidecar ⇒ start empty, no crash +- :209 `rebuild(from:)` | full rewrite from sorted [(key,bytes)]; used on stale-sidecar detection +- :247 `append(key:bytes:)` | EAGER write (immediate sidecar rewrite) +- :277 `appendDeferred(key:bytes:)` | WRITE-BEHIND single-add path (production default via VectorStore.addPayload); sets isDirty, no disk write; caller must flush() +- :299 `appendBatch(records:)` | bulk path, ONE sidecar write per batch (not per record) | TASK #24 amortization +- :346 `flush()` | persists pending write-behind mutation; no-op if !isDirty +- :387/:420 `tombstone(key:)` (eager, writes) / `tombstoneDeferred(keys:)` (batch, no write, sets isDirty) +- :448 `compact()` | drops tombstoned slots, sorted-by-key rewrite, deterministic output +- :563/:610/:623 `writeSidecar` / `readSidecar` (mmap via .mappedIfSafe) / `parseSidecar` | every length field bounds-checked before trust; magic "VEC1" (:85 `kVecMagic`) +- On-disk layout: magic(4)|version(2)|kind(1)|stride(4)|count(4)|live_count(4)|tombstone_words(4)|tombstones|vectors|keys(variable)|partition_index(variable); ALL integers little-endian (cross-host byte-identity, arch spec §4.3) + +### Storage-facing types +- StoredVector.swift:20 `struct StoredVector: Sendable, Equatable` | decoded `vectors` row; `engram` non-nil ONLY for binary kind (float/int8 rows: use getPayload) +- VectorMatch.swift:19 `struct VectorMatch: Sendable, Comparable, Equatable` | itemID/distance/modelID; :43 `<` | (distance ASC, itemID ASC) universal tie-break + +### Storage actor | VectorStore.swift +- :128 `actor VectorStore` | the kit's single consumer-facing surface +- :327 `static let schemaDeclaration` | "vectors" table v3; UNIQUE(item_id, vector_index, model_id) == VectorRecordKey minus modelVersion +- :382 `static defaultSidecarURL(for:)` | `.sqlite` → `.vectors.vec`; nil for non-file backends +- :165 `mihThreshold: UInt32 = 50_000` (default) | promotion boundary, overridable at init +- :172 `mihBandCount: MIHBandCount` (default .m16) | pinned per §1.6 for 50k default threshold +- :185/:189/:197 `bruteForceIndex` / `mihIndex` / `hotIndex` | both allocated at init; hotIndex swapped by :1661 `_selectIndex()` (no rebuild on swap) +- :283 `floatIndices: [String: FloatBruteForceIndex]` | ONE PER modelID (uniform stride requirement); map-entry presence == "built" flag +- :451 `addVector(itemID:engram:modelID:modelVersion:filedAt:)` | convenience wrapper over addPayload +- :496 `addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` | rejects .int8 (throws int8QuantizationPolicyUndefined); table upsert THEN resident mirror; matches stale slots by (itemID,vectorIndex,modelID) | NOT full key | so modelVersion changes are treated as replacement (secfix/ws2-coredelete hard-delete contract); emits vectorkit.index.insert_latency_ms +- :669 `addPayloads(_:)` | bulk path; rejects batch containing ANY int8 (no partial writes); immediate mode rebuilds both indexes ONCE from final snapshot; emits vectorkit.index.batch_insert_latency_ms +- :874 `beginDeferredIndex()` / :898 `publishResidentIndex()` | bulk-burst mode: appends skip index rebuild until publish (O(N) not O(N²)); corpus ingest drain wraps a burst in this pair +- :253/:943 `deferredPendingRecords` / `_flushDeferredPending()` | memory-only-path back-pressure valve, capped at `deferredPendingLimit` (default 50_000, ctor param) | secfix/punt-vector unbounded-buffer fix +- :993 `flush()` | persists pending sidecar write-behind mutation +- :1052/:1066/:1088 `getVector` / `getPayload` / `vectors(forItemID:)` | read paths, all decode via :1803 `decodePayload` / :1840 `storedVector` +- :1134 `findNearest(probe:modelID:limit:)` | lazy `_ensureIndexBuilt()`, delegates to hotIndex.search, NO re-sort (engine already ordered); emits vectorkit.search.latency_ms + .result_count +- :1223 `findNearestFloat(probe:modelID:limit:)` | lazy per-model FloatBruteForceIndex build via :1558 `_ensureFloatIndexBuilt`; quantizes cosine distance ×10_000 rounded for cross-language integer comparison +- :1294 `findFarthestFloat(probe:modelID:limit:)` | same as findNearestFloat but calls searchFarthest; anti-similarity +- :1337 `findByKeyword(_:limit:)` | substring LIKE on item_id; NOT full BM25 (CorpusKit's job); emits vectorkit.search.keyword_result_count +- :1377 `deleteVector(itemID:modelID:)` / :1386 `deleteAllVectors(itemID:modelID:)` | both flush pending deferred burst FIRST if dirty, then delete+tombstone; deleteAllVectors invalidates that model's float index (lazy rebuild) +- :1442 `destroyAllVectors()` | full wipe: table+both binary indexes+sidecar+ALL float indices; used by GeniusLocusKit estate teardown +- :1487 `_ensureIndexBuilt()` | idempotent; sidecar trusted iff `snap.liveCount == tableCount` (live-vs-live, NOT snap.count vs table | avoids spurious rebuild after deletes, "C5 fix") +- :1558 `_ensureFloatIndexBuilt(modelID:)` | nil return (no cache) when model has zero float rows, so a later first-ingest can still build a real index +- :1736 `_deleteAndTombstone(itemID:vectorIndex:modelID:)` | scans ALL slots matching (itemID,vectorIndex,modelID) across modelVersions, tombstones every match (no break-after-first) | hard-delete contract +- :1803 `decodePayload(from row:)` | int8 rows ALWAYS decode to nil (symmetric fail-closed read guard, even for hand-crafted rows); guards every Int64→UInt8/UInt32 narrowing conversion against trap + +## INVARIANTS / GOTCHAS + +- I-7 ABSOLUTE: zero Hamming/XOR/popcount arithmetic anywhere in this package outside EngramLib calls. BruteForceIndex, MIHIndex, MaxSimScorer all delegate every distance to EngramLib.distances / EngramLib.Session.distances / EngramLib.distance. A raw popcount anywhere is a conformance violation. +- I-4 ABSOLUTE: cross-model vector comparison is forbidden. Every write carries modelID+modelVersion; every search is scoped to one modelID; float indices are one-per-model because different models emit different dimensions; the resident binary partition index scopes searches by modelID. +- Determinism boundary: binary lane (.hamming) is four-way bit-identical (Swift/Rust × platforms), gated by EngramLib's kernel. Float lane (.cosine/.l2/.dot) is reproducible-within-config ONLY | NOT four-way. Do not add a conformance test asserting float bit-identity; it will be correctly flagged as testing an undocumented, unintended guarantee. +- MIHIndex MUST equal BruteForceIndex output bit-for-bit on every input | this is a BLOCKER gate (MIHIndexTests.swift), not a best-effort property. BruteForceIndex is the oracle; never "fix" MIH by relaxing the gate. +- int8 is REJECTED fail-closed at both write (addPayload/addPayloads throw int8QuantizationPolicyUndefined) and read (decodePayload returns nil for kind==.int8). The case, field, and guards stay until a quantization policy is ratified | do not remove the guard as "dead code." +- Stale-slot matching in addPayload/addPayloads uses (itemID, vectorIndex, modelID) | NOT the full VectorRecordKey | specifically so a modelVersion change is recognized as replacement, not a new sibling slot. Matching by full key here is the recurring bug shape (secfix/ws2-coredelete); do not "simplify" to full-key equality. +- VectorRecordKey ordering (itemID, vectorIndex, modelID, modelVersion) is load-bearing: it is the resident-array partition order, the universal search tie-break, and the sidecar's on-disk key ordering. Do not reorder the Comparable fields. +- MIHBandCount is restricted to {4,8,16,32} by the enum itself (§1.7 conformance restriction) | sub_bits ∈ {64,32,16,8} guarantees no word-straddle in extractBand. Do not add m=2 or other values without a new word-straddle code path and a separate conformance harness. +- MIHIndex's enumeration-budget guard falls back to a full O(n) bruteScan (still EXACT, same heap, same distances) rather than hang on sparse/adversarial data. cumulativeChoose's saturating-overflow arithmetic MUST stay bit-identical to the Rust port's saturating_add so both fall back at the same radius. +- ResidentArrayStore sidecar is a REGENERABLE CACHE, never a second source of truth. The `vectors` table is authoritative. Staleness check compares live-vs-live counts (format 0x0002); comparing total slot counts instead (pre-C5-fix behavior) spuriously rebuilds after every delete. +- Sidecar single-add writes are WRITE-BEHIND (appendDeferred + isDirty): VectorStore.addPayload does not force a disk write per call. Callers relying on immediate on-disk durability of the sidecar must call `flush()`; crash safety does not depend on this because the table write already happened synchronously before the mirror. +- deferredIndexActive (beginDeferredIndex/publishResidentIndex) turns a bulk import from O(N²) to O(N) index rebuilds. deleteVector/deleteAllVectors both force-publish a dirty deferred window before deleting, so a delete never races an unpublished burst. +- deferredPendingRecords (memory-only deferred path) is capped at deferredPendingLimit (default 50,000); exceeding it triggers an intermediate flush that keeps the burst open. This is a back-pressure valve, not an error path | do not treat a flush mid-burst as anomalous. +- Float payload byte order is EXPLICIT little-endian (VectorPayload.init(floats:) / asFloats()), independent of host native endianness | required for the `.vec` sidecar (and any future float sidecar) to be byte-identical across Apple and Linux hosts. +- FloatBruteForceIndex establishes its stride from the FIRST vector added via add(key:vector:); a later vector of different byte count throws rather than corrupting the flat storage buffer. One index = one dimension, always. +- FloatBruteForceIndex is labeled "Lane C" in its own file header but "Lane D" in VectorStore.swift's comments | same type, inconsistent label, not a functional divergence. Do not "fix" one file to match the other without checking whether a Lane-lettering convention doc elsewhere disambiguates it first. +- MaxSimScorer (Lane E1) is NOT wired into VectorStore's public search API in this package version | it is a standalone scorer callers invoke directly with pre-fetched token-Engram arrays. Do not assume `findNearest` performs late interaction. +- Telemetry (Intellectus.report) is off by default; the emitted metrics (vectorkit.index.insert_latency_ms, .batch_insert_latency_ms, vectorkit.search.latency_ms, .result_count, .keyword_result_count, vectorkit.mih.enumeration_fallback) are named constants used by dashboards | renaming any of them is a breaking change to monitoring, not just to code. +- Pinned/default constants | changing any requires updating dependent conformance tests: mihThreshold 50,000, mihBandCount .m16 (sub_bits=16), deferredPendingLimit 50,000, compactionThreshold 0.25, poolSubmitThreshold-equivalent N/A (not used here), sidecar format version 0x0002, MIHBandCount ∈ {4,8,16,32}, cosine-distance quantization scale ×10,000. +- Actor boundaries: VectorStore, BruteForceIndex, MIHIndex, FloatBruteForceIndex, ResidentArrayStore are all actors | all mutation and reads are serialized per actor. ResidentVectorArray, VectorPayload, VectorRecordKey, DenseHit, VectorMatch, StoredVector are plain Sendable value types safe to pass across actor boundaries once constructed. diff --git a/packages/kits/VectorKit/docs/DETAILS.md b/packages/kits/VectorKit/docs/DETAILS.md new file mode 100644 index 0000000..95ce027 --- /dev/null +++ b/packages/kits/VectorKit/docs/DETAILS.md @@ -0,0 +1,703 @@ +--- +doc: DETAILS +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# VectorKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. First comes the module surface and errors. Then the embedding +seam. Then the shared engine foundation types. Then the three search +engines. Then the resident-array storage layer. Last comes the +storage-facing types and the `VectorStore` actor that ties everything +together. + +## VectorKit.swift + +This file provides the module surface. It is a short header comment +naming the kit's public pieces. It defines no types of its own. + +The file explains a boundary worth restating here. VectorKit imports +EngramLib internally to use the `Engram` fingerprint type. It does not +re-export that import. A caller that wants to construct an `Engram` +directly must `import EngramLib` itself. This keeps every package's +import list explicit. It matches the convention used across the rest of +the substrate-dependent kits. + +## VectorKitError.swift + +This file provides `VectorKitError`, the single error type for every +VectorKit operation. Per MOOTx01 convention, errors are concrete named +cases. They are not a generic failure plus a logged message. A caller +can branch on exactly what went wrong. + +`embeddingFailed(String)` reports an inference failure inside an +`EmbeddingProvider`. `modelUnavailable(String)` reports a model that is +not loaded on the current device. `storeUnavailable(String)` reports a +failure opening the backing storage. `notFound` is reserved for a future +throwing read path. Today's read functions return `nil` instead. +`invalidPayload` covers a structurally broken `VectorPayload`. This +means a wrong kind, a wrong byte count, or a dimension mismatch. The +payload's own decode functions throw it. So does every search engine's +input validation. `decodingFailure` covers a malformed key or row that +cannot be decoded. + +Two cases record product decisions rather than plain bugs. +`int8QuantizationPolicyUndefined` is thrown whenever a caller tries to +write an `.int8`, or quantized, vector. The rules for how to quantize, +and later reverse the quantization, have not been agreed on yet. Writing +one now would lock in behavior nobody has approved. The case documents +that this is deliberate and reversible. Once a policy is ratified, the +guards that throw this error are meant to be removed. +`embedFloatVocabMiss` is thrown by embedding providers whose vocabulary +is fixed in advance. An example is a statistical model trained on a +specific word list. The error fires when none of a query's words are in +that vocabulary. It is distinct from `embeddingFailed`, because the +provider is working correctly here. The input simply has nothing in +common with what the provider knows. + +## EmbeddingProvider.swift + +This file provides the `EmbeddingProvider` protocol. It is the single +seam between text and vector that every embedding source in VectorKit's +kit graph implements. + +The protocol is deliberately narrow. It defines a `modelID` and a +`modelVersion`. These are the tags every stored vector must carry, per +spec I-4. This tagging ensures that vectors from incompatible models +are never compared. The protocol also defines three ways to turn text +into numbers. This narrowness is what +lets VectorKit remain agnostic about where inference happens: CoreML, +ONNX, or a hand-written statistical model. A concrete provider, such as +`FloatSimHashEmbeddingProvider` or a sibling package's MiniLM adapter, +fills in the details. + +`embed(_:)` is the primary method. It returns the 256-bit binary Engram +for a piece of text. It throws `embeddingFailed` or `modelUnavailable` +on failure. Every conformer must return the substrate's canonical zero +Engram for an empty string. This is the one input every provider is +guaranteed to agree on. Treating it specially lets empty records from +different providers land on the same identical partition, instead of +colliding by coincidence. + +`embedFloat(_:)` returns the dense float vector a provider computed on +its way to the Engram. This is the vector before it was compressed into +256 bits. The protocol's default implementation simply throws +`embeddingFailed`. This means the float lane is opt-in. A provider with +nothing more precise to offer than the fingerprint declines, rather than +fabricate numbers. Providers that do real inference are defined in a +sibling package. Examples are MiniLM, mpnet, and EmbeddingGemma. They +override this method to return the vector they already computed, at no +extra inference cost. Empty input returns an empty array, never a +vector of zeros. A vector of zeros would look like a legitimate, and +misleading, nearest neighbor to every genuinely empty text. + +`embedPair(_:)` is a convenience for callers that need both outputs from +one text. Its default implementation calls `embed` and then +`embedFloat`, two inference passes. It swallows a float-lane opt-out +into an empty array, so the default matches historical two-call +behavior. A provider might compute both outputs from a single inference +pass. That provider should override this method, to avoid running its +model twice. + +`embedBatch(_:)` embeds several texts. It defaults to a sequential loop. +Providers with genuinely batched inference should override it for +throughput. Output order always matches input order. + +## FloatSimHashEmbeddingProvider.swift + +This file provides `FloatSimHashEmbeddingProvider`, the one concrete +`EmbeddingProvider` VectorKit ships. It is a mirror of the Rust +`vectorkit::FloatSimHashEmbeddingProvider`. + +The type is a thin wrapper. It holds a `modelID`, a `modelVersion`, a +`projectionSeed`, and an injected inference closure. That closure has +the shape `(String) async throws -> [Float]`, and the host that owns the +actual model supplies it. VectorKit itself never loads a model or owns a +tokenizer. Concrete text providers that do live in a different package, +CorpusKitProviders. Examples are MiniLM, mpnet, and EmbeddingGemma. They +conform to this same protocol, using this type as their low-level +building block. + +`embed(_:)` short-circuits to the canonical zero Engram for empty input. +It does this before ever calling the inference closure. This guarantees +the empty-string contract holds, even if the closure itself would have +produced something non-zero. Otherwise it calls the inference closure. +It then passes the result through +`SubstrateML.FloatSimHash.project(vector:seed:)`. This is a shared, +conformance-gated substrate primitive. It projects an arbitrary-length +float vector down to a 256-bit fingerprint using a technique called +SimHash. The `projectionSeed` matters here. Different seeds turn the +same float vector into different fingerprints. This is what keeps two +different models' outputs from accidentally landing in the same +fingerprint space and being wrongly compared. This is spec I-4's rule, +enforced at the projection layer as well as at storage. + +`embedFloat(_:)` returns exactly the float vector the inference closure +produced. This is the same vector `embed(_:)` feeds into the projection. +So a caller using both lanes never pays for two inference passes. + +## Engine/VectorRecordKey.swift + +This file provides `VectorRecordKey`, the identifier every stored +vector carries inside the search engines. Note that this file lives +under `Engine/`, VectorKit's internal folder for the search-engine +machinery shared by every index implementation. + +A key used to be a two-part pair of item and model. That design assumed +one vector per item per model. The assumption breaks for models like +ColBERT that produce one small vector per word instead of one vector +per document. So the key grew a third field. `itemID` names the owning +record, a drawer or a text chunk, as a UUID string. `vectorIndex` is the +position of this vector within its item's sequence. It is `0` for the +ordinary single-vector case, and `0` through `N-1` for a multi-vector +item. `modelID` and `modelVersion` complete the tuple. Vectors from +different model versions are never comparable, per spec I-4. + +`VectorRecordKey` is `Comparable`. It is ordered lexicographically by +`(itemID, vectorIndex, modelID, modelVersion)`. This order is not +incidental. It is the tie-break every search result uses when two +matches land at the same distance. It is also the order the on-disk +resident array is built in. So both the search output and the storage +layout agree on what sorted means. + +## Engine/VectorPayload.swift + +This file provides `VectorPayload`, the one envelope every vector's raw +bytes travel in. It also provides `VectorKind` and `VectorPayloadInput`. +`VectorKind` tags which numeric family the bytes represent. +`VectorPayloadInput` bundles a payload with its storage metadata, for +bulk writes. + +`VectorKind` has three cases. `.binary`, raw value `0`, is exactly the +32-byte Engram wire form. `.float32`, raw value `1`, is `dim × 4` bytes +of IEEE-754 numbers. `.int8`, raw value `2`, is reserved for a future +quantized representation. The case exists so that a future ratified +quantization policy does not require a new wire format. `VectorStore` +currently rejects every `.int8` write. See +`VectorKitError.int8QuantizationPolicyUndefined` above. The raw values +are stored on disk and must never be reordered. + +`VectorPayload.init(engram:)` builds a binary payload directly from an +Engram's wire bytes. There is no conversion and no loss, so every +existing binary test still holds. `init(floats:)` serializes a `[Float]` +to little-endian IEEE-754 bytes explicitly, byte by byte, rather than +relying on the platform's native byte order. This is a deliberate +choice, and not an oversight. It is what lets the on-disk `.vec` sidecar +be read back identically on an Apple device and on a Linux server. The +two platforms may not share the same native byte order convention. +`asEngram()` and +`asFloats()` reverse these conversions. They throw `invalidPayload` when +the payload's kind or byte count does not match what was asked for. + +## Engine/DenseHit.swift + +This file provides `DenseHit`, the one result shape every search engine +returns. It also provides `LaneTag`. This is an enum naming which +retrieval technique produced a given score. Fusion and multi-technique +retrieval code outside this package uses it. + +A `DenseHit` carries the matched `key`, a `rawDistance`, and the +`metric` that produced it. The tricky design point is that `rawDistance` +is a single `Int32` field shared by two very different kinds of number. +It can hold an integer Hamming distance for the binary lane. It can also +hold the bit pattern of a `Float` distance for the float lane. Two +computed properties translate it back. `hammingDistance` simply casts +it to `Int`. This is safe because Hamming distances are always in the +range 0 to 256. `floatDistance` reconstructs the `Float` from its stored +bit pattern. Packing both families into one field has one payoff. Code +elsewhere in the kit graph can handle a `[DenseHit]`. It need not care +which lane produced it. This design avoids giving each lane its own +result type. + +The file's header calls out an additive-only rule. Any future field +added here must have a default value. This lets existing callers who +build a `DenseHit` with the current initializer keep compiling. This +matters because both a Swift and a Rust version of this type exist. The +two must stay in lockstep. + +## Engine/DenseMetric.swift + +This file provides the metric vocabulary for the whole engine seam. It +defines `BinaryMetric`, with cases `.hamming` and `.jaccard`. It defines +`FloatMetric`, with cases `.cosine`, `.l2`, and `.dot`. It defines +`DenseMetric`, the umbrella enum wrapping either family. This means +`DenseIndex.search` needs only one metric parameter, regardless of which +lane it routes to. + +The file's real content is its documentation of a determinism boundary. +This boundary recurs throughout this package. `.binary(.hamming)` is +four-way bit-identical, because it is pure integer arithmetic computed +by a shared, conformance-gated kernel. `.binary(.jaccard)` is +bit-identical through its two integer counts. It ends with one final +IEEE-754 division, which is itself guaranteed identical, because +IEEE-754 mandates exact rounding for basic operations. Every `.float(_)` +metric is reproducible only within one build and platform. It is never +guaranteed identical between Swift and Rust. This is stated as a +documented property of floating-point math. It is not a defect to be +fixed. It is a warning aimed at a future reviewer. That reviewer might +otherwise try to force float parity that the underlying arithmetic +cannot honestly provide. + +## Engine/DenseIndex.swift + +This file provides the `DenseIndex` protocol. This is the single seam +that lets `VectorStore` treat three very different search engines, +`BruteForceIndex`, `MIHIndex`, and `FloatBruteForceIndex`, as +interchangeable. The file also provides three supporting types: +`IndexKind`, `SearchDirection`, and `MetadataFilter`. `IndexKind` is a +tag naming which implementation is behind a given index. Tests use it +to pick the brute-force oracle deliberately. + +`SearchDirection` has two cases. `.nearest` is most similar first, and +is the default. `.farthest` is most dissimilar first. Farthest search +supports a query for things unlike this one. It is not a trick of +negating a nearest-neighbor list. The farthest items are not among the +nearest top-k at all. So the index has to scan and sort toward the +opposite end. It uses the exact same distance calculation, only the +opposite sort direction. + +`MetadataFilter` restricts a search to one `modelID` and, optionally, +one `modelVersion`. Its `accepts(_:)` method is the single predicate +every engine calls per candidate. A `nil` field acts as a wildcard. + +The protocol itself declares four operations: `build(from:)`, +`search(probe:metric:k:filter:)`, `add(key:vector:)`, and +`remove(key:)`. It documents the contract every conformer must honor. +Results come back sorted by distance ascending. Ties break by the +matched key ascending. `BruteForceIndex` is the correctness oracle every +other binary engine is measured against. + +## Engine/BruteForceIndex.swift + +This file provides `BruteForceIndex`. This is the exact linear-scan +search engine for binary vectors, using Hamming distance. It is also +the conformance oracle every other binary engine is checked against. +Today that is just `MIHIndex`. + +The file is built around one hard rule, restated three times in its +comments. It performs zero Hamming arithmetic itself. Every distance is +computed by `EngramLib.distances`. This routes to a shared kernel, +selected once per process. The kernel uses NEON on Apple silicon +hardware, and a scalar fallback elsewhere. It is checked for identical +output across four build configurations. Reimplementing a bitwise +XOR-and-count here, even a correct one, would bypass that check. It +would risk silent divergence between platforms. This is the file's +version of spec I-7. + +`search(probe:metric:k:filter:)` validates the probe, which must be +exactly 32 bytes of `.binary` kind, and the metric. Only +`.binary(.hamming)` is supported, and other requests throw +`invalidPayload`. It narrows the scan to one model's slot range when a +filter is present. This is an `O(log m)` lookup into a sorted partition +index. The lookup avoids a full-array walk. It collects the live, +un-tombstoned +candidates in that range and hands their Engrams to +`EngramLib.distances` in one batch call. It deliberately avoids +`EngramLib.findNearest`, which applies a different, insertion-order +tie-break. This file sorts the returned distances itself, by distance +ascending and then key ascending. The engine's own contract requires the +full `VectorRecordKey` as the tie-break, not just the array's insertion +position. Otherwise two records under the same item but different model +or vector index could be returned inconsistently. + +`add(key:vector:)` implements upsert by tombstoning any existing slot +with the same key before appending the new bytes. It then rebuilds the +sorted model-partition index from the updated key list. `remove(key:)` +tombstones every matching slot without touching the underlying storage +bytes. Actual space reclamation is `ResidentArrayStore`'s job, not this +type's. `currentSnapshot()` returns a value-type copy of the live array. +This lets callers outside the actor, chiefly `VectorStore` when it needs +to scan for tombstoning, read it safely. + +## Engine/FloatBruteForceIndex.swift + +This file provides `FloatBruteForceIndex`. This is the linear-scan +search engine for the float32 lane. It supports three float metrics: +cosine, Euclidean, and dot-product distance. The code calls Euclidean +distance `l2`. Unlike the binary lane, this is both the correctness +reference and the production search path. There is no separate +accelerated float engine in this package. + +The file opens with an emphatic warning, repeated in this document +because it protects against a plausible but wrong fix. Float arithmetic +here is reproducible on one build and platform. It is not, and cannot be +made to be, bit-identical between Swift and Rust or across different +hardware. This is a documented property of IEEE-754 arithmetic, not an +oversight. A reviewer must not try to force it to match the binary +lane's four-way guarantee. + +`build(from:)` simply stores a reference to the supplied array. There is +no secondary structure to construct, so building is `O(1)`. The real +cost is whatever the caller paid to assemble the array. +`search(probe:metric:k:filter:)` validates three things. The probe must +be `.float32`. The requested metric must be a float metric. The probe's +byte count must match both its own declared dimension and the array's +fixed stride. A mismatch would otherwise read past the end of a slot, so +it throws instead. It then scans every live, filter-passing slot, computing +one of three distances per candidate. Cosine distance treats a zero +vector as maximally distant rather than crashing on a divide-by-zero. +`l2` is the plain Euclidean formula. `dot` is negated so that smaller is +nearer holds for every metric uniformly. The scan sorts ascending by +distance, then by key. `searchFarthest(probe:metric:k:filter:)` reuses +the identical scan and identical distance math. It changes only the sort +direction to descending. This is what makes finding dissimilar items a +real bottom-of-the-list scan, rather than a negated top-of-the-list one. + +`add(key:vector:)` establishes the array's dimension from the first +vector added. It rejects any later vector of a different byte count. A +mismatched stride would silently corrupt the flat storage buffer. +`remove(key:)` tombstones the matching slot. Actual compaction +happens the next time `build(from:)` runs with a freshly assembled +array. + +## Engine/MaxSimScorer.swift + +This file provides `MaxSimScorer` and `MaxSimHit`. Together these form +the exhaustive, or Exact-A, implementation of ColBERT-style +late-interaction scoring over binary token fingerprints. + +Some embedding techniques represent one document as many small vectors. +One vector exists per word or token, not one vector for the whole +document. Comparing two such documents means asking one question, for +every word in the query. Which word in this document is most like it. +The scorer then adds up +those best matches. That sum, `Σ (256 − minimum Hamming distance)` over +every query token, is the MaxSim score this file computes. It examines +every query token against every document token for every candidate +document. So it never skips a candidate. This exhaustiveness is +precisely what makes it the correctness reference for any faster, pruned +variant built later. The file's header explicitly reserves the +accelerated two-stage variant as out of scope here. + +`score(queryTokens:documents:k:)` iterates the supplied documents in +ascending itemID order. It sorts the dictionary's keys explicitly. A +Swift dictionary's own iteration order is not guaranteed. Relying on it +would make results non-reproducible. It computes each document's MaxSim +score. It sorts the results by score descending, then itemID ascending. +It truncates to `k` only after the full sort, never before. So a +document that would have scored well is never cut for appearing late in +an unsorted pass. Every Hamming distance again goes through +`EngramLib.Session.distances`. This session is constructed once per +`MaxSimScorer` and reused for the whole call. The one-time cost of +picking the fastest available kernel is paid once, rather than per +comparison. + +## Engine/ResidentVectorArray.swift + +This file provides `ResidentVectorArray`, the packed in-memory data +shape every search engine reads from. It also provides +`ModelPartitionEntry`, one entry in its per-model index. This is the +shared contract underneath `BruteForceIndex`, `MIHIndex`, and +`FloatBruteForceIndex`. All three read the identical layout. This is +what lets `VectorStore` build the array once and hand it to whichever +engine is currently active. + +The design reason is stated directly in the file's header comment. +Measurement on the pre-existing code path found a bottleneck. Fetching +and decoding rows from the database consumed eighty-seven percent of a +search's latency. The actual distance kernel took under one percent. A +fixed-stride, contiguous byte array removes the fetch-and-decode cost +from every query after the first. The whole array is loaded once. It is +then scanned as a flat block of memory with no per-row allocation. + +The type stores `kind` and `stride`, the bytes per vector slot: thirty-two +for binary, `dim × 4` for float32. It stores `count`, including +tombstoned slots, and `storage`, the packed bytes themselves. On Apple +platforms, `storage` may be memory-mapped read-only from a sidecar file. +It also stores a `keys` array parallel to `storage`, a sorted +`modelPartitions` index, and a `tombstones` bitmap. `liveCount` walks the +tombstone bitmap to compute how many slots are still valid. +`partitionRange(for:)` binary-searches the sorted partitions to find one +model's slot range in `O(log m)`. `isTombstoned(_:)` and +`vectorBytes(at:)` are the two per-slot accessors every engine's scan +loop calls. + +## Engine/ResidentArrayStore.swift + +This file provides `ResidentArrayStore`, the actor that owns the +optional on-disk `.vec` sidecar file. This file is a packed binary cache +of a `ResidentVectorArray`. It lets a reopened store skip rebuilding the +array from every database row. + +The file documents its own on-disk format in full. A fixed header holds +magic bytes, format version, vector kind, stride, count, a live-slot +count, and the tombstone bitmap. This header is followed by the packed +vector bytes, then variable-length key records, then the +model-partition index. Every multi-byte integer is explicitly +little-endian. So the same file reads identically on an Apple device +and on a Linux server. `writeSidecar` writes to a temporary file and +atomically renames it into place. So a crash mid-write never leaves a +half-written sidecar behind. `readSidecar` memory-maps the file where +the platform supports it. This is a load-time optimization, not a +difference in the bytes returned. `parseSidecar` does the actual +decoding. It +checks every length field against the remaining buffer size before +trusting it. So a corrupted or hand-edited file is rejected with a +`decodingFailure`, rather than crashing the process. + +The file is explicit about one policy. The `vectors` database table is +always the durable source of truth. This sidecar is a regenerable cache, +and never a second copy of record. `load()` reads the sidecar if +present. If it is missing or fails to parse, the store simply starts +empty. It then waits for `VectorStore` to rebuild it from the table. + +Three write paths exist. A single write policy could not serve both a +low-latency single insert and a large bulk import well. +`append(key:bytes:)` is the eager path. It writes the sidecar +immediately after every single addition. `appendDeferred(key:bytes:)` is +the write-behind path a single insert uses in production. It updates the +in-memory array and marks the store dirty, without touching disk. It +trusts the caller to call `flush()` at a natural pause. This is safe +because the database row was already written durably before this call. +Losing an unflushed sidecar only costs a rebuild on the next open, never +data. `appendBatch(records:)` is the bulk-import path. It extends +storage, keys, and the tombstone bitmap for the whole batch in one pass. +It writes the sidecar exactly once, so importing a thousand vectors +costs one disk write instead of a thousand. + +`compact()` rewrites the sidecar, keeping only live, non-tombstoned, +records. It sorts them by key for a deterministic, reproducible layout. +This is triggered automatically whenever the tombstone ratio exceeds +`compactionThreshold`, twenty-five percent by default, after any eager +write. + +## StoredVector.swift + +This file provides `StoredVector`. This is the public, fully decoded row +shape that `VectorStore.vectors(forItemID:)` returns. Callers want this +convenient binary form, rather than the raw typed payload. + +Its fields mirror the `vectors` table's columns directly. The fields +are `id`, `itemID`, `vectorIndex`, `modelID`, `modelVersion`, `engram`, +and `filedAt`. The `id` is a stable value assigned on insert. The +`itemID` is the owning record. The `vectorIndex` is the position within +a multi-vector item. The `modelID` and `modelVersion` are spec I-4's +tags. The `engram` is the decoded fingerprint. The `filedAt` field is +the time the row was written. This last field is round-tripped through +the database's text-based ISO 8601 timestamp column. That column loses +sub-millisecond precision. The file notes this explicitly, so a caller +comparing timestamps at fine granularity is not surprised. +`StoredVector.engram` is non-nil only for binary rows. A float or int8 +row must be read through `VectorStore.getPayload` instead. This +convenience type only round-trips the binary case. + +## VectorMatch.swift + +This file provides `VectorMatch`, the public search-result shape +`VectorStore.findNearest` and its float-lane counterparts return. + +It carries `itemID`, `distance`, and `modelID`. The `itemID` is the +matched record. The `modelID` lets a caller confirm which model +actually produced the match. `distance` is a Hamming distance for the +binary lane, an integer from zero to two hundred fifty-six. For the +float lane it is a scaled, quantized cosine distance, explained in +`VectorStore`'s float-search functions. +`VectorMatch` conforms to `Comparable`. It is ordered by `distance` +ascending, with ties broken by `itemID` ascending. This is the same +universal tie-break rule used throughout the engine layer. So a sorted +array of matches reads nearest to farthest, from front to back. A +caller need not know the sort convention. + +## VectorStore.swift + +This file provides `VectorStore`, the actor every consumer of VectorKit +actually talks to. It is the largest file in the package. Every other +piece is wired together here into one consistent API. This includes the +durable `vectors` table, the resident arrays, the three search engines, +and telemetry. + +### Storage and schema + +`VectorStore` wraps a PersistenceKit `Storage` backend. Supported +backends include SQLite and an in-memory test backend. A PostgreSQL +backend may be added in the future. The kit never sees which backend is +chosen. That decision belongs to the application. `schemaDeclaration` is +the static schema description passed +to `storage.open(schema:)` before the store is used. It describes one +`vectors` table, version three. Its `UNIQUE(item_id, vector_index, +model_id)` constraint is exactly `VectorRecordKey` minus `modelVersion`. +This constraint makes an upsert on a changed model version a true +replacement of the old row. It is not treated as a duplicate. + +### Two hot-path structures kept in sync + +Every write updates three things together: the durable `vectors` table +row, the in-memory resident array, and the on-disk cache. The resident +array updates through `bruteForceIndex` and `mihIndex`. The on-disk +cache updates only when a sidecar was configured at construction. Both +`bruteForceIndex` and `mihIndex` are always kept current. Only one is +ever the active `hotIndex`. +`_selectIndex()` compares `liveBinaryCount` against `mihThreshold`, +fifty thousand by default, after every write that changes the count. It +swaps `hotIndex` between `bruteForceIndex` and `mihIndex` with a plain +reference assignment. No rebuild is needed on promotion or demotion, +because both indexes were already current. +`init(storage:sidecarURL:mihThreshold:mihBandCount:deferredPendingLimit:)` +allocates both index actors up front. So this swap never needs to +construct anything at query time. + +### Write path + +`addVector(itemID:engram:modelID:modelVersion:filedAt:)` is a +convenience wrapper for the common single binary vector case. It builds +a `VectorPayload` and delegates to `addPayload`. +`addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` +is the general write. It rejects `.int8` payloads immediately. See +`VectorKitError` above for that case. It writes the row through an +upsert keyed on the table's unique constraint. It then mirrors the +write into the matching resident array, but only for `.binary` and +`.float32` kinds. For a binary write, `addPayload` first finds any +existing slot at the same logical position. This position is defined by +`itemID`, `vectorIndex`, and `modelID`. The match ignores `modelVersion`. +This lets a version change be treated as a true replacement, rather +than leaving a stale duplicate slot behind. This matching is +deliberately looser than full key equality, specifically to catch that +case. It then tombstones the stale slot, if any, and appends the new +one in both +`bruteForceIndex` and `mihIndex`. It updates `liveBinaryCount` only when +the write was genuinely new, and calls `_selectIndex()`. Every write +emits a `vectorkit.index.insert_latency_ms` telemetry metric through +IntellectusLib. This metric is a short-circuited no-op unless monitoring +has been explicitly turned on. So the cost on the default path is one +boolean check. + +`addPayloads(_:)` is the bulk-import counterpart. Its whole reason for +existing is complexity. Importing N vectors one at a time through +`addPayload` costs N sidecar rewrites and, without care, N index +rebuilds. This function upserts every row to the table, unavoidable +because the table is the durable source. It then rebuilds both binary +indexes exactly once, from the final merged array rather than once per +row. This cuts the amortized cost from `O(N²)` bytes written to `O(N)`. +`beginDeferredIndex()` and `publishResidentIndex()` extend this further +for very large or multi-call bulk imports. While a deferred window is +open, `addPayloads` appends to storage but skips the index rebuild +entirely. It seeds an in-memory tracked set of live keys, so replacement +detection stays cheap across the whole window. `publishResidentIndex()` +performs the single rebuild the whole burst needed, once, when the +caller signals the burst is finished. `deferredPendingRecords` is +capped at `deferredPendingLimit`, fifty thousand by default. This cap +guards the memory-only variant of this path against unbounded growth. +This applies if a caller holds the window open indefinitely. Crossing +the cap +triggers `_flushDeferredPending()`, an internal intermediate merge that +keeps the deferred window open for the caller, while bounding peak +memory. + +### Search path + +`findNearest(probe:modelID:limit:)` is the binary-lane search. It +lazily builds the resident array on first use. It builds this from the +sidecar if one is current, or from the table if not. It then delegates +entirely to `hotIndex.search`, converting the returned `[DenseHit]` +into `[VectorMatch]` without re-sorting. The engine has already applied +the required order of distance ascending, then itemID ascending. +`findNearestFloat(probe:modelID:limit:)` is the float-lane equivalent. +It lazily builds a `FloatBruteForceIndex` per model. The map's presence +is the already-built flag. There is no separate boolean. It then +searches by cosine distance. It quantizes the resulting `Float` distance +to an integer, by multiplying by ten thousand and rounding. So +results from different languages' fixtures can be compared exactly, +rather than approximately. `findFarthestFloat(probe:modelID:limit:)` is +identical, except it calls the engine's farthest-ranking search, for an +anti-similarity query for things unlike this one. +`findByKeyword(_:limit:)` is a plain substring match on `item_id`. It is +explicitly documented as a quick pre-filter for hybrid retrieval, and +not a real keyword search. Full keyword scoring is a different +package's responsibility. + +### Delete path + +`deleteVector(itemID:modelID:)` deletes the row at `vectorIndex` zero +and tombstones every matching resident slot. +`deleteAllVectors(itemID:modelID:)` deletes every vector index for an +item and model. This is used when a multi-vector item, such as all of a +ColBERT item's token vectors, needs complete removal. Both functions +first flush any in-flight deferred-index burst, so a delete never races +an unpublished bulk import. `destroyAllVectors()` wipes the entire +store: every row, both resident indexes, the sidecar, and the per-model +float indexes. This runs as part of a coordinated estate teardown. An +estate is one user's complete memory store in MOOTx01. + +### Coherence helpers + +`_ensureIndexBuilt()` is the one-time, per process, function that +populates both binary resident indexes. It trusts a sidecar when its +recorded live-slot count matches the table's live binary-row count. If +the counts disagree, it rebuilds from the table and rewrites the +sidecar. Comparing live count to live count avoids a spurious full +rebuild after ordinary deletions leave tombstoned slots behind. The +older approach compared total slot counts instead. +`decodePayload(from:)` and `storedVector(from:)` are the row-decoding +functions every read path shares. Both explicitly guard every narrowing +integer conversion, for example a negative `dim` or an out-of-range +`kind` byte. So a hand-crafted or corrupted row is rejected with `nil`, +rather than crashing the process on a Swift trap. + +## Rust Port and Conformance + +The `rust/` directory mirrors the Swift implementation file for file. +It provides `vector_store.rs`, alongside `engine/brute_force.rs`, +`engine/mih.rs`, `engine/float_brute_force.rs`, `engine/max_sim.rs`, +`engine/resident.rs`, `engine/resident_store.rs`, `engine/key.rs`, +`engine/payload.rs`, `engine/hit.rs`, `engine/metric.rs`, +`engine/seam.rs`, plus `embedding_provider.rs`, +`simhash_embedding_provider.rs`, and `error.rs`. Three things are +specified precisely enough in the Swift source comments: the `.vec` +sidecar format, the MIH band-hashing algorithm, and the MaxSim scoring +algorithm. This includes colex enumeration order, the enumeration-budget +guard's integer arithmetic, and the little-endian sidecar layout. Both +ports are expected to agree exactly on the binary lane. `rust/tests/` +holds integration suites for bulk ingest, the float lane, int8 +rejection, the SimHash provider, the vector store, and telemetry. These +suites exercise the same behaviors described above. The package's own +`MIHIndexTests.swift` gates `MIHIndex` against `BruteForceIndex` +directly within Swift. Cross-language conformance for the binary lane +rests on both ports implementing the same documented algorithm. It does +not rest on a single shared fixture file. This differs from +LatticeLib's shared JSON fixtures. The float lane is, by design, exempt +from cross-language +bit-identity. See `DenseMetric.swift` and `FloatBruteForceIndex.swift` +above. Only within-platform reproducibility and rank correctness are +asserted for it, in both languages. diff --git a/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 8d4a0af..0000000 --- a/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,79 +0,0 @@ -# VectorKit Interface Doctrine - -For coding agents using VectorKit in product code or downstream kits. - -## 1. The kit is PersistenceKit-backed - -VectorKit's storage is `any Storage` from PersistenceKit. The application picks the backend (InMemory for tests, SQLite for Apple-platform deployment, PostgreSQL for server-side). VectorKit does not open SQLite directly any more. - -```swift -let storage = SQLiteStorage(configuration: EstateConfiguration( - estateID: estate.id, - backend: .sqlite(url: dbURL) -)) -try await storage.open(schema: VectorStore.schemaDeclaration) -let store = VectorStore(storage: storage) -``` - -A kit consuming VectorStore receives `any Storage` from its caller. It does not pick a backend. - -## 2. Engrams are tagged - -Every vector stored carries `(modelID, modelVersion)` per spec I-4. Cross-model comparisons are forbidden. `findNearest(probe:modelID:limit:)` filters to a single model; the caller is responsible for ensuring the probe was produced by that same model. - -```swift -let probe = try await provider.embed("query text") -let matches = try await store.findNearest( - probe: probe, - modelID: provider.modelID, - limit: 10 -) -``` - -## 3. Use FloatSimHash for new providers - -External embedding providers (MiniLM, BERT, EmbeddingGemma) project dense float vectors through `SubstrateLib.FloatSimHash` to produce 256-bit Engrams. Each provider picks a stable, deterministic seed so its fingerprints are model-tagged independent of vector content. - -```swift -private static let providerSeed: UInt64 = 0x... // unique per model -let engram = FloatSimHash.project(vector: pooledFloats, seed: providerSeed) -``` - -Do not invent ad-hoc projections. Sign-bit folding, hash-into-bits, etc. are gone from the kit graph; they did not preserve cosine similarity faithfully. FloatSimHash does (cookbook section 3.6). - - -VectorKit currently embeds a deterministic stand-in tokenizer inside `MiniLMProvider` so the provider can ship before CorpusKit lands. After mission 7, tokenization moves to CorpusKit and providers consume it through CorpusKit's tokenizer protocol. Do not extend the MiniLMProvider tokenizer; it is internal and migrating. - -If you need text tokenization in a non-MiniLM context today, wait for CorpusKit. If you need it before CorpusKit lands, file a decision record. - -## 5. Hybrid retrieval composes elsewhere - -VectorKit's `findByKeyword` is a coarse substring filter on drawer IDs, kept for backward compatibility with hybrid-retrieval callers that need a fast pre-filter. Full BM25 keyword search with proper tokenization lives in CorpusKit. Composition of vector + keyword scoring (RRF, MMR) lives in NeuronKit's hybrid recall (paper section 10.2). VectorKit stays focused on vector primitives. - -## 6. The schema is owned by VectorStore - -`VectorStore.schemaDeclaration` is a public static. Callers pass it to `Storage.open(schema:)` before constructing the store. The schema declares one table (`vectors`) with the columns and constraints documented inline. Do not reach into the table directly from outside the kit; go through VectorStore's API. - -If you need a column VectorStore does not expose, file a decision record proposing the schema change. Schema version bumps follow the migration rules in PersistenceKit's INTERFACE_DOCTRINE. - -## 7. findNearest is currently a linear scan - -The v1 implementation scans every row in the `vectors` table for the given `modelID` and computes Hamming distance in memory. This matches the pre-refactor SQLite behavior. The migration to PersistenceKit's `VectorIndex` protocol with sqlite-vec or pgvector ANN is a follow-on; it does not change the public API. - -If your call sites need sub-linear retrieval today, gate them on corpus size or fall through to a higher-tier path. P99 on a 10k-row corpus is roughly 50ms on Apple Silicon. - -## 8. Sendable everywhere - -VectorStore is an actor. Its public methods are async. EmbeddingProvider is `Sendable`. StoredVector and VectorMatch are `Sendable` value types. If you wrap VectorKit in another kit, that kit's public surface stays `Sendable` too. - -## 9. When in doubt, file a decision record - -If you find yourself wanting to: - -- Bypass the model tag (modelID, modelVersion) on stored vectors -- Add a new projection besides FloatSimHash -- Expose VectorStore internals (the underlying Storage, the schema, the table name) -- Add a kit dependency from VectorKit on something other than SubstrateLib, EngramLib, or PersistenceKit -- Cross the kit boundary into CorpusKit-shaped tokenization concerns - -Stop. Write a decision record in `docs/decisions/` proposing the change. The kit boundaries are deliberate. diff --git a/packages/kits/VectorKit/docs/OVERVIEW.md b/packages/kits/VectorKit/docs/OVERVIEW.md new file mode 100644 index 0000000..4aaccfc --- /dev/null +++ b/packages/kits/VectorKit/docs/OVERVIEW.md @@ -0,0 +1,218 @@ +--- +doc: OVERVIEW +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# VectorKit Overview + +## What This Library Does + +VectorKit turns a piece of text into a vector. A vector is a fixed-size +code that stands in for the text's meaning. VectorKit stores that vector. +A later query can then find the most similar ones. MOOTx01 is an +on-device AI memory system. It stores what an AI observes over time. It +helps the AI recall this later. VectorKit is the part of MOOTx01 that +answers one question: which stored memories are most like this one. + +VectorKit stores two kinds of vector for the same piece of text. The +first is a 256-bit binary fingerprint called an `Engram`. A sibling +library, EngramLib, defines this type. A fingerprint is a short +fixed-size code computed from a piece of content. Similar content +produces similar fingerprints. The system can then compare things +quickly without reading them in full. Two Engrams are compared by +Hamming distance. This is the count of bit positions where they differ. +A smaller count means the two are more similar. The second kind is a +dense float vector. This is a list of several hundred decimal numbers. +An embedding model, such as MiniLM, produces this list directly. +VectorKit keeps both kinds because they serve different needs. The next +section explains why. + +## The Problem It Solves + +An AI memory system must answer one question: find memories like this +one. It must do so without sending private text to a server. VectorKit +runs entirely on the device that captured the memory. + +A 256-bit fingerprint is compact and fast to compare. Comparing two +fingerprints means counting differing bits. This is pure integer +arithmetic. The comparison never uses floating-point math. So it +produces the exact same answer on every device and every operating +system. VectorKit calls this property four-way determinism. VectorKit +never computes a fingerprint comparison itself. Every comparison is +delegated to EngramLib. EngramLib in turn delegates to a shared, +conformance-gated kernel. A conformance fixture is a recorded +input-output pair. Both an original and a ported implementation must +reproduce it exactly. EngramLib's kernel is checked this way across four +build configurations. This rule is spec I-7 in VectorKit's own design +documents. The kit performs no Hamming math of its own. + +A fingerprint is compact, but this compacting has a cost. Packing +hundreds of numbers into 256 bits throws information away. Some queries +need the finer detail that the original float numbers carry. One +example is telling a passage apart from its own echoed question. The +fingerprint's collapsed form cannot always make that distinction. For +these queries, VectorKit also stores the float vector. It compares this +vector with cosine distance, a measure of the angle between two vectors. +Float math is reproducible on one platform and one build. It is not +guaranteed to produce byte-identical results on a different platform. +VectorKit documents this openly as a boundary, and not as a defect. The +binary fingerprint lane is the four-way-identical lane. The float lane +is the lane that stays reproducible within one configuration. + +Every stored vector carries the model identifier and model version that +produced it. Two different embedding models turn the same word into +different numbers. These differences have nothing to do with meaning. +So comparing vectors from different models gives a meaningless answer. +VectorKit enforces this rule at multiple levels. The rule is called spec +I-4. Storage partitions vectors by model. Every search is scoped to one +model. + +Finally, an on-device memory store must stay fast as it grows. Scanning +every stored vector for every query works fine at a few thousand +records. It becomes slow at a larger scale. VectorKit solves this two +ways. It keeps an in-memory copy of all fingerprints, so no per-query +database read is needed. It also switches search strategy once the +count of live fingerprints crosses a threshold. Below the threshold it +does a full scan. At or above it, VectorKit switches to a sub-linear +search structure. This structure returns the identical answer faster. + +## How It Works + +Writing a vector has two parts. `VectorStore.addPayload` first writes +the vector as one row to a `vectors` table. This table is the durable +source of truth. Nothing counts as stored until this write succeeds. +`addPayload` then mirrors the vector into an in-memory packed array. A +binary vector goes into one shared array. A float vector goes into a +per-model array instead. This mirroring means a later search never has +to re-read the database. The in-memory array can optionally be backed +by an on-disk cache file, called a sidecar. This lets the store reopen +without rebuilding the array from every database row. + +Searching has two independent paths, one per vector kind. A binary +search, `findNearest`, compares the query fingerprint against the +in-memory array using Hamming distance. Below a configurable threshold +of live fingerprints, the search does a full linear scan. The default +threshold is fifty thousand fingerprints. A full scan stays fast enough +below that scale. At or above the threshold, VectorKit switches to a +technique called Multi-Index Hashing. This technique slices each 256-bit +fingerprint into several shorter bands. It uses per-band lookup tables +to rule out most of the collection without touching it. Multi-Index +Hashing is provably exact. It returns precisely the same neighbors the +full scan would have returned, only faster. A test suite checks this by +running both searches on the same random and adversarial inputs. Both +searches must produce identical output. + +A float search, `findNearestFloat`, compares the query's float vector +against a separate in-memory array. That array holds only one model's +float vectors. The comparison uses cosine, Euclidean, or dot-product +distance. Different models produce vectors of different length. +Because of this, VectorKit keeps one float array per model rather than +mixing them. + +A third comparison method, `MaxSimScorer`, serves a different kind of +model. Some models produce many small vectors per item instead of one. +One example is a fingerprint per word, an approach known as ColBERT-style +late interaction. Rather than compare one vector to one vector, +`MaxSimScorer` compares every query-word fingerprint against every +document-word fingerprint. For each query word, it keeps the best match +found in the document. The document's overall score is the sum of those +best matches. This scorer is exhaustive. It never skips a candidate +document. That property makes it the correctness reference for any +faster method built later. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. + +![Figure 1. Topology of VectorKit](topology.svg) + +*Figure 1. Topology of VectorKit. Text enters through an +`EmbeddingProvider`. It becomes an Engram and, optionally, a float +vector. `VectorStore` writes both to the durable `vectors` table. It +also mirrors them into in-memory resident arrays. Reads dispatch through +the `DenseIndex` seam to one of three interchangeable search engines. +The dashed regions mark the durable storage boundary and the optional +on-disk cache.* + +`EmbeddingProvider` is the seam a host application implements. It +supplies whatever inference technique turns text into numbers, for +example a CoreML model. VectorKit ships one concrete implementation of +its own, `FloatSimHashEmbeddingProvider`. This type takes those numbers +and projects them into an Engram fingerprint. It uses a shared substrate +primitive, `FloatSimHash`, to do this. Every provider in the MOOTx01 kit +graph then produces fingerprints the same deterministic way. + +`VectorStore` is the actor every caller talks to. It owns the durable +`vectors` table, through a PersistenceKit `Storage` backend such as +SQLite. It also owns the resident in-memory arrays. Three foundation +types flow through every layer beneath it. `VectorRecordKey` names which +record this is. `VectorPayload` holds the raw typed bytes of one vector. +`DenseHit` is one scored search result. All three are shared, +additive-only types. No search engine defines its own private version of +them. + +Underneath `VectorStore` sits the `DenseIndex` protocol, a pluggable +engine seam. Three concrete engines implement it. `BruteForceIndex` is +the always-correct binary linear scan and the conformance reference. +`MIHIndex` is the sub-linear binary search, gated against +`BruteForceIndex`. `FloatBruteForceIndex` is the float-lane linear scan, +built once per model. `VectorStore` decides which binary engine is +active. It compares the live fingerprint count against `mihThreshold`. +It swaps between the two engines without rebuilding either one, because +both are always kept in sync with every write. + +`ResidentArrayStore` manages the optional `.vec` sidecar file. This file +is a packed, fixed-format binary cache of the in-memory array. It is a +regenerable cache, and never a second source of truth. The sidecar might +be missing, stale, or corrupted. In that case, `VectorStore` rebuilds +the array from the `vectors` table. That table is the only durable +source. + +## What Ships in the Package + +The package ships the Swift sources listed above. It also ships a +mirrored Rust implementation in `rust/`. It ships no bundled data +artifacts. VectorKit has no fixed reference tables of its own. +Everything it stores comes from caller-supplied text and caller-supplied +embedding models. diff --git a/packages/kits/VectorKit/docs/topology.svg b/packages/kits/VectorKit/docs/topology.svg new file mode 100644 index 0000000..6541f48 --- /dev/null +++ b/packages/kits/VectorKit/docs/topology.svg @@ -0,0 +1,124 @@ + + + + + + + + + + + + + VectorKit: text in, ranked matches out + + + + EmbeddingProvider + text → Engram + floats + + + VectorStore + write + search API + + + VectorMatch + ranked results + + + + + + + Durable storage boundary (PersistenceKit) + + vectors table + SQLite / InMemory / … + + + + Optional on-disk cache (regenerable) + + ResidentArrayStore + .vec sidecar, mmap load + + + + + + + ResidentVectorArray + packed in-memory contract, all engines read this + + + + + + + BruteForceIndex + Lane A · binary oracle + Hamming, always active below threshold + + + MIHIndex + Lane B · sub-linear exact + active at/above mihThreshold + + + FloatBruteForceIndex + Lane C/D · one per modelID + cosine / l2 / dot + + + + + + + + + + DenseIndex.search dispatch (hotIndex or per-model float index) + + + + Standalone (not wired into VectorStore) + + MaxSimScorer + Lane E1 · ColBERT MaxSim +