feat: optional derived vector index (#49) - #85
Merged
Conversation
- VectorIndex protocol + NullVectorIndex default (zero vector deps)
- Optional ChromaVectorIndex backend (lazy import, hotmem[vector] extra)
- Index supplies oversampled cosine candidates; SQLite re-scores with the
identical hybrid formula, preserving the /v1/search ranking contract
- Rebuild reads SQLite only (embeddings reused, never recomputed, no file
I/O); rebuild marker records the store fingerprint for staleness
- Admin endpoints: POST /v1/vector-index/rebuild (emits index.rebuilt),
GET /v1/vector-index/status, DELETE /v1/vector-index
- hotmem serve --vector-index {none,chroma}; CI vector job; docs + changelog
valiantone
added
enhancement
The rebuild marker now records the store fingerprint taken BEFORE the all_rows() read. A mutation landing mid-rebuild (concurrent insert) thus changes the live fingerprint and the marker reads stale, so search takes the deterministic fallback until the next rebuild. Snapshotting after the read could bless an index that silently misses rows inserted during the rebuild. Adds a regression test simulating the mid-rebuild insert.
_stream computed line_start = offset + pos in (carry+chunk) coordinates, overstating file offsets by the carry length for every line after a chunk-spanning line. Affected unsupported_reason offsets and byte_ranges (feeding FileInspection -> API/MCP/CLI). Offsets now use base + pos with base = offset - len(carry), verified for exact parity against the spike's C scanner (bench/native_spike). Adds a chunk-boundary regression test; discovered by the native helper spike (#48, #84).
…, formatting (#91) - gen_corpus.py: malformed-line accounting used a hardcoded 38 instead of the actual 41 bytes (40 + newline); counter now derives from the line. The insertion threshold is unaffected (drift began after insertion), so the committed manifest stays valid for regenerated corpora. - run_bench.py: report rows column now takes the first arm with a measured row count instead of blindly the first arm; reformatted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional, disposable, rebuildable derived vector index (Chroma backend) that accelerates HotMem search while SQLite/files/bundles/manifests remain canonical storage.
Why
Issue #49 (M5: Performance + Inspectors). HotMem needs optional vector acceleration for large stores, but the filesystem-first architecture requires that the index never becomes a source of truth: HotMem must run with no vector backend installed, index loss must never lose memory, and the default response shape and ranking contract must be preserved whether acceleration is present or absent.
Changes
src/hotmem/vector_index.py(new) —VectorIndexprotocol (upsert/search/delete/is_stale/status/clear/apply_rebuild_marker),NullVectorIndex(default; never imports chromadb — proven via subprocess test),ChromaVectorIndex(lazy import, persists under<mount>/hotmem-vector-index/, cosine space),get_vector_index()factory that degrades to Null with a warning when chromadb is absent, andrebuild_vector_index()that reads only SQLite rows — embeddings reused, never recomputed, zero backing-file I/O.MemoryDB.search_by_ids()(new, src/hotmem/db.py) re-fetches candidates with the identical TTL-live/archived predicates andcosine_simUDF, and the unchanged hybrid scorer (cosine + FTS + importance) recomputes final ranking → byte-identical results vs fallback.(COUNT, MAX(rowid), MAX event seq)compared againstrebuild_marker.json; catches inserts, deletes,INSERT OR REPLACErewrites, and rowid reuse. Stale/missing/erroring index → silent fallback to the deterministic SQLite scan.POST /v1/vector-index/rebuild(admin; emitsindex.rebuiltevent; 400vector_index_disabled/vector_dependency_missing),GET /v1/vector-index/status(backend, dependency availability, counts, staleness),DELETE /v1/vector-index(clear)./v1/searchresponse shape untouched.create_app(vector_backend=...),hotmem serve --vector-index {none,chroma}(default none).vectorextra (chromadb>=1.0,<2) in pyproject.toml; uv.lock updated additively. No version bump (auto-tag workflow fires on version changes).vectorjob (py 3.13) installing.[dev,vector]so the real Chroma path is exercised.tests/test_vector_index.py(28 tests) covering all issue acceptance criteria: no-backend operation, rebuild parity across mixed seeds (inline/file-backed w+wo summary/archived/TTL-expired), stale-index fallback, index-loss safety, no eager file reads (SpyAdapter proves zero reads on a 2 MB backing file), deterministic fallback, endpoint behavior, and real-Chroma tests. Plus a golden additive test proving identical scores/ordering/shape with vs. without acceleration.Also in this PR (v0.2.4 scope)
JSONLInspector._streamline-offset fix for chunk-spanninglines (+ chunk-boundary regression test; offset parity with the spike's C
scanner), CHANGELOG entry under 0.2.4.
gen_corpus.pymalformed-line byteaccounting,
run_bench.pyreport rows column + formatting.mainto carry the native-helper spike benchtree that Spike bench nits: corpus byte-count off-by-3, ruff format, report rows column #91's fixes modify.
Release
__version__+CHANGELOG, plus
uv.lockrefresh); merging auto-tagsv0.2.4via auto-tag.