Release 2.6.1: catalog gap-fill + memory primitives - #17
Merged
Conversation
Vector apps built on 2.6.0 kept reimplementing the same primitives outside
the library — pending-write buffers, edge tables, JSON counter increments,
TTL sweepers, change feeds, range filters. 2.6.1 brings them into the
catalog as a coherent, additive set with no public API breaks; existing
databases upgrade transparently on first open.
What you can now do without leaving the library:
- Update a vector in place: collection.update_embedding(id, vec) buffers
the change in a transactional overlay and flushes to HNSW on demand,
replacing the remove+re-add churn pattern.
- Wrap several mutations atomically: with db.transaction(): ... opens a
SAVEPOINT around catalog writes; usearch effects are buffered and
applied only on commit, with collection.tx() as the single-collection
shorthand.
- Walk a graph alongside the index: collection.edges supports add/get/
update/delete with weight, bonus, hits, last_touch as real columns;
numeric deltas (dweight=+0.02, dhits=+1) compile to a single atomic
UPDATE and stack safely under contention.
- Increment counters atomically: collection.increment_metadata(id,
{"hits": 1, "drift": 0.02}) chains json_set + json_extract in one
statement; safe under WAL with concurrent writers.
- Filter by range: similarity_search, keyword_search, hybrid_search,
edges.get_edges, and events.read all accept Mongo-style operator
dicts ($eq $ne $gt $gte $lt $lte $in $nin $exists $between) plus
tuple shorthand ("range", lo, hi).
- Subscribe to changes: collection.events.read(since=, kind=) and
.subscribe(...) expose an append-only feed populated automatically by
every mutating method; cross-process visibility comes from WAL.
- Expire docs by clock: collection.ttl.set(id, seconds=, on_expire=)
with an opt-in background sweeper.
- Defer rebuilds: collection.maintenance.rebuild_if_needed(...) gates a
full rebuild_index() behind pending / tombstone / wall-time thresholds.
- Run multi-process: PRAGMA busy_timeout=5000 and foreign_keys=ON at
every connection-open site reduce DatabaseLockedError pressure and
cascade-delete aux rows on doc deletion.
The sqlite-vec package was never imported and the v1->v2 migration code could not have worked without loading the extension anyway. Removes the dependency, MigrationRequiredError, VectorDB.check_migration, the auto_migrate flag, the catalog legacy helpers, and their tests.
- Drop sqlite-vec dep + v1 migration path from changelog (mirrored to docs/CHANGELOG.md). - Correct the db.transaction() atomicity claim: SQL writes roll back via SAVEPOINT, but coarse vector mutations (add_texts/delete) do not; point users at update_embedding + pending.flush() for commit-gated vector changes. - Note that the events table is intentionally FK-less so the audit trail survives doc deletions. - _DBTransaction.__exit__ now logs at ERROR and re-raises when the outermost conn.commit() fails (was silently swallowed at DEBUG). - Cap filter $in/$nin lists at 999 items to stay below the universally safe SQLITE_MAX_VARIABLE_NUMBER. - Document the _table_name validation invariant in CatalogManager. - Drop the dead self-import in _CollectionTransaction. Adds two regression tests: filter-list cap and tx commit-failure propagation.
Each AsyncVectorCollection / AsyncVectorDB method opened with the same three-line pattern: get the running loop, dispatch to self._executor, wrap the sync call in a lambda. Replace 41 of those with a private _run(fn, *args, **kwargs) helper using functools.partial. Behaviour preserved (still uses self._executor, not asyncio.to_thread's global default pool).
Verified gaps on the new public APIs:
- $between now rejects lo>hi (silent empty result -> ValueError).
- update_embedding rejects non-finite vector elements before they reach
the pending buffer and corrupt distance math on flush.
- ttl.set validates seconds/expires_at finiteness and constrains
on_expire to {"delete","callback"}.
- ttl.start_background validates interval>0 and finite (was a busy loop
on 0/negative/NaN).
- ttl.stop_background no longer drops the thread handle when join times
out, so the next start_background can't spawn a duplicate sweeper.
- ttl.sweep escalates the index-remove failure log from DEBUG to
WARNING -- catalog/HNSW divergence is a correctness issue users want
to see, not a debug-only event.
- Edge add/upsert/update_edge reject NaN/inf for weight/bonus and the
numeric deltas; otherwise the column silently traps NaN and every
later range filter returns wrong results.
lefthook now runs the same gate set on pre-commit and pre-push; pre-commit autofixes (ruff format, ruff check --fix), pre-push runs the formatter in --check mode so a push can't regress what was clean at commit time. Tree changes required to make the new gates pass: - Apply `ruff format` across the repo (45 files, whitespace and trailing-comma normalization only). - Rename the local `expires_at` rebinding inside `_TTLNamespace.set` to `resolved`; mypy could not narrow the original `float | None` parameter through the chained guards introduced by the previous fortification commit.
Adds 33 tests (26 -> 59 in this file). Splits roughly into two halves:
Coverage gap fillers for APIs that shipped with 2.6.1 but had no
direct tests:
- counters.get default and missing-row paths
- $exists operator on present/absent metadata keys
- events.last_seq, events.prune semantics, events.subscribe yield
Boundary guards introduced by the fortification commit:
- update_embedding rejects NaN, inf, and non-1-D vectors
- $between rejects lo>hi, non-finite bounds, wrong arity, plus
the inclusive-range happy path and the ("range", lo, hi) shorthand
- edges add/upsert/update_edge reject NaN/inf for weight, bonus,
dweight, dbonus
- ttl.set rejects neither/both of seconds/expires_at, invalid
on_expire, and non-finite seconds or expires_at
- ttl.start_background rejects zero, negative, and NaN intervals
- ttl.stop_background lets a clean stop be followed by a fresh start
Walks every AsyncVectorCollection / AsyncVectorDB wrapper in one run — CRUD, search variants, hierarchy, edges, counters, pending vectors, TTL, events, maintenance, clustering, multi-collection helpers, and async context lifecycle. Reports per-call pass/fail so a single broken API doesn't mask the rest. Intended as a manual smoke runner for the 2.6.1 'advanced memory' surface, not part of the pytest suite. Run: uv run python scripts/exercise_async_collection.py
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af03c2b683
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Owner
Author
|
Both P1 review comments addressed in e2b50c1. Numeric filter type guard ( Atomic TTL sweep ( Regression coverage (
Full lefthook gauntlet (ruff format/check, mypy, bandit, pytest-cov) green on push: 774 passed, 2 skipped, 90% coverage. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
busy_timeout, FK cascade), and async wrappers for all of the above.sqlite-vecdependency and the v1.x auto-migration path that was already non-functional.run_in_executorboilerplate behind a_runhelper.docs/Features.md; README slimmed to highlights + quickstart;docs/examples.mdrewritten with a v2.6.1 Memory primitives section. Newscripts/exercise_async_collection.pysmoke-runner walks the entire async surface.Test plan
uv run ruff format --check— 85 files already formatteduv run ruff check— all checks passeduv run mypy src— no issues in 21 source filesuv run bandit -r src -ll -c .bandit— 0 medium/high findingsuv run pytest tests/ -q— 767 passed, 2 skippeduv run pytest tests/ --cov=src/simplevecdb— 90% coverage (pre-push hook)uv run python scripts/exercise_async_collection.py— 52/52 async surface calls pass