Skip to content

v1.4.0

Choose a tag to compare

View all tags
@mbocevski mbocevski released this 04 May 09:06
· 6 commits to main since this release
v1.4.0
This tag was signed with the committer’s verified signature.
mbocevski Marko Bocevski
GPG key ID: 19760C609191026A
Verified
Learn about vigilant mode.
454a124
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

AgentDB v1.4 ships hybrid search: BM25 lexical + semantic vector arms fused via Reciprocal Rank Fusion. Three new MCP tools (db_bm25_search, db_hybrid_search, db_reembed_all), schema-declared searchable: true per-field opt-in, BM25 k1/b tuning, persisted BM25 stats with v1→v2 lazy upgrade, durable disk-mode embedding storage, and a v1.3→v1.4 migration helper.

Highlights

Hybrid search

BM25 (lexical) + semantic (vector) arms fused via Reciprocal Rank Fusion (Cormack et al. 2009; k=60 default). Catches exact-term matches that semantic search misses and paraphrases that keyword search misses. Per-arm runtime failures are absorbed via .catch; degrades gracefully when one arm is unavailable; throws only when neither is configured. Library API: Collection.bm25Search and Collection.hybridSearch. MCP: db_bm25_search, db_hybrid_search.

Schema-declared BM25 fields

Mark which fields participate in BM25 indexing with searchable: true per-field. Code-wins precedence in mergeSchemas. Tunable k1/b per collection via bm25: { k1, b } on the schema. Non-string types with searchable: true are rejected at defineSchema time and via db_set_schema.

Persisted BM25 stats

TextIndex JSON bumped to version: 2 — per-doc TF maps and document lengths persist with the collection. v1 indexes (no TF data) load cleanly and upgrade lazily on the next add() for that document. IndexFileTooLargeError thrown instead of silently degrading past the 256 MB text-index cap.

Disk-mode durability

Records inserted in the current session (WAL only, not yet compacted) are now visible to all four search methods — same-session insert + search works without close/reopen. appendEmbeddings writes embedding batches to a fresh JSONL per batch (bypasses LRU eviction race). Mid-flight compaction during large reembedAll runs bounds JSONL accumulation. HNSW rebuilt from disk on reopen.

v1.3 → v1.4 migration

v1.3 incorrectly included _id in the embedding text, so stored vectors and query vectors disagreed. Call col.reembedAll() once after upgrade — returns structured ReembedResult { embedded, failed, errors } for partial-failure introspection. db_reembed_all MCP tool wraps it (admin/DESTRUCTIVE).

Unicode tokenizer

Switched from ASCII \\w to \\p{L}\\p{M}\\p{N}. Accented Latin (café) and CJK (東京) are now indexed and searchable. CJK is retained as whole-run tokens (no segmentation; substring matching not supported in v1.4).

Performance knobs

embeddingBatchSize, diskConcurrency, cacheSize, rowGroupSize — all available on AgentDBOptions (db-wide default) and CollectionOptions (per-collection override). diskConcurrency defaults to 16 for non-FS backends to avoid S3 per-prefix throttling.

Changed

  • Collection.searchByVector is now async (was sync). Return type: Promise<{records, scores}>.
  • Collection.reembedAll returns ReembedResult (was number).

Fixed

  • Same-session disk-mode search returning empty results.
  • HNSW dimensions=0 crash with auto-detect providers (e.g. Ollama default).
  • v1→v2 mixed-corpus ghost results from BM25.
  • _id leaking into BM25 fallback index and into embedding text.
  • JSONL filename collisions under concurrent appends.

Stats

  • 38 → 39 core MCP tools (40 → 41 with HTTP transport).
  • 1058 → 1255 tests passing (+25 BENCH-gated benchmarks).
  • 69 commits over 9 review rounds + 2 smoke-test passes.

Install

npm install @backloghq/agentdb@1.4.0

Full per-feature detail in CHANGELOG.md.

Assets 2
Loading