Skip to content

DS4 Context Engine 0.3.8

Choose a tag to compare

View all tags
@Alucard24 Alucard24 released this 06 Sep 12:36
· 63 commits to main since this release
v0.3.8
18f36e1

Release 0.3.8 — Indexed FTS key deletes via rowid mappings (schema 16)

Version analyzed: DS4 Context Engine 0.3.8
Commit: 18f36e1
Coordinated packages: ds4-context-core 0.3.8, ds4-context-reference-adapter 0.3.8, ds4-context-engine 0.3.8

Summary

Bugfix release: FTS deletes were full virtual-table scans (FTS5 UNINDEXED
key columns), making session-index rebuilds on large sessions stall for
hours. The FTS tables are now rebuilt over rowid mapping tables, turning
every keyed delete into an indexed lookup. No behavior, configuration or
search-surface change: the fix is transparent and addresses the
fork/resume stall on large sessions.

Root cause

DELETE FROM entries_fts WHERE entry_key = ? cannot use the FTS5
vocabulary because entry_key (and project_snippets_fts.snippet_id) are
declared UNINDEXED. Every row removal scanned the entire FTS index.
A rebuild of a session with ~11k entries performed thousands of those
scans: measured 65k pread64/s against a 615MB database and ~100 full
index passes per ~140s, with an estimated hours to completion —
surfacing as an indefinite hang of the agent session after a session
fork/resume. The behavior class pre-dates 0.3.7; the 0.3.7 work sessions
made it visible by pushing the database into the hundreds of MB.

Indexing the keys directly inside FTS5 was benchmarked and rejected:
it would make key text part of the MATCH vocabulary (false positives)
and shift the bm25 column weights. See ADR 063.

Changes

  • Migration 16 fts-rowid-key-mappings: two derived mapping tables
    (entries_fts_keys, project_snippets_fts_keys) backfilled from the
    live FTS rows in one scan, inside a single transaction.
  • SessionIndexRepository: per-row deletes go through the mapping
    (DELETE … WHERE rowid = ?); the rebuild stale-row cleanup joins the
    mapping with entries and uses the session index; the mapping is
    upserted from the FTS last_insert_rowid in the same transaction.
  • ProjectKnowledgeRepository: same pattern for per-snippet deletes
    (replaceFile) and per-project cleanup (clearProject).
  • The FTS schemas, the MATCH queries and the bm25 weights are
    byte-identical to 0.3.7; search results do not change.
  • Tests: tests/integration/fts-rowid-keys.test.ts covering the
    v15 → v16 upgrade (data preservation, key ↔ rowid correctness, search
    surface unchanged), rebuild/append/stale-cleanup consistency and the
    snippet replaceFile/clearProject paths; version-pinned tests moved
    to CURRENT_SCHEMA_VERSION.

Compatibility and migration

  • Non-destructive by design: the FTS indexes are derived state; the
    migration is transactional, so on any failure the previous schema
    remains intact and the extension keeps working with the old behavior.
  • No data stored in the mapping tables: the base tables remain the
    single source of truth.
  • Upgrade runs automatically on first open after the extension update
    (measured 467 ms on a 615MB database with 31k entries / 62k snippets;
    no user-visible step required).
  • The mapping backfill is one scan; a very large database adds only
    seconds to the first open after update.
  • Existing guarantees (fail-open, atomicity, privacy, pins, compaction,
    hard limits) are unchanged.
Assets 2
Loading