Secondary indexes
A secondary-index backfill on a large table now converges (#2536, #2539). Two independent defects kept a build from ever finishing. runIndexing discarded its own resume checkpoint — ordered-binary sorts undefined lowest, so the running-minimum guard never fired and every retrigger rescanned from the first record — and on a plain RocksDB index it never yielded the event loop, because RocksIndexStore.put is synchronous and the outstanding counter the yields were keyed to never left zero. A 19M-row build ran as one ~18-minute turn until the worker was terminated.
Backfills now resume from the minimum persisted checkpoint across the attributes being built, and yield every 100 scanned entries regardless of write-completion timing. Because the checkpoint is actually consumed now, it also has to be trustworthy: it is written only after the index writes it covers have settled and the RocksDB store has been flushed (index stores run with no WAL), at most once per 5 seconds and never before 10,000 more records, and it never advances past a failed index write.
Each checkpoint is stamped with its own key. A checkpoint left by an earlier release — which could advance past failed and unflushed writes — is not trusted, so the trigger rebuilds instead of resuming past a gap. The one-time cost on upgrade is that an in-progress legacy backfill restarts from record 0.
No thread serves a partially built index (#2537, #2543). On a live 15-node cluster the same table and attribute at the same instant answered search_by_value with [] and a 200 on one thread and threw IndexRebuildingError (503) on another, while search_by_hash returned the record. It was a thread-role difference, not a race: isIndexing is a per-thread cache of the attribute descriptor's persisted indexingPID, and only the schema declare path wrote it. initStores — the schema load path every thread runs on boot and on every resetDatabases() — read the same descriptor and never stamped it, so any thread that never declares a schema held a stale false.
The load path now assigns readiness from the catalog it already read, for handles it opens and handles it reuses. The query planner and executor were also disagreeing: searchByIndex refuses a rebuilding index, but the planner ranked its condition by the partial index's cardinality and could hand it the lead. The planner now treats a rebuilding index as unusable across every comparator branch, including relationship paths and the adaptive filter's lazy switch to indexed retrieval, so a query combining a rebuilding attribute with an indexed sibling leads with the sibling and applies the rebuilding one as a record filter — a complete answer instead of a 503. An abandoned build also leaves a durable marker now, and the abandoned-marker lock acquisition is bounded rather than spinning forever.
Storage
Audit retention applies continuously to RocksDB transaction logs (#846, #2338). RocksDB audit retention runs on the existing self-rearming cleanup cadence instead of depending on startup or disk-pressure signals, so eligible transaction-log segments are reclaimed continuously according to logging.auditRetention. Retiring the loop now returns a drain barrier that dropDatabase() and the legacy dropTable() arm await, so a pass suspended mid-delete cannot leave a write pending against a DBI that is about to close. closeDatabase and dropDatabase also deregister storage reclamation, which previously leaked a handler and the store it pinned on every re-open of the same path.
Operators should note that the retention window doubles as the replication safety window: a peer offline longer than logging.auditRetention can resume past a purged prefix. Continuous retention makes that reachable in steady state rather than only after a restart.
Active RocksDB scans stay alive during TTL eviction (#2556). Native range progress did not renew the read-only idle budget, so the snapshot monitor could abort a snapshot while its HTTP scan was still progressing — an HTTP 500 with Next failed: Iterator not initialized. RocksDB 2.9.0 correctly binds ranges to their transaction and exposed the lifetime bug that 2.8.0 hid by ignoring the supplied transaction.
Range reads now record activity without extending idle write holders, preserve commit-retry handoffs, and raise a defined ReadSnapshotExpiredError (503) before invalid native access rather than failing opaquely. Primary-key scans, secondary-index scans, and entries later filtered out are all covered. Poisoned write holders keep their existing 422 rollback behavior.
Dependencies and build
rocksdb-js moves to 2.9.0 (#2549) — see the rocksdb-js v2.9.0 notes.
The musl lockfile entry is restored (#2555). package-lock.json was missing the @harperfast/rocksdb-js-linux-x64-musl@2.9.0 record, so npm ci could not reproduce the complete RocksDB optional-platform graph and failed outright on Alpine-style installs.
The Docker shrinkwrap guard accepts hoisted dependencies (#2560). The check now validates the installed invariant directly — Harper's exact shared-dependency pins must satisfy rocksdb-js's declared ranges, and both consumers must resolve the same canonical CommonJS module entry — instead of comparing raw manifest specs. That accepts the intentional caret ranges rocksdb-js now publishes without weakening duplicate-instance detection for msgpackr and @harperfast/extended-iterable.
Also in this release
Cherry-pick conflict resolutions for the index-consistency and RocksDB-eviction changes above, and a test-scope correction dropping a transaction-log reclamation test that a cherry-pick had added wholesale to this branch.