Release v5.1.27 · HarperFast/harper

Query planning

Table sizing for query planning no longer scans the entire key space (#2477). estimatedEntryCount() called RocksDB's getKeysCount(), a synchronous native walk of every key, once per store per 10-second memo window — and the memo never took effect, because the window was stamped before the scan rather than after it. On a 28.6M-row table that is roughly 11 seconds on the main thread, and the planner calls it once per additional condition, so a 10-condition query measured around 100 seconds of main-thread stall with the Operations API unresponsive throughout. The planner now reads the O(1) rocksdb.estimate-num-keys property instead, which returned the identical count on the affected table. The estimate can skew high on delete-heavy data until compaction and can report 0 for a populated table once tombstones outnumber live records; the two arithmetic sites that a 0 would break (the AND-group divisor and the ne null subtraction) are guarded, so a stale estimate cannot flip a request onto a full index scan or produce a negative pagination total.

LMDB→RocksDB migration now writes the version/metadata prefix on migrated records (#2020, backport of #2014). The v5.1 line has been affected since v5.1.2: every migrateOnStart run wrote prefix-less records, silently dropping record versions. On v5.1 the point-read wrapper repairs prototypes, so a migrated instance looks healthy — until it is upgraded to 5.2 and hits the prototype-loss symptom — but the version loss is live on v5.1 today. The fix is on the write side and ships with a staged, verified promotion (staging directory, atomic rename, all handles closed, exact-header tripwire), a verifyMigratedDatabase diagnostic that no longer leaks RocksDB handles on a partial-open failure, and discovery exclusion for staging directories.

A downgrade refusal no longer hangs a non-interactive start (#2059). Starting a 5.1.x binary against a store that 5.2.0 had upgraded reached the downgrade confirmation prompt and blocked on stdin forever under systemd, containers, or CI, with hdb.log ending at "Checking if HDB software has been updated" and no diagnosis. Without a TTY the answer is now resolved from the CONFIRM_DOWNGRADE override (env var or argument): no override produces an actionable error naming the store version, the binary version, and the override, an unrecognized value produces a descriptive error, and startup exits instead of hanging. Both downgrade branches also log the store version, binary version, and outcome through the logger, so the log self-diagnoses when stdout is not captured. Interactive behavior is unchanged.

CI maintenance on the v5.1 branch: dispatched test workflows pinned to a read-only token (#2319), and bot caller workflows synced with main (#2330, #2354, #2386).

Full Changelog: v5.1.26...v5.1.27