v5.2.10

Secondary indexes

A secondary-index backfill on a large table now converges (#2536, #2539). Two independent defects kept a build from ever finishing. runIndexing discarded its own resume checkpoint — ordered-binary sorts undefined lowest, so the running-minimum guard never fired and every retrigger rescanned from the first record — and on a plain RocksDB index it never yielded the event loop, because RocksIndexStore.put is synchronous and the outstanding counter the yields were keyed to never left zero. A 19M-row build ran as one ~18-minute turn until the worker was terminated.

Backfills now resume from the minimum persisted checkpoint across the attributes being built, and yield every 100 scanned entries regardless of write-completion timing. Because the checkpoint is actually consumed now, it also has to be trustworthy: it is written only after the index writes it covers have settled and the RocksDB store has been flushed (index stores run with no WAL), at most once per 5 seconds and never before 10,000 more records, and it never advances past a failed index write.

Each checkpoint is stamped with its own key. A checkpoint left by an earlier release — which could advance past failed and unflushed writes — is not trusted, so the trigger rebuilds instead of resuming past a gap. The one-time cost on upgrade is that an in-progress legacy backfill restarts from record 0.

No thread serves a partially built index (#2537, #2543). On a live 15-node cluster the same table and attribute at the same instant answered search_by_value with [] and a 200 on one thread and threw IndexRebuildingError (503) on another, while search_by_hash returned the record. It was a thread-role difference, not a race: isIndexing is a per-thread cache of the attribute descriptor's persisted indexingPID, and only the schema declare path wrote it. initStores — the schema load path every thread runs on boot and on every resetDatabases() — read the same descriptor and never stamped it, so any thread that never declares a schema held a stale false.

The load path now assigns readiness from the catalog it already read, for handles it opens and handles it reuses. The query planner and executor were also disagreeing: searchByIndex refuses a rebuilding index, but the planner ranked its condition by the partial index's cardinality and could hand it the lead. The planner now treats a rebuilding index as unusable across every comparator branch, including relationship paths and the adaptive filter's lazy switch to indexed retrieval, so a query combining a rebuilding attribute with an indexed sibling leads with the sibling and applies the rebuilding one as a record filter — a complete answer instead of a 503. An abandoned build also leaves a durable marker now, and the abandoned-marker lock acquisition is bounded rather than spinning forever.

Storage

Audit retention applies continuously to RocksDB transaction logs (#846, #2338). RocksDB audit retention runs on the existing self-rearming cleanup cadence instead of depending on startup or disk-pressure signals, so eligible transaction-log segments are reclaimed continuously according to logging.auditRetention. Retiring the loop now returns a drain barrier that dropDatabase() and the legacy dropTable() arm await, so a pass suspended mid-delete cannot leave a write pending against a DBI that is about to close. closeDatabase and dropDatabase also deregister storage reclamation, which previously leaked a handler and the store it pinned on every re-open of the same path.

Operators should note that the retention window doubles as the replication safety window: a peer offline longer than logging.auditRetention can resume past a purged prefix. Continuous retention makes that reachable in steady state rather than only after a restart.

Active RocksDB scans stay alive during TTL eviction (#2556). Native range progress did not renew the read-only idle budget, so the snapshot monitor could abort a snapshot while its HTTP scan was still progressing — an HTTP 500 with Next failed: Iterator not initialized. RocksDB 2.9.0 correctly binds ranges to their transaction and exposed the lifetime bug that 2.8.0 hid by ignoring the supplied transaction.

Range reads now record activity without extending idle write holders, preserve commit-retry handoffs, and raise a defined ReadSnapshotExpiredError (503) before invalid native access rather than failing opaquely. Primary-key scans, secondary-index scans, and entries later filtered out are all covered. Poisoned write holders keep their existing 422 rollback behavior.

Dependencies and build

rocksdb-js moves to 2.9.0 (#2549) — see the rocksdb-js v2.9.0 notes.

The musl lockfile entry is restored (#2555). package-lock.json was missing the @harperfast/rocksdb-js-linux-x64-musl@2.9.0 record, so npm ci could not reproduce the complete RocksDB optional-platform graph and failed outright on Alpine-style installs.

The Docker shrinkwrap guard accepts hoisted dependencies (#2560). The check now validates the installed invariant directly — Harper's exact shared-dependency pins must satisfy rocksdb-js's declared ranges, and both consumers must resolve the same canonical CommonJS module entry — instead of comparing raw manifest specs. That accepts the intentional caret ranges rocksdb-js now publishes without weakening duplicate-instance detection for msgpackr and @harperfast/extended-iterable.

Also in this release

Cherry-pick conflict resolutions for the index-consistency and RocksDB-eviction changes above, and a test-scope correction dropping a transaction-log reclamation test that a cherry-pick had added wholesale to this branch.

v5.3.0-alpha.1

First alpha of the Harper 5.3 line. It covers everything merged to main since v5.2.4, which is where the v5.2 patch branch was cut — fixes released as 5.2.5 through 5.2.9 originated here and are included.

Application database branches

An application can now take a private, durable fork of a database, work in it, and have it torn down with the component.

• A component gets its own branch of a base database, with the base's blob tree cloned in by hard link rather than copied (#2352).
• Tables can be declared into a branch through @table, ensureTable and defineTable (#2523).
• drop_component removes the branched databases the application created (#2517).
• A branched database no longer silently discards its most recent writes when the node is restarted after a crash or hard kill (#2414).
• Branch identity is now owned from disk rather than from memory alone: a branch can reclaim an identity held by its own stranded roots, survives a bad marker instead of bricking, holds identity through cleanup, and deletes only the roots it recorded. create_database is closed to a branch identity so it cannot clone the wrong volumes, and an incomplete blob-volume clone is refused rather than adopted.
• An empty branchedDatabases declaration loads under LMDB (#2413).

Data integrity

• copy-db no longer produces a silently corrupt, non-restorable copy (#2098) — a copy that looked successful could not be restored.
• Transaction-log replay fail-stops at a corrupt frame and discards the transaction it truncated, instead of replaying past the damage (#2087).
• A backup captures only whole blobs, substituting a marker for a blob still being written (#2265), so a backup taken during a write is no longer half a blob.
• Resolved attribute values are kept out of durable records at every layer that writes them (#2368) — previously a computed value could be persisted as if it were stored data.
• An audit entry's field offsets no longer depend on its previousVersion value (#2499), and record version and transaction-log key are exposed as separate audit clocks (#2497).
• getRecordAtTime walks back from the audit-store key rather than the record version, so partial-record history resolves correctly under LMDB.
• Blob content created from a plain Uint8Array decodes as text rather than as byte values (#2415).
• Boot fails when the data-version stamp was not recorded, rather than continuing against an unknown on-disk version (#2398).

Transactions, locking and storage

• Writes issued after a mid-scope commit are atomic with the scope that owns them (#2239); a self-committing transaction in the context slot now starts a real scope (#2325); an ImmediateTransaction commits the handle it opens during its own commit (#2291).
• A transaction parked in its commit phase is no longer poisoned (#2086) — this was destroying the deploy payload blob.
• The RocksDB transaction handle is released on abort and on a failed direct commit (#2128).
• Request-path commit conflict retries are bounded by the request's queue-time budget (#2459), and the exclusive update-attributes lock wait is bounded and released structurally (#2252).
• A wedged commit now reports the long-lived transaction holding it up (#2473).
• table.lock(id) provides exclusive record locks serialized across worker threads (#2462).
• The RocksDB WriteBufferManager no longer stalls every writer by default (#2492).
• Audit retention is applied continuously to RocksDB transaction logs (#2338), and log_files_deleted is reported instead of discarded (#2472).
• Harper boots when its storage volume is full or at quota, instead of failing at startup exactly when you need it to diagnose the problem (#2245).
• File-backed blobs support opt-in deflate compression (#2460).
• A database can be opened into a caller-owned table graph instead of the global map (#2285).

Query engine and APIs

• The query planner uses storage-level statistical range estimates (#2163), with a guard for the condition estimates a zero entry-count estimate breaks (#2479).
• SQL configuration is wired to the real config: sql.engine, allowFullScan, maxSortRows, maxHashRows (#2484) — these settings previously had no effect.
• Indexed array-element scans return each record once, before paging (#2493); element-scoping semantics for queries over array-valued properties are pinned (#2437).
• REST supports total-count pagination via Prefer: count= with a Content-Range response (#2147), and the query-string parser no longer falls through from group-by into sort (#2397).
• New put operation, plus a fix for the target-database authorization mismatch (#2347).
• The Operations API resolves @relationship attributes instead of returning null or rejecting them (#2302).
• set_configuration rejects unrecognized parameters instead of reporting success (#2272); install_node_modules honors the documented dry_run flag (#2340); list_agent_sessions orders by activity time (#2271).
• An invalid GraphQL schema fails on its parse error instead of gating every application for 30 seconds (#2432).
• MCP no longer manufactures create_* tools for Resources with no real create verb (#2405).
• HNSW auto-scales efConstruction and the search-ef ceiling with graph size, for graphs of 1M+ nodes (#2181).
• Per-table record-structure dictionary size is observable (#2250).

Security and authentication

• A SQL permission denial computed by processAST is honored rather than dropped (#2202).
• Under fail-closed mTLS, a client whose revocation status cannot be checked is rejected, and the issuer is recovered from Harper's trusted CAs when the socket chain lacks it. A certificate is never resolved as its own issuer, and an unchanged CA set is not republished.
• TLS state is published transactionally, so a failed rebuild cannot downgrade below the last-good configuration (#2384).
• Scoped authentication tokens: an inline role on create_authentication_tokens (#2176).
• A component-registered operation can be granted in a role's operations allowlist (#2260).
• An unrecognized app-port credential is rejected only once route ownership is known (#2419).
• liveSubscriptionAuth can revoke a single subscriber without ending its subscription (#2039).
• OIDC trusted publishing: deploy from CI with no stored credential (#2173).

Clustering and replication

• A table being created stays invisible to catalog scans on other threads, so replication can never announce a partial attribute list (#2381).
• Cluster-origin table definitions are additive-only, so a peer's partial schema snapshot cannot destroy locally declared attributes (#2258).
• A non-bare-host node identity is rejected, and IPv6 replication URLs are formed correctly.
• sourcedFrom cache-fill conflict convergence is fixed (#2065).
• RocksDB subscription events are delivered for source fills whose version differs from the log key.

Server, workers and deployment

• HTTP startup recovers after a pre-ready worker restart (#2129), and a pre-ready worker's event loop stays alive through startup (#2314).
• A worker counts as replaced when its replacement is serving, not when it exits ([#2363](h...

v5.2.9

Write stalls

The RocksDB WriteBufferManager no longer stalls writes by default (#2490). The process-wide WriteBufferManager is on by default at 1/3 of the block cache, and it was configured as a hard cap: when the budget filled, RocksDB parked every writer across every database in WriteBufferManagerStallWrites(), and nothing guaranteed the flush that would release them. ShouldFlush() only fires when mutable memory alone reaches half the budget, so a budget held by memory that is not mutable — the shape you get when it is spread across many column families — never dropped, and the only way out was a process restart.

The budget is now a soft cap: RocksDB schedules flushes more aggressively instead of blocking writers. Explicitly configured writeBufferManagerAllowStall values still win in both directions.

The trade is memory. With stalling off, each column family keeps its full derived conflict-check history (maxWriteBufferNumber * writeBufferSize) rather than having it derived to zero, so expect a higher memory floor on instances with many column families. rocksdb-js#821 is the other half of the fix.

Query planning

Table sizing for query planning is now O(1) (#2478). estimatedEntryCount() iterated the entire key space natively, on the main thread, once per store per 10-second memo window — and the memo never took effect, because the window was sampled before the call and had already expired by the time it returned. On a 28.6M-row table that is roughly 11 seconds per call, and the planner calls it once per additional condition, so a 10-condition operation measured around 100 seconds of main-thread stall with the Operations API unresponsive throughout.

It now reads rocksdb.estimate-num-keys, which is O(1) and returned an identical count on the affected table. The estimate skews high on overwrite- and delete-heavy data until compaction; every consumer is a relative-ordering or explicitly-estimated path, so that trade is deliberate. Two arithmetic sites that a zero estimate would have broken — the AND-group divisor, which yielded Infinity and silently disabled the adaptive filter/index switch, and the ne null subtraction, which could go negative — are guarded.

Replication configuration

The blob-gap escalation bounds are registered — replication.blobGapEscalationCycles and replication.blobGapEscalationMs now exist in CONFIG_PARAMS, the config validator, and the root config schema. env.get resolves only registered names, so without this the keys would sit in harper-config.yaml doing nothing while the compiled-in defaults stayed in force. The consumer is the Harper Pro escalation budget described in that release's notes; see the harper-pro v5.2.9 notes for the operator-facing behavior.

Also in this release

CI workflow sync for the cancellation migration (#2387), and test-only cleanups alongside the changes above.

v5.1.27

Query planning

• Table sizing for query planning no longer scans the entire key space (#2477). estimatedEntryCount() called RocksDB's getKeysCount(), a synchronous native walk of every key, once per store per 10-second memo window — and the memo never took effect, because the window was stamped before the scan rather than after it. On a 28.6M-row table that is roughly 11 seconds on the main thread, and the planner calls it once per additional condition, so a 10-condition query measured around 100 seconds of main-thread stall with the Operations API unresponsive throughout. The planner now reads the O(1) rocksdb.estimate-num-keys property instead, which returned the identical count on the affected table. The estimate can skew high on delete-heavy data until compaction and can report 0 for a populated table once tombstones outnumber live records; the two arithmetic sites that a 0 would break (the AND-group divisor and the ne null subtraction) are guarded, so a stale estimate cannot flip a request onto a full index scan or produce a negative pagination total.

Migration and data integrity

• LMDB→RocksDB migration now writes the version/metadata prefix on migrated records (#2020, backport of #2014). The v5.1 line has been affected since v5.1.2: every migrateOnStart run wrote prefix-less records, silently dropping record versions. On v5.1 the point-read wrapper repairs prototypes, so a migrated instance looks healthy — until it is upgraded to 5.2 and hits the prototype-loss symptom — but the version loss is live on v5.1 today. The fix is on the write side and ships with a staged, verified promotion (staging directory, atomic rename, all handles closed, exact-header tripwire), a verifyMigratedDatabase diagnostic that no longer leaks RocksDB handles on a partial-open failure, and discovery exclusion for staging directories.

Startup and upgrades

• A downgrade refusal no longer hangs a non-interactive start (#2059). Starting a 5.1.x binary against a store that 5.2.0 had upgraded reached the downgrade confirmation prompt and blocked on stdin forever under systemd, containers, or CI, with hdb.log ending at "Checking if HDB software has been updated" and no diagnosis. Without a TTY the answer is now resolved from the CONFIRM_DOWNGRADE override (env var or argument): no override produces an actionable error naming the store version, the binary version, and the override, an unrecognized value produces a descriptive error, and startup exits instead of hanging. Both downgrade branches also log the store version, binary version, and outcome through the logger, so the log self-diagnoses when stdout is not captured. Interactive behavior is unchanged.

Also in this release

• CI maintenance on the v5.1 branch: dispatched test workflows pinned to a read-only token (#2319), and bot caller workflows synced with main (#2330, #2354, #2386).

Full Changelog: v5.1.26...v5.1.27

v5.2.8

Storage engine

• A commit parked on a write intent that was never released could wedge the worker thread (#2466). rocksdb-js 2.8.0 bounds the coordinated-retry park with ROCKSDB_JS_PARK_TIMEOUT_MS (HarperFast/rocksdb-js#744), so a commit waiting on an intent that never gets released now wakes and consumes a retry attempt instead of never settling. Previously that wait had no upper bound, and the thread holding it stayed parked.

  The root msgpackr pin moves to 2.0.6 alongside it. check-shrinkwrap-pins.mjs requires the root pin to equal rocksdb-js's own so that one module instance — and therefore one structure and extension registry — is shared, and rocksdb-js 2.8.0 depends on msgpackr 2.0.6. Over 2.0.5 that is a single decode-hardening change: array and map lengths that exceed the remaining source data are now rejected rather than trusted. The lockfile loses the nested msgpackr copy under rocksdb-js accordingly. This is the version main already runs.

Replication & subscriptions

• RocksDB subscription listeners dropped every source-fill event as out-of-order, and catch-up replayed patch entries with no value (#2445, backport of #2409). A source fill writes with the transaction's commit timestamp, but the RocksDB transaction-log reader overwrote the decoded record version with the log key. The subscription listener's staleness check then compared a record version against a log position and discarded the event. The same reader left auditRecord.localTime unset, so collection startTime replay called getValue() with no reconstruction time and delivered patch entries with value: undefined.

  The entry's own version is now exposed as AuditRecord.recordVersion, surviving the log-key override, and the staleness check compares record versions; raw same-thread aftercommit entries are stamped with the same two clocks the decoded path reports. version deliberately stays the log key — replication resume relies on version === log key — so cursor advancement, transaction grouping, and resume semantics are unchanged. Only the RocksDB engine was affected; LMDB was already correct.

MCP

• Every table Resource manufactured a create_* MCP tool, whether or not it had a real create verb (#2405, fixes #1945). detectVerbs()'s create check was typeof p.post === 'function' || typeof p.update === 'function', but Resource.prototype defines a base post() (resources/Resource.ts), so the first clause was true for every Resource subclass regardless of what it overrode — making the check unconditionally true and the update clause moot. Agents were offered create tools that had nothing behind them.

  The check now reuses toolRegistry.ts's hasClassLevelVerbs(), which identity-compares against Resource.prototype and so counts only a real override, or the implicit update() override that Table.ts's write path relies on. An instance-level create() override counts directly as well: Resource.post's loadAsInstance === false branch dispatches straight to it, bypassing both post and update, and a resource built that way would otherwise have silently lost its create_ tool.

Also in this release

• Test coverage for the two fixes above: a transaction-broadcast grouping test for the subscription path, a catch-up regression test pinning the client-observable "a replayed patch carries a full value" contract on both storage engines (#2444), and an MCP fixture corrected to match Table.ts's actual shape (it defines create() alongside update()).

Full Changelog: v5.2.7...v5.2.8

v5.2.7

Data integrity

• Resolved @computed and @relationship values were written into durable records, and the affected rows then became unreadable and undeletable (#2368). A record resolved from a cache source carries the record prototype, whose response projection (toJSON) surfaces scalar @computed values — and @enumerable relationships since 5.1.0. msgpackr consults an instance's toJSON when encoding it, so the durable encode ran the response projection and wrote the resolved values as stored fields. Materializing such a record then assigns the stored value back through the resolver accessor, with two distinct outcomes:

  • a computed attribute has no setter, so every read, query, invalidate and delete on that record threw attribute.set is not a function — reachable in 5.2.0 through 5.2.6;
  • a relationship setter dereferences the stale value, so a dangling foreign key crashed the same way (since 5.1.0), and a scalar collision silently destroyed the foreign key.

  The write side now enforces the invariant at the layers that own it. recordUpdater projects the record — and the audit entry's own record, gated so message and publish payloads stay verbatim — to its stored fields before anything durable is written, dropping any name a resolver owns whether it arrived through the response projection, from a source, or in a peer payload. A source that returns a related object instead of its foreign key still has the key derived through the writable resolver's setter before the name is dropped. structon is pinned to 1.1.0, which writes own properties only, matching msgpackr's object writers and closing the prototype-walk route for every struct encode.

  The read side makes the records that affected releases already wrote recoverable: the four paths that promote a plain decode to a record instance now skip resolver-owned names instead of assigning them through the accessors, and the accessor itself drops such an assignment (warning once per table) rather than calling an absent setter. A schema reload clears attribute.set alongside attribute.resolve, so changing an attribute from @relationship to @computed cannot retain a stale setter. If you are on 5.1.x–5.2.6 and have hit attribute.set is not a function, upgrading is what makes those rows readable and deletable again.

• A prototype-chain walk could copy inherited properties into a durable record (#2368). assignStoredFields used for..in, so an inherited enumerable — including anything on a polluted Object.prototype — could reach a materialized and then durable record on exactly the paths that its own Object.assign fallback and storedFieldsOnly already restrict to own properties. It now reads own keys only.

Schemas

• @computed(from:) expressions failed to compile in an inline-loaded schema (#2360, landed via #2368). vm.Script requires a string filename and inline schemas were loaded with a null one, so any computed expression in such a schema threw. Reported and fixed by @kylebernhardy.

Behavior changes worth knowing on upgrade

• Assigning to a @computed attribute now throws a client error naming the attribute, instead of a TypeError from the missing setter.
• Harper does not set msgpackr 2.1.0's useToJSON opt-out. That is deliberate: an encoder-wide opt-out would also silence the legitimate toJSON of nested values, so the projection is enforced at the layers that write durable records rather than in the encoder.

Also in this release

• CI: the bot caller workflows are synced with main, and the always-on review arm no longer cancels ready_for_review runs (#2331, #2358). resources/DESIGN.md records the durable-vs-response projection convention behind the fix above.

Full Changelog: v5.2.6...v5.2.7

v5.2.6

Transactions and data integrity

• An explicit transaction() was not atomic when the context's transaction slot already held a released placeholder (#2325, cherry-picked as #2327). In that state — and on a context that never held a transaction at all, such as an instance load — txnForContext installs an ImmediateTransaction, which reports itself open but whose save() is the commit. Both join sites gated on the open flag alone, so transaction(ctx, cb) ran the callback and returned without ever reaching its own commit: every write self-committed as it happened, a throw partway through left the earlier writes durable, the error path's abort() never ran, and the handler still returned success. Both join sites now gate on whether the transaction stages its writes for a later commit, so an explicit transaction() on a released slot is atomic exactly as it is on a fresh one. Reachable in 5.2.1 through 5.2.5.

Restarts and shutdown

• A worker respawn could resurrect the thread pool mid-shutdown (#2316). Terminal process shutdown now latches worker creation and every replacement path, so a late worker exit or an overlapping rolling restart can no longer bring the pool back while the process is tearing down. The restart path sets that latch as its first act rather than waiting for shutdownWorkersNow(), closing the window where a debounced component reload could pre-start an HTTP replacement inside a process that is already exiting.
• Container restarts gain a bounded post-compaction exit watchdog (#2316). Every way the watchdog's shell could give up was a silent success — an unreadable procfs, a base image without sleep — so a successful spawn was not evidence it would ever fire. It now emits a readiness token once both facilities are proven, and arming reports failure unless that token arrives, so the "restart teardown is unbounded" warning reaches operators in exactly the environments that need it.
• The published image runs tini -g as PID 1 (#2316). This gives the container a reaper and makes a reliable SIGKILL fallback possible, while preserving compatibility with volumes written by earlier PID-1 images. Worth knowing on upgrade: group signal forwarding changes which signals component subprocesses receive on docker stop; the user-visible consequences are recorded in DESIGN.md.
• Shutdown listeners preserve requested failure exit codes and still terminate the process when PID-file cleanup fails, rather than exiting 0 or hanging (#2316).

Also in this release

• CI: the dispatched test workflows are pinned to a read-only token (#2318).

Full Changelog: v5.2.5...v5.2.6

v5.2.5

Transactions and data integrity

• A write issued after its transaction scope had completed could be silently dropped (#2291, cherry-picked as #2307). When a table call found no live transaction on its context — including the placeholder a completed scope leaves behind — it built an ImmediateTransaction whose native handle the commit path then discarded, staged writes and all. There was no error and no log line: the caller's await resolved normally over a record that was never written. On 5.2.4 this failed Central Manager's POST /Cluster on every request. 5.1 and 5.2.0 were unaffected — the loss is latent in the commit path but only became reachable once a completed scope started leaving a released-transaction placeholder in the slot.
• Writes after an explicit mid-scope commit are atomic with the scope that owns them again (#2239). A handler that commits its own transaction mid-scope — the documented await getContext().transaction.commit() — had each subsequent write in that scope serviced and committed on its own, so a handler that failed halfway left the earlier half durable and unrollbackable. That is the mechanism behind a failed cluster delete leaving a cluster marked TERMINATED with its instances still RUNNING. A successful commit that is not the scope's final one now rotates the owning transaction to a fresh open generation. LMDB has always behaved this way; only the RocksDB path diverged, so this closes a property the 4.7 → 5.x upgrade quietly dropped rather than adding a new one.

Operations and observability

• get_status now waits for every live worker thread (#1952). Collection was sized from a logical worker count and could return before all physical threads had replied. During a rolling restart an old and a replacement thread can share a logical worker index, so one response overwrote the other and could hide an error. The collector now snapshots the eligible physical thread IDs and completes only once each has replied, keeping overlapping generations distinct internally. The response payload is unchanged — duplicate generations still collapse to the existing name@worker-N label, retaining the worst status. Worth knowing on upgrade: a routine redeploy can now surface a real transient loading or error state that the previous overwrite accidentally masked.

Vector search

• HNSW index quality scales with graph size (#2181). A fixed construction candidate list eventually stops creating enough useful edges, so recall degrades as a graph grows. efConstruction now auto-scales from a base of 100 as min(1024, 100 × sqrt(nodes / 250K)), and the search-ef ceiling resumes scaling above one million nodes, up to 2048. Graphs below 250K nodes keep the previous default, and nodes inserted before a scale threshold keep their existing edges — the ramp applies to new inserts and to a reindex. An explicit efConstruction, efConstructionSearch, or per-query ef remains an authoritative cost ceiling, so a pinned index can return fewer rows than limit unless the query supplies a larger ef.

Also in this release

• CI: the v5.2 branch now pins the same ai-review-prompts revision as main, restoring AI review coverage for PRs that target the release branch (#2308).

Full Changelog: v5.2.4...v5.2.5

v5.2.4

Regression fix: using a context after a mid-handler commit

Committing the current transaction mid-handler and then continuing to use the context — the pattern documented in the v5 migration notes and the 4.5.0 notes — started returning 500s on 5.2.1 and later. releaseContext() (added in #2030 to stop a long-lived context pinning a completed transaction) set context.transaction = null, and the next touch threw Cannot read properties of null (reading 'commit').

On 5.2.3 this took out Central Manager's fabric connect and cluster create/delete for any non-super-user, because getUserPermissions() commits the caller's transaction mid-request.

A completed transaction now leaves a frozen, process-wide released placeholder in the slot: commit() and abort() are no-ops and reads through it see the latest committed state — the behavior the slot had before #2030 — while retention stays O(1) per process. Every route that adopts a caller-supplied transaction or context refuses the placeholder rather than silently operating on it. (#2230, closes #2229)

Data integrity

• A replicated delete-then-put no longer strips a record's secondary indexes. A replicated delete K; put K transaction — the shape a replace-all writer produces — could leave a follower holding a live, correct record with none of its secondary index entries: invisible to every indexed search, and not repairable by rewriting the record. Same-key writes now execute in the order they were staged, and a transaction applied from a leader stages its same-key writes in the leader's order. (#2235)
• Blind-write transaction bookkeeping corrected. Every write with no preceding read — invalidate, publish, crash-recovery replay, reload markers — installed a native handle without the per-handle reference bookkeeping, leaving the read-transaction count as NaN for each consumer that reads it (commit(), doneReadTxn(), disregardReadTxn(), releaseContext() and abort()). Those transactions are also now visible to the long-transaction monitor. (#2232)
• Failed blob saves finish cleaning up before they reject. A failed re-streamable blob save could settle its rejection before PENDING-marker cleanup and blob unlock completed, so a caller attempting immediate recovery could not treat rejection as a lifecycle boundary. Cleanup is now a barrier ahead of rejection. Companion to harper-pro #732. (#2228)
• Asynchronous commit callbacks are awaited. Promise-returning transaction commit callbacks are now awaited by LMDB's optimistic and exclusive paths and by RocksDB before the native transaction commits, so a reload marker's immediate-visibility contract holds on both engines. (#2208)

Availability

• The analytics storage metric can no longer OOM the main thread on boot. NODE_STORAGE walked the entire Harper root with an unbounded recursive Promise.all, holding a path, a dirent and a pending stat for every file at once. On a node with a large blob store the main thread grew about 72 MB/s until V8 aborted — and because the metric runs on the first analytics cycle after start, it happened on every boot (124 restarts observed on 5.2.3). The walk now streams each directory through opendir and stats one file at a time. (#2242, fixes #2240)
• storage.debugLongTransactions no longer breaks reads. With the flag on, any search() whose transaction had not opened its own read handle threw TypeError: Cannot read properties of undefined (reading 'push') and returned a 500 — the diagnostic broke exactly the reads an operator enables it to diagnose. (#2225, refs #2222)

Configuration and observability

• replication_receiveQueueHighWaterMark is now a registered configuration setting, so the bounded replication receive queue added in harper-pro #735 can be tuned. (#2233)
• harper status reports process uptime, and system_information returns process_uptime in seconds in its time response. (#2209)

Also in this release

Windows CI deflakes for early-hints deployment and the QA-782 LMDB control arm (#2227, #2243/#2248); promoted test coverage for streaming delivery and stream-error contracts (#2070) and six QA eviction/removal/reclaim data-integrity anchors (#1916); an in-repo guard for the new config registration; and a package-lock.json sync.

Full Changelog: v5.2.3...v5.2.4

v5.2.3

Blob durability — dangling references and in-flight readers

Two related fixes close windows where a committed record could point at blob bytes that were no longer on disk.

• A transient replication save could leave a record referencing a missing, PENDING, or incomplete blob file. When the same record is later re-delivered by a base copy, Harper now repairs that existing file in place rather than writing a new orphan the duplicate-skipped record would never reference. The repair is fail-atomic — it holds the blob lock, writes and verifies a sibling temp file, flushes, and renames over the target — so a failed repair leaves the referenced file byte-for-byte unchanged. The orphan sweeper respects active repair locks. Adds the shared blob-header classifier and the replication.blobGapReconnectMs config key consumed by harper-pro (#2177).
• Superseded blob files were unlinked on a fixed 500 ms timer with nothing checking whether a reader still needed the bytes. Because a blob file is opened lazily by path at stream()/bytes() time, a reader that resolved a record just before a concurrent write could open a file that was already gone — on the HTTP path that ENOENT lands after response headers are committed, reaching the client as a truncated body while every signal reports success. The delay is now configurable via storage.blobRetention and defaults to 2s, and reclamation is reference-aware so an in-flight reader can take a retention hold (#2145).

Deploy — by reference, with a durable credential

harper deploy gains two opt-ins that together let an app be deployed from its git repository instead of an uploaded payload:

• by_ref=true (or ref=<committish>) resolves the app's GitHub owner/repo and commit from the local working copy (or from GitHub Actions env) and deploys git+https://github.com/<owner>/<repo>.git#<sha>. Everything that could move is resolved client-side to a SHA — including an explicit ref=, locally first and then via git ls-remote — so cluster peers, which resolve the package independently, cannot diverge on a moved tag. An unpinnable ref fails closed rather than being sent as a name the cluster would resolve for itself, and an attached credential is pinned to the package host (#1850).
• setup=true is a guided, client-side credential provisioning flow — what harper login is for auth, but for deploy tokens. It fetches the cluster's public key, sources a token (a fine-grained PAT, your gh session, or an npm token), seals it locally into an enc:v1: envelope, and stores only ciphertext in the component-scoped secret tier. The plaintext never leaves the machine; the cluster decrypts in memory only at deploy/rollback time (#1851).

package_component streams instead of base64

package_component returned the whole component tarball as a base64 string inside the JSON envelope, peaking at roughly 4.7× the archive size resident in a shared (on Fabric, multi-tenant) Harper process, and hard-failing with ERR_STRING_TOO_LONG once the base64 result exceeded V8's string cap. The operation now streams the archive; project-resolution failures other than ENOENT are rethrown instead of being swallowed (#2152, #2150).

Dependency pinning for load-bearing modules

A caret range does not bind the version that reaches a running node: harper-pro, a rebuilt container, or a plain npm install of published harper could resolve a newer native or encoder dependency with no Harper PR and no human merge. The load-bearing ranges — rocksdb-js, the encoder/iterator modules whose object identity crosses its boundary, structon, and the optional native addons — are now pinned to the versions the lockfile already resolved. No installed dependency moves; only the allowed resolution narrows. Docker smoke additionally requires the root msgpackr and @harperfast/extended-iterable specs to be byte-equal to rocksdb-js's requirements and rejects a nested copy (#2179). A follow-up aligned the structon pin with the lockfile after it broke npm ci on main (#2196).

Also in this release

A report-mode review-coverage CI check that surfaces cross-model review counts and a stale Human-Review-Need: footer without blocking (#2183); an integration-test helper refactor (#1904); and non-major dependency updates (#2131, #2189, #2190).

Full Changelog: v5.2.2...v5.2.3