v5.4.0-alpha.1

First alpha of the Harper 5.4 line. These notes cover everything merged to main since v5.3.1; some fixes were already backported to 5.3.2–5.3.4.

Headline work: schema-defined sharding, MQTT QoS 2, HTTP conditional requests, PR preview deployments, a dedicated replication worker pool, a ~40% smaller per-thread heap, full-text search on Tantivy 0.27, and a large set of data-correctness fixes for out-of-order replicated writes, queries and eviction.

Upgrade notes

• Full-text indexes are rebuilt on upgrade. Full-text search moves to Tantivy 0.27 (fulltext engine 0.7.0), which changes English stemming. Existing indexes, including 0.5.0 checkpoints from 5.3.4, are rebuilt from Harper's records, and search on an index waits until its rebuild covers it. Downgrading directly to an older full-text engine is not supported. (#3680, #3714)
• New installs seal JavaScript intrinsics instead of freezing them. The new default applications.lockdown: seal-after-load stops code from adding or deleting properties on Object, Array, Promise and the other intrinsics and their prototypes, but leaves existing properties writable. Freezing slowed common paths such as msgpackr decoding by about 4×. Upgraded installs keep the freeze-after-load value already written to their harper-config.yaml; set freeze-after-load before downgrading to a release that does not know the new modes. (#3552)
• replication.threads now defaults to half the HTTP worker count (floor(threads.count / 2)) on nodes with two or more workers and a dedicated replication port, so replication runs on workers that load no application code. Applications that register Table.setResidency, setResidencyById or an indexed computed attribute log an error while the pool is active; set replication.threads: 0 to opt out. (#3035, #3232)
• X-Replicate-To and the operations API replicateTo / replicatedConfirmation fields require a super user. Application code that sets them on a context is still trusted. (#3084)
• Malformed harper-config.yaml is rejected at startup, including duplicate keys, instead of being backfilled. (#2986)
• Read-only mode refuses writes with a 405 on both LMDB and RocksDB, before they reach storage. (#3427)
• Source maps are off by default. Start Node with --enable-source-maps to turn them back on; workers inherit it. (#3072)

New features

• Schema-defined sharding with database-wide slots. A table declared @sharded places records by primary-key hash, with persisted, versioned shard maps enforced on every write path. (#3527)
• MQTT QoS 2 exactly-once delivery, inbound and outbound, surviving reconnects. Last wills are now published for non-super users. (#3263, #3302)
• HTTP conditional requests: If-Match, If-None-Match, If-Modified-Since and If-Unmodified-Since. Matching GET/HEAD returns 304; a failed precondition returns 412. On RocksDB, single-record writes evaluate the condition under a lock (rest.preconditionLocks). (#3229)
• Multipart forms for REST resources, with uploaded files streamed as Blobs. (#3126)
• server.registerBearerValidator lets a component accept its own Bearer tokens and map them to Harper users. (#3313)
• Deploys:
  • PR preview deployments, with payload routing, scoped deploy grants, preview expiry and harper preview deploy|drop|list. (#3505)
  • harper deploy setup=true provider=github-actions sets up GitHub Actions deploys. (#3127)
  • A restarting deploy is certified in a canary worker before it rolls out. (#2981)
  • Rolling deploys are tracked per node, with release and provenance reporting. A rolling restart that does not restart every node now fails. (#3502, #3498)
  • One isolated application can start or stop without restarting the shared workers. Package deploys honor branchedDatabases. (#3339, #3083)
• Logger status and health checks: a .status() API on loggers, configurable storage, memory and CPU checks, and a hierarchical status view. (#372)
• Analytics record per-worker event-loop delay, per-thread duration and whole-request time. (#3110)
• threads.v8Flags applies V8 flags such as --optimize-for-size to every worker thread. (#3138)
• Agent controls: agent.httpFetch (turn off or host-limit fetches), agent.maxTokens, agent.configScope, per-session spend caps by tokens or priced cost, capped tool results with paged file reads, credential masking in config and log reads, and model backends configurable through set_configuration. (#3011, #3064, #3098, #3117, #3177, #3205, #3412)
• Embeddings: models.embed() accepts truncate: true to cut input at the model's context window. OpenAI embeddings honor configured task prefixes. (#3706, #3450)
• Type-strip mode runs Harper directly from its .ts sources (--conditions=typestrip), for development. (#562)
• 41 configuration parameters that were never registered can now be set via environment variables, CLI flags and set_configuration. (#3113)

Performance and memory

• ~40% less base JavaScript heap per thread (idle worker 57.7 → 34.6 MB): smaller emitted JS, source maps off by default, and heavy dependencies loaded on first use. (#3072)
• Live-subscription re-authorization scales to 100k+ subscriptions: a user or role change rechecks only the subscriptions it affects, and token expiry runs on a timer. (#3141)
• Queries that select, filter, join or sort on a relationship read related records in the background. (#3615)
• Less per-row work on RocksDB indexed range queries, especially with a select. (#3605)
• Audited writes no longer re-read the node-id mapping record. Relayed duplicates of a committed head are skipped without reading the transaction log. (#2930, #3476, #3686)
• A derived index resumes without a rebuild when retention purges an idle log's anchor, and a coverage wait no longer stalls up to a second on commits to other tables. (#3675, #3708)
• The HTTP request queue no longer sheds a burst with 503 because of an earlier slow cycle. (#3200)

Replication and transactions

• Out-of-order replicated writes resolve correctly. Full PUT replacement semantics are preserved. Patches on audit: false tables no longer overwrite newer values, resurrect deleted records or apply increments twice. A transaction that writes one key twice applies both writes. Equal-version writes from unnamed origins are ordered. Subscriptions receive commits that finish out of timestamp order. (#3469, #3411, #3506, #3401, #3576, #3029, #3499)
• A dropped table stays dropped: table generations and a durable drop marker stop a peer that missed a replicated drop_table from bringing the table back, and a peer's retired generation is refused. (#2962, #3119)
• Resume positions: per-origin resume cursors and certified origin-closed timestamp floors, and per-origin subscription checkpoints that resume on any node. Subscription resume positions are checked against retained history, and durable MQTT sessions that can no longer catch up are reset. (#3080, #3085, #3109, #3356, #2946)
• replicatedConfirmation counts only a record's residents, refuses a count they cannot meet before writing, waits for peers on LMDB, skips writes that replicate nothing, and reports a confirmation failure separately from a local write failure. (#3148, #3392, #3423, #3187)
• A transaction is never aborted once its entry is in the transaction log. abort() no longer spins on read references taken after commit. Every local transaction-log key is minted locally. (#3266, #3337, #3510)
• Record locks: releases relayed to a non-coordinating thread are forwarded instead of dropped. Locked writes keep their transaction's version on retry or replay. A restarted lock home ends quarantine by a ring-wide recall. (#3038, #3572, #3259, #3269, #3132)
• Residency: the writing node keeps the full record, a patch is never stored over a stub as a complete row, a fetched record stays a local replica, and source fills index what they store. (#3128, #3322, #3375, #3610)

Query and data correctness

• SQL: WHERE NOT (...) no longer selects, deletes or updates the wrong rows. SQL and REST return the same rows whichever condition leads when attributes are null or missing. COUNT(*) works for roles with attribute permissions. (#3202, #3213, #3601, #3286)
• REST queries: list values, (...) groups, range values and chained conditions now parse correctly. Nested relationship filters and pagination are honored. Select projections are fixed for missing relationships and stored nested JSON. (#3415, #3279, #3221, #3171, #3456)
• Indexes and search: secondary-index scans include boolean false rows. Composite keys work in id filters and OR unions. Full-text and autocomplete pages use Harper's paging. (#3418, #3426, #3301)
• Writes: a patch add on a BigInt attribute adds instead of overwriting. Concurrent auto-increment id ranges on RocksDB no longer overwrite each other. Blob MessagePack frames are no longer corrupted. Contract bodies validate like Table.validate. (#3577, #3021, #3215, #3579)
• Record history no longer shows a later put's data for a patch it overwrote, or a field before the patch that first wrote it, and is reconstructed correctly after eviction. (#3414, #3578, #3214)
• RocksDB eviction no longer deletes renewed records or the blobs of surviving records, and no longer leaves a phantom open transaction. (#3413, #3575, #3323)
• Caching and expiry: expired records are skipped in subscribe() current values, treated as absent on read-only servers, and treated as uncached under only-if-cached. Concurrent sourcedFrom misses share one source call. A record whose source reports it missing is removed with its index entries in one commit. (#3209, #3495, #3623, #3417, #3092, #3529)
• A JSON response that fails mid-stream is cut off instead of ending as a complete array. (#3664)
• Startup no longer fails on a table with no primary key. (#3244)

MQTT and subscriptions

• Durable MQTT topics start below in-flight writes and at the newest committed log key, so a reconnect neither resends nor skips a write. Unacknowledged messages are redelivered after a reconnect. (#3602, #3629, #3643, #3103)
• Re-subscribing a restored topic takes the new QoS. Single-record subscriptions no longer resend the current record on resume. A PUBREC for a restored QoS 2 delivery is accepted. (#3479, #3440, #3480)
• Subscribers on other threads receive non-JSON payloads. Records with a contentType no longer abort durable subscribers. Each client receives the CONNACK refusal code of its own protocol version. (#3446, #3482, #3402, #3303)
• A subscription replay that crosses a table reload marker is refused or re-snapshotted instead of losing rows. (#3503)

Security

• Line breaks in SSE event, id and retry values can no longer inject fields or events. (#3116)
• get_component_file validates the project name. The UDS mirror sockets directory is owner-only. A missing static file returns 404 instead of a path-leaking 500. (#3111, #3019, #3437)
• Logged error details are capped at 256K characters, with credential-shaped fields masked. URL credentials containing / ? # are redacted from component status. (#2939, #3381)
• The MCP endpoint enforces http.maxRequestBodySize. Legacy Fastify routes honor a rejected credential, and every authentication 401 carries a challenge. (#3398, #3558)
• renew-certs no longer empties the certificate table when TLS has no privateKey, and TLS certificate record ciphers follow tls[] edits. First-boot passwords use the configured hash function. Native addons built for the wrong V8 ABI are refused instead of crashing. (#3289, #3237, #3410, #3287, #3140)

Operations

• Backups: restore_backup verifies a backup can be read before replacing the database, and online restore is no longer refused after a table redeclare. The backup CLI refuses offline restore/delete/purge against a loopback login. (#3121, #3201, #3320, #2648, #2649)
• HTTP: HEAD answers like GET for streamed reads. Static files under dot-directories are served. An unread multipart upload is discarded so the client does not stall. Gzip from withNodeAdapter is no longer corrupt on Node 26.11+. (#3497, #3227, #3069, #3108)
• Modules: pure-ESM and import-only package exports load under the compartment loader on Node and Bun. package_component no longer packs the Harper install. (#3093, #3492, #3216, #3483)
• TLS renewals are still detected after a quickly cancelled renewal. Root-config hot reload works on Bun. Windows log-file identities compare correctly. CLI messages name the running product. (#3010, #3433, #3036, #3078)

Also in this release

Dependency updates (rocksdb-js 2.11 → 2.12, undici 8, chokidar 5, @fastify/compress 9, @fastify/static 10, alasql 4.20, extended-iterable 1.0.5), many new caller-impact, upgrade/rollback and integration tests (minor-upgrade checks against 5.2.15 and 5.3.1), unit tests sharded in parallel on Node 24/26, and review and CI tooling.

Full Changelog: v5.3.1...v5.4.0-alpha.1