Remembra 5.6.0 — distributed Remembra

Distributed Remembra: cross-instance primitives, an optional Redis backend, and an
explicit process split. A single-process installation is unaffected — with no
REMEMBRA_REDIS_URL the optional package is never imported, no shared store is
opened, and the readiness payload is byte-identical to 5.5.1.

Runtime dependencies are unchanged at four. redis is an optional peer and is
not even a devDependency, which is what keeps its missing-package startup failure
a real test rather than a mocked one. Six new published subpaths: ./lock,
./job-store, ./worker, ./redis, ./shared-state, ./process-roles.

Not verified: a live Redis server. The quota scripts are executed through a
Lua 5.3 VM with a shim of the Redis commands they use (28 checks, covering their
logic but not a real server), and the failure matrix verifies our response to a
store outage rather than Redis itself. Treat shared-state operation as
reviewed-and-unit-tested until a Redis instance is in CI.

Added

• The §30 failure matrix (src/test/failure-matrix.test.ts, with a real
  second OS process in src/test/matrix-peer.ts): all nine distributed-failure
  scenarios, each asserting a stated invariant rather than "did not crash". The
  two-process rows force the same expected version into both processes rather
  than relying on interleaving, so they detect the lost-update defect that
  motivated the milestone. A live Redis is still not exercised — the
  outage rows verify our response to a store outage, not Redis itself.

• Process roles (src/process-roles.ts, published as
  @hilbras/remembra/process-roles): remembra serve, remembra worker, and
  remembra scheduler. The no-subcommand path is unchanged and still requires no
  configuration, so a single-process install is unaffected. A role without an HTTP
  surface refuses --http/--port and refuses data verbs rather than ignoring
  them — a worker that silently bound a port would serve traffic from a process
  whose premise is that it has no HTTP surface. worker runs a durable worker over
  a SQLite ledger; scheduler only enqueues periodic jobs, so exactly one worker
  anywhere claims each one.

• Shared-state configuration with honest degradation
  (src/shared-state.ts, published as @hilbras/remembra/shared-state). The
  capability manifest advertises distributed unconditionally — it describes the
  build, like webhooks, not the configuration. A single-process readiness
  payload is byte-identical to the previous release: it does not even gain a field
  saying "absent". When a configured store becomes unreachable the limiter
  fails closed with 503 rather than degrading to per-instance limits, which
  would let a fleet exceed a tenant's quota while every instance reported a limit
  it was not enforcing; readiness reports unready so a load balancer drains the
  instance, liveness stays up so it can be diagnosed, and it recovers on its own
  at the next successful operation. Reports carry a classified label only — never
  the URL, host, port, or password.

• An optional Redis adapter (src/redis.ts, published as
  @hilbras/remembra/redis) providing RedisLockProvider and
  RedisQuotaRateLimiter. redis is an optional peer dependency and not a
  runtime dependency; the only reference to it is a dynamic import(). There is
  no silent fallback: with REMEMBRA_REDIS_URL unset the process imports no
  Redis code at all, and with it set but unreachable, startup fails rather than
  degrading to per-instance state that would report a limit it is not enforcing.
  Locks use SET … NX PX so the lease and its expiry are one atomic operation,
  and renew/release are owner-checked by the server.
  Quota is evaluated as one Lua script across every dimension rather than one
  limiter per dimension, because the in-process limiter's mutex does not cross
  instances and interleaved dimension checks would let a shared organization
  budget be exceeded. Precedence and policy validation match the in-process
  limiter exactly, and each dimension keeps its own window.
  The script bodies are verified by running the shipped text through a Lua 5.3 VM
  (scripts/verify-redis-lua.mjs, 28 checks, not a release gate). A live Redis
  server is not exercised by the test suite.

• A durable worker (src/durable-worker.ts, published as
  @hilbras/remembra/worker), the shared-execution counterpart to the in-memory
  JobQueue, which is unchanged and remains what a single process uses. Its
  centrepiece is the partition guard: a claimed job's lease is renewed while it
  runs, and if a renewal ever fails the job's AbortSignal fires — finishing
  anyway would be the lost-update failure below wearing a lease. A refused
  complete() is reported as lease_lost rather than as success. Claims are
  restricted to the types a worker declares, so a heterogeneous fleet never
  claims work it cannot run, and stop() releases in-flight leases so a peer
  takes over immediately instead of waiting out the lease. The poll timer is
  deliberately not unref'd, since §28 allows a worker-only process that has
  nothing else holding its event loop open.

• A durable job ledger (src/job-store.ts, published as
  @hilbras/remembra/job-store), the second distributed primitive. All six §27
  states and ten fields, with InMemoryJobStore and SqliteJobStore behind one
  JobStore interface. A claim is a single UPDATE ... RETURNING, so the
  database picks the winner and returns the row it actually claimed — a
  read-then-write is the shape that produced the lost update below. Claims are
  ordered oldest-first with the id as tie-break, so every claimer agrees on the
  next job. A dead worker's job returns to retrying once its lease expires, and
  renew/complete/fail return false for a lease the caller no longer
  holds. Payloads, job types, tenant digests, and stored error labels are all
  bounded. 28 tests, with the whole behavioural suite run against both
  implementations so they cannot drift.

• Lease-based locking (src/lock.ts, published as @hilbras/remembra/lock),
  the first primitive of the distributed-runtime milestone. Every acquisition is a
  lease with an owner and an absolute expiry rather than a lock, so a crashed
  holder is reclaimed instead of deadlocking. release() is owner-checked and
  cannot clear a peer's lease; renew() returns false once a lease is lost, so
  a partitioned worker stops rather than resuming on a belief it still holds it —
  the same failure as the lost update below, wearing a lock. Two implementations
  ship: an in-process provider (the default) and a file-backed one that reclaims
  on the expiry timestamp rather than pid liveness, which is what makes it
  portable beyond one host. lockKey hashes any non-digest part, so a raw tenant
  cannot reach a key that is written to disk. Documented in
  docs/lock.md. 15 tests, two of which caught defects in the
  implementation: an unreadable lease file livelocked acquire forever, and
  isHeld() never observed the lease it had just written.

Fixed

• A cold-start race in the batch idempotency ledger. Two processes
  starting against a fresh ledger directory could observe each other's
  not-yet-0600 claims.sqlite — better-sqlite3 creates the file with the
  process umask and it was only chmoded after the open — and one would refuse to
  start with "claim database must not be group/world accessible". The permission
  check is unchanged; the file is now created 0600 with O_CREAT|O_EXCL before
  it is opened, so there is no window to lose. Without this a fleet cannot
  cold-start against a fresh store.

• A spurious IO_ERROR from a racing delete on the file backend. Two
  MemoryStore instances deleting the same memory could have one unlink the file
  after the other resolved it; the resulting ENOENT surfaced as an error to a
  caller that had done nothing wrong. ENOENT on the unlink is now the
  already-reported "missing" result, so a racing delete is genuinely idempotent.
  This was a pre-existing intermittent failure, caught by a release gate and
  disproving the audit's own assessment of forget as safe — idempotent state
  is not the same as a non-throwing call.

• A cross-instance lost update on the SQLite backend, with a success
  response. MemoryBackend requires mutating calls to be safe under
  "cross-process concurrency (advisory lock)", and SqliteBackend only
  satisfied the same-process half: withLock is a promise queue over one
  instance's field, with no BEGIN IMMEDIATE and no db.transaction() anywhere
  in the file. update read the row, checked expectedVersion in application
  code, and then wrote WHERE id = ? with no version predicate — so two
  instances that read the same version both passed the check and both wrote.
  Reproduced 5 times out of 5 with two instances over one data.sqlite: both
  were told they wrote version 2, and one write was discarded with no error.
  The write is now a compare-and-swap on the version it read, and a zero-row
  result distinguishes NOT_FOUND from CONFLICT so a caller does not retry a
  write whose row is gone. Six tests in src/test/sqlite-cas.test.ts cover the
  race, the error text, delete-in-between, and the two behaviours that must not
  change. Found by
  v5.6.0-audit.md finding S1, the first task of the
  distributed-runtime milestone — reachable before any distributed work, and
  routine after it.

Known limitation, now measured

The native RemoveEnvironmentCleanupHook abort previously documented as a test
teardown nuisance can kill a running server on Node 24. Measured against the
published 5.4.0 and 5.5.1 packages under 2 500 requests of mixed write,
search, and scrape traffic:

Runtime	Result
Node 18.20.8	2 000 requests, no abort
Node 24.21.0, better-sqlite3 11.x	aborted at 750, 750, and 1 750 requests
Node 24.21.0, better-sqlite3 12.11.1	aborted at 751 requests

Three things this corrects: the abort is not confined to process teardown, it
is not fixed by upgrading the binding, and it is not a regression from the
5.5.0 work — 5.4.0 aborts identically. It is a garbage-collection-timing race
in the native binding, so the point at which it fires is not reproducible.

No code change is possible from this repository: there is no wrapper that can
contain a process abort, and the store already caches prepared statements, so the
allocation churn is not ours to reduce. docs/troubleshooting.md now says so
plainly instead of calling it contained. An operator who needs an abort-free
Node 24 deployment should run Node 18, where the exposure has not reproduced.

Fixed in this release

• A cross-instance lost update on the SQLite backend. update was not a
  compare-and-swap: two instances both read version N and both were told they
  had written N+1, and one change silently vanished. Reproduced 5/5.
• A spurious IO_ERROR from a racing delete on the file backend.
• A cold-start race in the batch idempotency ledger. Initialization becomes
  visible at three separate moments, so two processes starting against one fresh
  directory could observe an intermediate state and refuse to start. The
  integrity checks are unchanged; a state only a live initialiser can produce is
  now polled for up to two seconds first. Verified by reverting it: 1 failure in
  24 simultaneous cold starts without the fix, 0 with it.