v5.17.0

Nuthatch 5.17.0 lets /sql and declared queries answer in Arrow (RFC-0079), ships a separate Flight
SQL build and two new release binaries (Intel Mac and static musl Linux), gives the embeddable engine a
hybrid open, a second nest in one statement and a host package (RFC-0081), and fixes what a cold
audit of this release's own code found before it was tagged. Three of those were serious: a runtime nest
on 5.14.0 to 5.16.0 could commit blocks without its rows, a failover writer's lease could lapse while
it booted, and three of main's required checks could not fail on a pull request. JSON stays the
default and is unchanged. There is no storage migration, and no NID, registry hash or sealed segment
changes. The feed's fields are unchanged, but on a failover writer its fence now carries the lease's
fence, so a failover deployment should upgrade its followers after its writers, as set out under
Compatibility.

Answers in Arrow (RFC-0079)

/sql and /q/{name} answer as one Arrow IPC stream when asked:

curl -G localhost:8288/sql --data-urlencode 'q=SELECT * FROM weth__transfer' \
  -H 'Accept: application/vnd.apache.arrow.stream' -D head.txt -o answer.arrows

• How to ask. Accept: application/vnd.apache.arrow.stream on GET or POST /sql and on
  /q/{name}, or format=arrow in /sql's query string or POST body. /q/{name} takes Accept
  only, since its query string carries the query's own parameters. Without either, JSON as before.
• The envelope is the JSON answer less its rows, escaped to ASCII, in the x-nuthatch-answer
  response header: columns, count, truncated, degraded, degraded_tables, tip_unavailable,
  cached and provenance. Provenance is read after the statement runs, which is why it is a header
  and not schema metadata.
• The rows are the JSON answer's, in the same order, held row for row by a test over sealed and hot
  rows on both routes. Types differ, values do not: 256-bit integers are the same decimal text
  (Utf8), bool columns the same 'true'/'false' text, and dates and timestamps keep their Arrow
  types where JSON writes text.
• The same caps: max_rows, truncated when another row existed, the byte cap counted on the
  stream, and the cost meter. A refusal is JSON either way. An Arrow answer neither reads nor fills the
  answer memo, so cached is always false.
• When it pays. On a 100,000-row export IPC encoded about 5 times and decoded about 16 times
  faster than JSON. On a point read the schema makes it the larger answer and no faster.

Arrow Flight SQL ships as a separate build, as graph and scaled do. The release attaches
nuthatch-flight-aarch64-apple-darwin.tar.gz, nuthatch-flight-x86_64-unknown-linux-gnu.tar.gz and
nuthatch-flight-aarch64-unknown-linux-gnu.tar.gz, each with its .sha256 and attested, and
publishes the image ghcr.io/nuthatch-org/nuthatch:5.17.0-flight, never tagged latest. The
default binary and image are unchanged and flight stays off by default. In the flight build,
nuthatch dev --flight 127.0.0.1:8815 serves Flight SQL beside the HTTP API, for one nest or a
runtime. A statement is answered by the handler behind POST /<route>/sql, under its guards, with
the caller's authorization passed on; the mount is named by the x-nuthatch-route header and the
envelope returns as x-nuthatch-answer. A refusal is a gRPC status carrying /sql's JSON error:
401 is Unauthenticated and 404 (no such mount) NotFound, with the rest in docs/operators.md.
The catalogue calls GetCatalogs, GetDbSchemas, GetTables (with each table's schema when asked),
GetTableTypes and GetSqlInfo answer, listing what /sql on that route can read; prepared
statements and updates do not. Proven in CI against the real binary over a two-mount runtime
with Apache Arrow ADBC's Go Flight SQL driver (v1.12.0, the one ADBC's C, Python, R and Java bindings
load) and arrow-flight's Rust client: rows equal to /sql's value for value, the route header
picking the mount, refusals as gRPC statuses, and listing tables. JDBC, DBeaver and pyarrow.flight
have never been run against it.

docs/operators.md has both under the service surface ("Answers in Arrow" and "Arrow Flight SQL"),
and the builder skill has an export recipe.

Two more binaries

The release attaches the default binary for two more targets, each with its .sha256 and attested:
x86_64-apple-darwin, for Intel Macs, and x86_64-unknown-linux-musl, a static Linux binary with no
glibc dependency (#2642). The musl binary uses musl's allocator rather than jemalloc, and its
behaviour under load has not been measured; the Intel binary was built in CI and has not been run
on Intel hardware by us. The scaled, graph and flight variants are not built for either.

Also new beside the binary: examples/paywalled-chains, an Etherscan-compatible address-history
template for Base, BNB Chain and Gnosis at one /api routed by chainid (#2643), proven end to end on
Gnosis with keyless RPCs. Base and BSC need an archive RPC that serves debug_traceBlockByNumber, and
no keyless one we tried does. AGENTS.md is a guide for coding agents using nuthatch, every command in
it checked against --help.

Fixes

• An ORDER BY ... LIMIT over a UNION ALL could return the wrong rows (#2583, #2584,
  burrmill#126). Where each arm of a union puts a literal in a column ('burn' AS kind), the per-arm
  top-k dropped that constant column from its sort, and the arms shared one top-k cut-off, so whichever
  arm filled first could push out rows that sort ahead of it. The binary's answer depended on which arm
  finished first, and the module's was wrong every time: defi-pulse-nest's largest_supply_moves_7d lost
  two of fifteen rows in the module, and 5.16.0's binary gets a plain inline union wrong. A top-k below
  a merge now sorts by the merge's whole ordering. Who is affected: anyone with a view or statement that
  limits a union of literal-tagged arms; such answers may change, to the correct rows.
• A restart seed that failed part way served a partial entity (#2555, #2539). An authored entity
  (entities.toml) whose seed failed after it began (a segment read, the offchain feed) was left
  marked available with whatever had been folded. It is now put back to unavailable, with its reason,
  on any error. Nests with authored entities, in the binary and the tab.
• nest nid or nest inspect during a directory's first dev could print a wrong NID (#2557,
  #2530). The admin token is written under .admin-token.<pid> before it is linked into place, and a
  listing in that moment hashed it. Those names are now left out of a nest's inputs. dev's
  stale-schema refresh also stops rewriting schema.json, llms.txt, SKILL.md and semantic.toml
  when their bytes are unchanged.
• A runtime's union fetch asked static contracts' events of the whole chain (#2553, #2410). Once a
  factory nest on a cursor passed 500 children, the cursor fetched every co-tenant's topics with one
  address-less getLogs, so a static ERC-721 Transfer returned every such log on the chain. Both
  cursors now widen per template: statics and factory bases are always asked by address. Multi-nest
  runtimes with a large factory nest beside others; the rows were right, the RPC bill was not.
• A watch can name its chain (#2554, #2470). On a directory of chains, POST /api/watch and
  /api/unwatch took every chain; they now take an optional chainid. Without it, every chain, as
  before. Address-history (rotki) users on more than one chain.
• rpc_urls is optional when the run supplies the RPC (#2556, #2471). A directory of chains reads
  NUTHATCH_RPC_<chain id>, yet the loader required the line. A chain with no RPC from any source is
  now refused at startup with both remedies named, before any cursor starts.
• Burrmill's statement thread runs on 8 MiB (#2560, #2460), the planner's stack, where a deep
  statement overflowed the default 2 MiB. Seen on a Graph network replay; any build could meet it.
• Two aggregates of one kind over different columns (#2549, #2485) were folded together by the
  batch oracle the entity circuit is tested against. The circuit was right; tests only.
• The repository's roll script stopped without a word (#2569) when awk exited early and printf
  died of SIGPIPE under pipefail; it now reads from a here-string, as does
  scripts/release-gate-run.sh. The benchmark's RAM label rounds MemTotal up, so a 16 GiB runner no
  longer reads as 15 and fails the point-read gate on its label. Ours and CI's; a 64 GB Linux box with
  over 1 GiB reserved now labels itself 63 rather than 62.

Also: tests for the per-window and seal-time snapshot publishes (#2552), a faster mirror test fake
(#2551), two load-dependent tests fixed (#2558), and a correction to the 5.16.0 notes (#2566).

What a cold audit of this release found

Before tagging, the release's own code was read cold by reviewers who had not written it, and what
they found was filed and fixed (#2585 to #2636). The ones an operator could meet:

• A runtime nest could commit blocks with none of its rows (#2607). On the buffered cursor, the
  default for runtimes since 5.14.0, a nest took a buffered window without checking that the window's
  fetch had asked for its addresses. A nest re-admitted after a retryable quarantine, or mounted on a
  running runtime, could therefore commit windows fetched while it was absent, as empty, and report
  last_block at the tip with rows missing and no error. A test with three nests to block 45 stored
  45, 24 and 45 rows. A nest now fetches any range a window did not ask for. Who is affected: a
  runtime on 5.14.0 to 5.16.0 with more than one nest on a chain, any of which was quarantined and
  re-admitted or mounted live. Re-index such a nest; a nest that only ever started with its runtime
  was not affected.
• A failover writer's lease could lapse while the runtime booted (#2608). Renewal began only once
  every chain and nest was built, so a boot longer than two thirds of the term (about 6.7 s at the
  default) lapsed an uncontested writer, which fenced itself off, and one longer than term plus margin
  let a standby take the lease while the booting writer committed. Each lease now renews from the
  moment it is taken, and each store is claimed before its cursor starts.
• Followers checked the wrong fence, and webhook delivery ids restarted after a takeover (#2611,
  #2598). The feed carried the host's local claim counter rather than the lease's fence, so a follower
  could refuse a legitimately promoted writer and accept a stale one; and a promoted writer numbered
  X-Nuthatch-Delivery from the old writer's last pending id, or 0, so a receiver deduplicating on it
  dropped new alerts. The feed now carries the lease fence, and the outbox counter is mirrored.
• A writer's /sql could miss rows during a seal (#2613). A request on a pinned snapshot read the
  live catalogue, so when a seal folded a provisional segment the answer lost its rows: count(*) 2
  where 5 is right, unmarked. A snapshot now carries the catalogue that matches its watermark.
• Under strict isolation, the maintained-view builder ran a tenant's views outside the strict
  guards (#2610): a CREATE TABLE in views/*.sql executed, with no deadline and no memory cap. A
  build now runs as a mounting tenant, under that mount's limits, and not before one is mounted (#2636
  keeps that seat current through unmounts).
• A buffered unmount retired the nest before its task stopped (#2593), so a slow window left it
  served and not indexing. It is now retired only once the task has stopped.
• An Arrow answer over a view-typed string column ignored the byte cap (#2599, #2616, #2631): a
  1 MiB cap returned 3.2 MB, and the engine's copy returned 128 MiB unmarked. Slices are now sized and
  compacted by their own rows, in lists and structs too.
• Flight SQL released its response-byte permit before the client read the stream (#2590).
• The warm-restart seed stopped at the first failing entity (#2618), leaving every later one
  unavailable on every restart. Each entity is now seeded on its own.
• A follower pruned hot rows when the writer could not attach a catalogue (#2592). It now keeps
  them and resnapshots.
• The engine's verify could report equal for a column the server left out, and verified for a
  head that was refused (#2614, #2615); and a kept engine store reopened under other filters resumed
  past history it never fetched (#2619). Those are now incomplete, the outcome as it happened, and a
  refusal.

Two were in how the project checks itself. Three required checks passed on every pull request
without running (#2609), because the slow tier ran only in a merge queue that had never been turned
on; it is on now, and the required-checks drift test fails without it. And cargo-deny had never
checked the wasm engine this release attaches (#2617); it does now, and the engine crate gained the
MIT OR Apache-2.0 licence field it lacked. The PR reviewer is back, on Claude Haiku 5.5, and its
approval is required again (#2597).

The embeddable engine (RFC-0081)

The binary is still the product. The engine, nuthatch-events.wasm, is a separate artefact for data
that belongs to one user on one device, and a binary user gains nothing from it and loses nothing
without it. Its interface stays abi: 1: everything below is an optional key.

• Hybrid open. nh_open takes source: rpc, mirror or mirror+rpc. mirror+rpc hydrates
  from a published mirror the whole tables its filters' events name, chosen from the events and never
  from a value, then reads them through the filters and follows the RPC from there. Filters are
  private by default; a private filter on a mirror nest is refused by name before anything is
  fetched, so no request to the mirror names or is chosen by a private value.
• attach: one statement over two nests. Another instance's Arrow answer, with its envelope, is
  bound under its own name for that statement, refused by name at another as_of or when truncated.
• Arrow answers in the module, as the node gives them.
• nuthatch-engine.js, the host package, a new release asset with its .sha256, attested like the
  module. One ES module for browsers and Node, no dependency: Engine.open, follow, sql with
  attach, arrow, request, close. It owns the worker, the store, retries and batching.
  docs/examples/engine-defi-pulse.mjs opens defi-pulse-nest from the public mirror in 29 lines and
  matches the 5.16.0 binary. No npm package.
• The pack is reused between visits. A reopened mirror nest fetches nothing from the mirror, after
  checking that the store names the dataset it was hydrated from and every packed segment hashes to its
  address; otherwise it opens fresh once.
• verify, a check of a hosted answer. Engine.verifier(opts).verify(answer, statement, filters)
  opens the nest the answer's provenance.nid names (the caller brings its files, checked against the
  NID), at its as_of, runs the same statement and compares column by column: equal, differs with both
  values, or not computable, named. A statement that reads outside the caller's filters is reported as
  outside them, never as a mismatch, by rules that ask nothing of the server. Proven against a 5.16.0
  node serving graph-allocations-nest: 266 of 266 columns equal for S7's seven wallets, and one value
  altered by a proxy reported on that column alone. Lodestar's own hosted nest cannot be verified yet:
  its files are not a published version.
• complete: true, catching an RPC that leaves a log out, under a verified head. Each header's
  logsBloom is tested for the filters; a block it admits with no log returned has its receipts fetched
  and hashed to the header's receiptsRoot. An omitted log is recovered and a forged receipts list stops
  sealing by name. On mainnet it costs receipts, not headers: blooms are mostly full, so one range of
  1,894 blocks fetched 966 blocks' receipts, about 1 GB, 17 s against 1.4 s without it. A partial
  omission (one of a block's two matching logs) is not caught, and a log's transaction hash is the RPC's
  word.
• Firefox and the wasmtime host. S5's 13 shapes, and the kill and reopen, pass in Firefox as in
  Chrome, in CI. The wasmtime reference host answers get, so it opens mirrored and hybrid nests, and
  seals under a Helios-verified head (a test dependency of that host only).

Measured on Lodestar's verify-locally page (arb1's public RPC, the public mirror): S7's slowest
wallet went from 1,995 s cold to about 107 s, fetching 179 MiB from the mirror once. The second visit
fetched 0 bytes and answered in about 6 s. Across S7's seven wallets, 409 figures, none differing from
the server's. Headless Brave and Node ran the page; Safari and a phone have not run the engine. Reuse is
per store, and a store is per set of pools, so a different wallet fetches the 179 MiB again.

Partial sync: not in the default binary

RFC-0080's server half, GET /<route>/sync on a follower, is built only with
--features sync-server. Its third S0 measured the route as built at 461% of a follower's apply
against a 10% gate (RFC-0080 §20), so the release binary has no /sync route and refuses a mount that
writes [mounts.sync], naming the feature. A runtime that never wrote [mounts.sync] is unaffected.

Compatibility and upgrade

Replace the binary and restart. A 5.16 store is used as it is. decode, seal, scaffold and
identity are untouched since v5.16.0, no pinned NID or segment hash moved, and arrow and parquet
are the same versions, so 5.17 seals the same bytes as 5.16. The feed's fields and their types are
unchanged; what changes is the value a failover writer puts in fence, below.

Things to know:

• A failover deployment: upgrade the writers first, then replace and restart each follower. A 5.17
  writer reports its lease fence in the feed's fence field where 5.16 reported its local claim count
  (#2611), and a follower refuses any fence below the highest it has seen since it started. A follower
  that followed a writer before that writer's upgrade may therefore refuse it afterwards, until the
  follower restarts; replacing the follower's binary with 5.17 at that restart also gives it the
  corrected check. Writers without [mounts.publish.failover] send no fence and are unaffected.

• A failover writer started while another still renews the lease exits after about 17 seconds,
  term plus margin plus two polls at the defaults, with "the writer lease ... is held by ..., who is
  still renewing it. Start this host as a follower, or stop that writer first". Before 5.17 the first
  writer's lease could lapse while it booted (#2608), so the second often won the race instead. A host
  meant to wait and take over should set standby = true under [mounts.publish.failover]
  (docs/operators.md), which waits for as long as the lease is renewed.

• Deduplicate webhooks on the nest and X-Nuthatch-Delivery together. The id is per nest, and the
  header is not covered by the signature, which is over the body (#2587, docs/operators.md).

• A strict runtime builds a maintained view only once a mount has seated it (#2610), so a copy is
  not written before the nest is mounted.

• An engine store kept from before 5.17 is refused once with open_failed ... open it fresh
  (#2619), because it does not record the filters it was opened with. nuthatch-engine.js reopens it
  fresh by itself, which re-indexes it once.

• A directory holding a leftover .admin-token.<pid>, from a dev killed on its first start
  between writing the token and linking it, had that file in its NID. It no longer does, so such a
  directory's NID returns to the one a clean directory has.

• A client asking for Arrow from a 5.16 node gets JSON. During a rolling upgrade, read the
  Content-Type before decoding.

• A nest that leaves out rpc_urls loads on 5.17 and is refused by 5.16, so do not write one
  until every host that loads it runs 5.17.