v2.5.0
Twenty-seven commits since 2.4.0. Two silent-wrong-answer fixes, a health endpoint that stops
lying, structured logs, and the decode path under a fuzzer for the first time.
Derived from git diff v2.4.0..main rather than from PR titles, because 2.4.0 shipped two
user-visible changes that no PR title mentioned.
Health endpoints stop lying about a wedged cursor (#578)
/ready used to answer one question: is the source still answering? A cursor whose RPC endpoint was
perfectly healthy but which had stopped advancing - stuck on a window it could not safely seal -
reported 200 ready for as long as you cared to watch it.
It now also asks whether the cursor is making progress. No advance in last_block within the
threshold and /ready returns 503, with "wedged": true in the body so the two failure modes
are distinguishable. /health is unchanged and stays a plain 200 ok.
Two details that matter if you have this wired to a load balancer:
- A cursor that has never yet made progress is judged from
started_at, so one wedged since block
one eventually reports unready rather than "just starting up" forever. - A caught-up cursor is not wedged.
last_progressonly stamps on an actual value change, so a
nest at the tip stops stamping the moment it arrives. Without a guard onlag > 0every healthy
tip-following nest would have reported unready after the threshold.
Structured logs: --log-format json (#302)
A global flag, valid before or after the subcommand:
nuthatch --log-format json dev
nuthatch dev --log-format jsontext remains the default. json emits one object per line - level, target,
message/fields, timestamp - so you can point an aggregator at stdout instead of scraping a
human-readable format that was never promised to be stable. A TipHeartbeat event is emitted on the
tip path, so "is it alive" is answerable from the log stream alone.
Two silent wrong answers
A dropped row in the hot store (#567). recent_by_table parsed each stored row and, on failure,
skipped it. A malformed row therefore returned a short result set rather than an error: the query
looked healthy and was quietly incomplete. It now errors.
An ADOPT dropped the incoming nest's authored files (#587). migrate's adopt path used an
allowlist, so anything it had not been taught about - views/, semantic.toml, an llms.txt, a
.claude/skills/ tree - was silently left behind. It now skips only what is unambiguously the
source's own derived data and copies the rest. Contributed by PaulieB14.
Absent data rendering as success is the failure mode this project treats most seriously, and both of
these were instances of it.
nuthatch doctor runs on a nest the serving path refuses (#582)
doctor loaded config through the same strict path as serve, so the tool you reach for when a nest
misbehaves refused to run on precisely the nests that were misbehaving. It now loads for diagnostics:
it parses, reports what is wrong, and does not apply the refusals whose purpose is to stop a bad nest
from serving.
A webhook declaring finality = "tip" is refused at load (#577)
finality parsed fine either way and nothing downstream honoured it, so a [[webhooks]] entry
asking for tip delivery got sealed delivery and no warning. Config load now refuses it, naming the
offending entries. Same shape as the [[calls]] refusal from #262: a load error costs a restart, a
silent wrong answer costs trust.
ChainCursor::shutdown (#407)
"Stopped" is a state a cursor reaches rather than a process that is aborted. Five e2e fixtures each
hand-rolled their own abort-and-hope; they now call one method. Contributed by PaulieB14.
Fuzzing the decode path (#290)
Three cargo-fuzz targets - abi_json, abi_arbitrary, decode_log - with a checked-in seed
corpus, plus a stable cargo test oracle so the guard is exercised on every run and not only under
libFuzzer.
Stated plainly: the fuzz job is advisory, not a gate. It runs continue-on-error because dbsp
ICEs the compiler under sanitizer instrumentation (#581), reproduced on two nightlies and racy with
it. Coverage exists; a gate does not. #593 tracks what has to happen first, and the honest answer is
that the decode path needs extracting into a crate that does not link dbsp.
CI can no longer deadlock itself (#584, #586)
Swatinem/rust-cache gates its save step on success(), so a job killed by an expired
timeout-minutes seeded nothing and the next run started cold too - a cold build that cannot fit in
the ceiling never gets the warm cache that would let it fit. Proven step-level rather than inferred:
on run 31824503060 the rust-cache post step was skipped while checkout's succeeded in the same
grace window.
cache-on-failure: true on all seven invocations turns that deadlock into one slow run. Raised
ceilings (#585) alone would not have fixed it. The last invocation to get it was the fuzz job's,
which is the one that matters most: 40m26s observed against a 45-minute ceiling, and
continue-on-error means a timeout kill reports success, so that particular deadlock would have
tightened with nothing on screen to say so.
A reviewed-by signature check now refuses a PR whose body carries no Reviewed-by: line (#593).
Documentation that had drifted
verification.mdnow states which release each level was verified on; a bare "yes" ages
silently. Level 5 had read "verified across machines" since v0.9.3 - two majors and a 2.0 that
changed the unit of storage. Re-run on 2.4.0 across two real machines: lease handover with a real
owner_fenceincrement, 2.2 M blocks indexed by a remote writer into the other machine's Postgres,
3 M blocks indexed straight through a control-plane outage. Clock skew is the one line still
reading v0.9.3, and says so.prod-readiness.md§11 claimed a multi-machine run had never happened, a day after one had.README.md: the chains paragraph was false -initworks on any EVM chain via--rpc, not
only the ones listed. Query-error hints were undocumented.docs/operators.mdpinned its two copy-pastedocker runcommands at:2.0.0, five releases
stale. Nothing was broken by it, which is why it survived five releases: an old image starts up
perfectly well.- Production verified through 2.4.0 (#441), and the live credential probe run (#428).
scripts/version-check.sh is added so the next release does not need three passes at this.
Upgrading
Drop-in, as every release since 0.3.0 has been: swap the binary and restart. No data migration, no
flag changes, no unit changes.
Artifacts
nuthatch-x86_64-unknown-linux-gnu.tar.gz (embedded, zero external services),
nuthatch-scaled-x86_64-unknown-linux-gnu.tar.gz (adds worker / control / serve --hot-store),
nuthatch-aarch64-apple-darwin.tar.gz, and ghcr.io/nightswatchhq/nuthatch:2.5.0 plus
:2.5.0-scaled. Each tarball ships its own .sha256; glibc floor is 2.34.