Releases: nightswatchhq/nuthatch
Release list
v3.0.1
A one-line fix to a metric, and a great deal of measurement that changed what we believe about our own
system. Nothing in the data path moved.
The fix
nuthatch_sealed_through reported 0 after every restart while the query path knew the true
watermark all along. The gauge was only ever written when a seal happened, so between a restart and the
next seal /metrics said 0 and /sql provenance said the truth. Two surfaces disagreeing about one
fact, and the wrong one is where Prometheus looks.
It has always done this - a 2.7.1 nest restarted alongside a 3.0.0 one behaves identically - but it
matters more now that 3.0.0 ships six nuthatch_entity_* series inviting alerts on that surface. An
alert of the form "sealed_through has not advanced" fired after every restart of a perfectly healthy
nest, and an alert that cries wolf gets muted.
It now seeds from the store at startup, where the watermark has always been durable.
What we measured, and what it changed
None of this alters the binary. It is recorded because a claim without a number is an opinion, and
several of ours turned out to be wrong.
Sealed segments are never compacted. The oldest production nest holds 10,923 files at a 6 KB
median, and the smallest is one row in 545 bytes. It costs: a table with 3.1x more rows spread over
3.1x fewer segments answers COUNT(*) 20% faster. A file costs roughly 0.14-0.18 ms.
docs/bench/segment-layout.md.
The benchmark noise floor now has a number, and it has two halves. For a single client, medians of
15 runs reproduce to 3% - but a single run varies by up to 44%, and 92% on the cheapest query. Under
concurrency the median stops being the right statistic entirely: at four clients it falls to 12 ms
while p95 holds at 92 ms, because the distribution goes bimodal and the median tracks the fast mode
while hiding the tail. Throughput saturates around 120 req/s. docs/bench/noise-floor.md.
The Linux binary has two ABI floors and the README named one. It links libstdc++ - because it
embeds DuckDB - and needs GLIBCXX_3.4.29. Every platform we list clears it, so this was
incompleteness rather than a broken promise, but a reader on new glibc with an old libstdc++ met a
requirement nobody had stated.
We do not verify that the chain we indexed is the chain consensus agreed on. Hash-linking proves
the headers we were served form an internally consistent chain. It does not prove that chain is the
canonical one, because settling that needs consensus-layer data nuthatch does not read. An endpoint
that lies consistently is indistinguishable, to us, from one that tells the truth.
docs/verification.md now says so as a stated non-claim.
RFC-0042, for those following it
Slice 0 is complete on both release targets. DuckDB is 93% of native artefact bytes and the sole
reason a C++ runtime is linked - but it is 10.6% of clean build time on Linux and 8.0% on macOS,
while wasmtime and cranelift are roughly twice that on both. The RFC's premise that it dominates
build time is measured false.
Six DuckDB roles, not the four §9 assumed, and two are product-visible: the admissible function
vocabulary for entities.toml comes from duckdb_functions(), and the engine version string is
written into grafting identity.
The RFC is now unfrozen in full. Nothing has been decided about removing anything.
v3.0.0
Authored incremental entities, stable. A nest declares a relation in entities.toml and nuthatch
maintains it as blocks arrive instead of recomputing it on every query. That is RFC-0041, and it is
what the major bump is about.
3.0.0-alpha.1 carried the feature; this carries it plus what a day of soaking it on real chains
found.
Drop-in, with one sentence of small print
Replace the binary and restart. No config migration, no data migration, no re-index. A nest that
declares no entities behaves exactly as it did on 2.7.x. Verified on production data: all 73 tables of
the Lodestar Horizon nest re-counted at a pinned watermark before and after the swap, identical,
2,376,135 rows.
The small print: adding an entities.toml to a nest that already has history means its next restart
seeds the entity from the whole sealed corpus. Measured rather than warned about: 249,979 rows across
733 sealed segments in 1.9 s, and a larger Horizon corpus of 346,288 rows across 2,985 segments in
2.4 s. Independently corroborated on macOS at 2 s, including after a SIGKILL.
What an entity buys
The Lodestar indexer_rewards panel, which was an authored view over 733 sealed segments and is now a
maintained relation. Same nest, same data, same query, p50 over 25 runs:
| p50 | p99 | |
|---|---|---|
| authored view, recomputed per query | 2.15 s | 2.20 s |
| maintained relation | 87.7 ms | 92.1 ms |
Identical rows, all 82 of them. One block's update against a relation of 309,548 groups is 285 µs,
against 61 groups it is 195 - flat, which is the point, and not true until late in the work.
Verify who built your binary
New, and the reason to bother is that a .sha256 sidecar downloaded beside the tarball it describes
proves only that the file did not corrupt in transit. Anyone able to replace one could replace the
other in the same breath.
gh attestation verify nuthatch-x86_64-unknown-linux-gnu.tar.gz --repo nightswatchhq/nuthatchEvery release artifact now carries a build provenance attestation, signed by GitHub's identity for the
workflow run that produced it and recorded in a public transparency log: which repository, which
commit, which workflow. --repo is load-bearing - without it an attestation from any repository is
accepted. The release workflow runs the same check against its own assets before publishing, so a
release whose provenance does not verify never goes public.
Every third-party Action in the build is also pinned to an immutable commit SHA rather than a movable
tag, because provenance signed by a workflow running whatever a tag was retagged to proves rather less
than it appears to.
Five defects the soak found that the test suite could not
Each needed a real nest and real chains.
A pool-wide 429 killed a nest with block N alone exceeds the provider's getLogs result cap - a
diagnosis that was never true, whose stated remedy (a bigger provider) does nothing about throttling.
Every endpoint refusing one request was escalated to "narrow it", the chunker halved to a single block,
and there was nothing left to narrow. Found on a production box crash-looping twice an hour.
An answer could be labelled more current than its own rows. /sql executed against a maintained
relation and then read applied_through afresh - two lock acquisitions - so a batch landing between
them produced rows from block N carrying the label N+8. 1 in 12 reads on a 0.25s-block chain. The
relation was never wrong; the citation was.
A backfill that stopped advancing ran forever. It now fails by name after 64 attempts without
progress. At tip it warns instead, because a tip loop faulting on somebody else's outage turns a
transient problem into a dead nest.
A restart silently kept only groups mod 10,000 of a relation (fixed in the alpha), and
one block's update cost 72 ms and grew with history (likewise). Both invisible from CI.
Also in here
--seal-direct refuses a nest declaring an entity rather than completing with an empty relation. Six
Prometheus series per entity: applied_through, current, rows, faulted, unavailable,
seconds_since_progress. [[alerts]] gains format = "discord" for sinks wanting {"content": …};
the default is unchanged and every existing sink is byte-identical.
A /sql request was defining a view for every table in the nest's manifest whether the statement named
it or not, at roughly 62 µs per sealed segment - SELECT 1 cost 2.5 s on a 38,428-segment nest
before reading a row. Views are now defined for what a statement can reach. degraded_tables narrowed
with it: a query reports degraded segments for tables it read, and the nest-wide sweep runs
separately rather than on the caller's time. That is the one behaviour change on an existing surface.
What has been soaked, and what has not
An entity followed Arbitrum for over 24 hours without once falling behind its head, checked against a
full recompute at the same block every hour: exact every time. A parallel Ethereum nest took a real
reorg and converged. Six restarts alternating SIGKILL and SIGTERM on a live nest, exact after each.
Not yet soaked: a process carrying an entity restarted after days rather than hours, and the two
per-cursor RSS admission constants remain one measurement on one machine.
docs/releases/3.0.0-alpha-stresstest.md says what is still open and in what order it gets answered.
v3.0.0-alpha.1
A nest can declare a relation in entities.toml and nuthatch maintains it as blocks arrive, instead
of recomputing it on every query. That is RFC-0041, and it is what the major bump is about.
This is an alpha, and it is an alpha for a reason. The endurance evidence is thin and still
accumulating: at the time of this cut an entity had followed a chain for twenty hours without falling
behind, and no process carrying one had been restarted after days of uptime.
docs/releases/3.0.0-alpha-stresstest.md says what is still unanswered and in what order it gets
answered; docs/bench/3.0.0-alpha-live.md carries the figures as they stand, including the two items
that are not finished and the one the run cannot test at all.
Drop-in, with one sentence of small print
Replace the binary and restart. No config migration, no data migration, no re-index. A nest that
declares no entities behaves exactly as it did on 2.7.x.
The 2.x to 3.0 jump says this is significant, not this will break you.
The small print: adding an entities.toml to a nest that already has history means its next
restart seeds the entity from the whole sealed corpus. Measured, so it takes a figure rather than a
warning: on the Lodestar nest that is 249,979 rows across 733 sealed segments in 1.9 seconds, and
on a larger Horizon corpus 346,288 rows across 2,985 segments in 2.4 seconds. A restart gets
slower by about that much, once, per entity.
What an entity buys
The Lodestar indexer_rewards panel, which was an authored view over 733 sealed segments and is now
a maintained relation. Same nest, same data, same query, p50 over 25 runs:
| p50 | p99 | |
|---|---|---|
| authored view, recomputed per query | 2.15 s | 2.20 s |
| maintained relation | 87.7 ms | 92.1 ms |
It returns the same 82 rows the view does, exactly.
And it costs what it should to keep current: one block's update against a relation of 309,548 groups
is 285 microseconds, and the same update against 61 groups is 195. Flat, which is the whole point,
and not true until late in the work.
Faster for every nest, entities or not
A /sql request was defining a view for every table in the nest's manifest whether or not the
statement named it, at roughly 62 microseconds per sealed segment. SELECT 1 cost 2.5 seconds on
a 38,428-segment nest before reading a row. Views are now defined for what a statement can reach.
The nest-wide corruption signal moved with it. A query reports degraded segments for tables it
read; the whole-nest sweep runs separately rather than on the caller's time. If you parse
degraded_tables out of a /sql response, that field has narrowed in meaning, and it is the one
behaviour change on an existing surface here.
Four defects the measurements found
None was visible from the test suite. Each needed a real nest.
A restart silently kept only groups mod 10,000 of a relation. DBSP splits a transaction into
internal steps and each writes the output handle separately; nuthatch read it once, after. On a real
corpus 309,549 distinct keys went in and 9,549 groups came out - nothing faulted, nothing logged, and
every group it did keep held the right value.
One block's update cost 72 ms and grew with history. The published relation was copied in full
after every batch, so a block cost the size of the answer rather than the size of the change.
/explain answered differently for the same query depending on whether another request had
warmed the pooled DuckDB connection: 400 table does not exist cold, 200 valid warm.
A tip race was classified as a provider cap and killed a nest. Asking for a block the provider has
not served yet is normal at tip; the error carried the words block range, matched the cap marker
list, and the chunker narrowed a window that was never too large until it gave up. Found by an
overnight run, three and a half hours in.
For watching it
Six new Prometheus series per entity - applied_through, current, rows, faulted,
unavailable, seconds_since_progress - because a maintained relation previously had no presence on
/metrics at all, and the only way to ask whether it was keeping up was to poll /ready and parse
JSON.
[[alerts]] gains format = "discord" for sinks that want {"content": …} rather than our payload.
Default is unchanged and every existing sink is byte-identical.
Also in here
--seal-direct refuses a nest that declares an entity rather than completing with an empty relation.
The DuckDB connection cache keys on content rather than on a clock it cannot trust. A zero-width
backfill window is refused instead of spun on. /ready answers for a seal-direct that has stopped
sealing.
nuthatch 2.7.2
A patch release for the operational work that landed after 2.7.1.
Drop-in. Replace the binary and restart. There is no config migration, data migration, or re-index.
Bounded runtime state and honest metrics
The runtime now bounds and evicts its DuckDB connections as mounts change, rather than keeping a
connection for every nest that has ever appeared. Departed nests also leave the process-wide metrics
registry. This keeps a long-lived multi-nest process from retaining dead state indefinitely.
Metrics no longer publish arbitrary cross-chain values as a global tip or lag. The process-wide
series are now scoped to values that can be represented honestly, while per-nest readiness and
progress remain the source of truth for a specific chain.
Analytics and control surfaces
Analytics invalidates cached DuckDB views when their nest inputs disappear. The SQL surface refuses
the mutation-shaped WITH forms DuckDB accepts, and the MCP path now invokes the advertised tool
rather than returning a description of one.
The wide-ABI admission path has a measured 2 GB ceiling, and the storage-path checks cover the clean
seal-direct route rather than merely an already-populated store.
Release engineering and documentation
CI has stronger required-context and behavioural checks. The operator, benchmark, launch, and
mapping documentation now records the measurements and limitations it relies on, including where a
normal query-time view is not incremental materialisation.
Not in this release
RFC-0041 authored incremental entities are still under implementation. The compiler gate and
authoring validation are deliberately not presented here as a shipped indexing-time feature. That
work belongs to 3.0.0 once its lifecycle and serving paths exist.
v2.7.1
A fix release. 2.7.0 was the binary an evaluator actually ran, and it lied in four places that
matter: --rpc still dialled public endpoints, /ready was green on a dead pool, init of stETH
scaffolded a nest that would index nothing, and an oversized eth_getLogs range retried forever.
Drop-in. Swap the binary and restart; no config changes, no data migration.
--rpc is the whole pool
Passing --rpc used to prepend your endpoint to the bundled public defaults. init wrote all of
them into rpc_urls, and dev --rpc still failed over onto nodies and drpc. A paid-only run was
neither paid-only nor measurable.
An explicit --rpc is now the complete pool, at init, add, dev, metadata, and in the
runtime. No --rpc still uses the configured or built-in list unchanged.
/ready is no longer green on a dead RPC
A nest whose only endpoint was http://127.0.0.1:1 announced API live and answered /ready 200
with ready: true, last_block: 0, stalled: false. A load balancer would have sent traffic at an
empty process.
A failed first poll now sets initial_poll_failed. /ready is 503 immediately. The 90-second
start-grace no longer covers a pool that has already proved itself dead.
init no longer scaffolds a nest that silently indexes nothing
stETH. The proxy is EIP-897. 2.7.0 resolved the proxy ABI (one table, ProxyDeposit) and warned
that zero rows would index. init now follows implementation(), then Sourcify, then keyless
Blockscout, then keyed Etherscan. Live: 32 tables, and a 4,000-block tip backfill stored 1,559
Transfer, 1,559 TransferShares, 152 Submitted.
Sourcify miss. 2.7.0 demanded ETHERSCAN_API_KEY as if it were the only option. Blockscout
serves the same getabi API with no key on mainnet, Base, and Gnosis. BSC has no instance we have
measured; the error now says so, and names --abi.
Upgraded proxies. init/add resolve the current implementation. For Graph staking that is
Horizon-era names; ~400 million blocks of Arbitrum StakeDeposited decode to empty tables and look
healthy. Two guards, neither of which reconstructs history: a height heuristic when the current impl
was deployed after the proxy, and a tip-vs-deploy log sample (HistoricalMismatch) when recent
events match and history near deployment does not. The operator is told to add a second
[[contracts]] entry with the legacy ABI. We warn. We do not auto-vendor the earlier ABI.
Oversized getLogs ranges shrink
Alchemy free-plan ranges over 10000 blocks are not supported classified as a blip, so
index_loop retried the same 81,920-block window until /ready went stalled. That phrasing is a
range cap. Chainstack's Block range limit exceeded is the same class. Both shrink.
What a nest costs, honestly
nuthatch_rpc_requests_total is HTTP POSTs: one per request or batch envelope. It cannot price a
provider bill. nuthatch_rpc_methods_total{method="…"} counts individual JSON-RPC methods,
including those inside a batch. A batch of 200 eth_getBlockByNumber is 200 on the method series
and 1 on the request series.
docs/operators.md prices Arbitrum tip-following against Alchemy pay-as-you-go as ~$134/month,
of which ~$93 is header fetches. Computation from published CU rates, not an invoice. Paid RPC (or
your own node) is the golden path; bundled public endpoints are the two-minute on-ramp.
Also on this tag
The nocturnal-nightjar machines landed in the same window and ride along: a recorded RPC tape for
bench (--record / --replay), a mutation-coverage nightly on the delightful core (red-capable,
required nowhere), and a CI gate that every documented command and flag is one the binary accepts.
Upgrading
Binary swap. On-disk state is forward-readable, dev flags are unchanged, no re-index. --rpc
behaviour is the one change an existing invocation will feel: if you passed it and relied on the
bundled public fallbacks, those fallbacks are gone. That was the bug. Minimum supported Rust is
1.95.
v2.7.0
A nest that declares [[calls]] can use --seal-direct again, a factory nest stops aborting its
backfill on a provider cap, and this page stops quoting performance numbers it cannot stand behind.
Drop-in. Swap the binary and restart; no config changes, no data migration.
A minor rather than a patch for one reason: 2.6.x refused --seal-direct on a nest with
[[calls]] and 2.7.0 does not. That is new behaviour on an existing surface, and calling it a
patch would be the small dishonesty this release is otherwise about.
A factory nest no longer dies on a provider cap
uniswap-v2 on mainnet aborted its backfill outright:
Error: getLogs (children) 25791463..=25811399
HTTP 400: Log response size exceeded
backfill_direct_factory fetches twice per chunk. Pass 1 - base contracts plus children already
known - caught an over-cap response and narrowed. Pass 2 - the children discovered in this very
chunk - carried a bare ?. It was the only one of nine is_result_too_large sites in the file
treating a provider cap as fatal, and it is the pass more likely to meet one, because a freshly
created child is exactly the contract whose first logs arrive in a burst.
Found in the field rather than by reading.
[[calls]] and --seal-direct are no longer mutually exclusive
2.6.0 shipped pinned eth_call (RFC-0023 tier 3) and 2.6.x refused --seal-direct whenever a nest
declared one, because a seal-direct run would sail past every sampled block and seal the range with
the calls table silently absent. The refusal was correct. It was also expensive: it forced the
entire range onto the slower path rather than only the blocks that needed a read.
All three seal-direct paths now resolve declared calls, so the refusal is gone and the sealed
segment matches what the streaming path would have produced.
Measured honestly, and the number is smaller than the one that prompted the work: on two identical
near-tip nests built from the same commit, one with a [[calls]] block and one without, the
difference was ~7% - not the 5.5x a single field report suggested. That field report was one run
on a 454-million-block nest with ~2,725 pinned reads, roughly 300x denser than the measurement.
Both are true of what they measured; only one of them is a measurement.
A declared event that had never fired took a whole view down
Declare an event a contract really emits, on a chain where it has not emitted yet, and no table was
created - so any view referencing it failed to load in its entirety. One view supplying fourteen
fields lost all fourteen because a single referenced table did not exist, and nothing in the logs
explained either state.
A declared-but-unfired table now resolves as an empty typed view. The nest reports which tables are
in that state at startup, so the two cases are distinguishable rather than silent.
Two narrower instances of the same staleness class are fixed alongside it: a schema.json that has
fallen behind the config no longer silently keeps its stale column set, and the skip that explains
why a hand-edited nuthatch.toml did not move schema.json now logs at info rather than debug,
where an operator running the default filter can actually see it.
Fewer requests, fewer retry storms
- Tier-3 reads cost one round trip per sampled block instead of two. The second fetched a block
hash that the batched header call already had. block_timestampsno longer descends serially. A 454-million-block backfill produced six retry
storms against an archive endpoint; the top-level split now runs its halves concurrently.doctorprobes with the nest's full contract set, not just the first declared contract. A
multi-contract nest was getting advice measured against a narrower question than the one its
backfill would ask, and the recommendation came out optimistic.
Operator and security hygiene
benchno longer prints a--state-rpcURL in full. Archive endpoints carry API keys, and
that URL was going to a log.mainstopped going red at random. A test asserted an exact request count across a
tokio::try_join!that cancels its losing branch, so the total was never a constant. It failed
nine runs in ten on one machine while CI showed green.
This page no longer quotes numbers it cannot stand behind
The honest headline of this release, and the least comfortable part.
docs/benchmarks.md carried ~8.7x for seal-direct and ~20x for the pipeline. Both were ratios
over a 289 events/sec baseline measured two weeks before a fix to the benchmark harness itself -
it had been writing one database transaction and one fsync per row, which the indexer has never done.
The multipliers were never recomputed, and they had been quoted in a grant document since.
Re-measuring did not produce better multipliers. It produced this:
- seal-direct came out at 0.92x - slower than the path it is supposed to beat - contradicting
both the architecture and a 5.2x measured hours earlier at the same commit - a 3.8x spread inside a single arm in one session on one machine
So the page now says what is true: these figures measure a public endpoint's mood and the machine's
neighbours as much as they measure nuthatch, the discrepancy is published rather than resolved by
picking the flattering number, and a deterministic replay rig is being built before any multiplier is
quoted again.
The benchmark harness itself was also measuring the wrong thing: every seal-direct path passed an
empty [[calls]] slice regardless of what the nest declared, so bench could not measure the
feature this release is named for. Both arms now resolve declared calls, and bench refuses rather
than silently under-measuring when a nest declares [[calls]] and no --state-rpc is given.
Upgrading
Binary swap. On-disk state is forward-readable, dev flags are unchanged, no re-index. Minimum
supported Rust is 1.95 - measured, not asserted.
v2.6.3
The first run stops stalling. If you tried nuthatch on Ethereum mainnet and concluded it was broken,
it was, and this is the release that fixes it.
Drop-in. Swap the binary and restart; no config changes, no data migration.
A first run on mainnet no longer stalls
Point nuthatch init at a contract, run nuthatch dev, and on 2.6.2 you would get 15 events in 90
seconds - or, behind a provider that refuses without explaining itself, no progress at all, forever.
The window controller grows fourfold on an empty range. An empty range says nothing about capacity,
so from a 1,000-block start it reached 100,000 blocks in four steps having never been served
anything near that. Every mainnet provider refuses a range that wide - Alchemy caps eth_getLogs at
10,000 - and discovering the real cap from there costs a binary search down: refused at 100k, 50k,
25k, 12,500 before anything succeeds. Fifteen round trips that returned nothing, each carrying a
backoff.
What hid it is that the recovery was invisible to the controller. The splitter halves a refused range
until the pieces succeed and returns the merged result, so the caller saw success and grew fourfold
again - on the strength of a success that never happened at that width.
Growth is now bounded by what a provider has actually served whole. One step past proven capacity is
exploration; four steps past it is guessing.
| crossing a 200,000-block prefix behind a 10,000-block cap | 2.6.2 | 2.6.3 |
|---|---|---|
| requests refused | 24 | 4 |
| widest range attempted | 100,000 | 16,000 |
| terminates | no | 0.04s |
This took five attempts, and two existing tests killed three of them - one defending cheap traversal
of the empty blocks in front of a late-deployed contract, one defending against an exponential
fan-out that turned 3 requests into 2,047 against a dead endpoint. Both requirements are real and both
still hold.
The front door, finished
- A bare
nuthatchprints the same grouped listing--helpdoes. It was gated on argument
count, so the grouping added in 2.6.2 was missing from the shortest way to ask for it. Still exits
2 on stderr: a missing subcommand is a usage error, and prettier output must not make a script that
runsnuthatchby mistake see success. initprints prose, all the way through. Warnings from the RPC layer used to arrive mid-block
with a timestamp, a level and ANSI colouring. They arrive as· messagenow, in the same idiom
as the ticks around them. Reformatted rather than suppressed: they are the only signal that a probe
is struggling whileinitappears to hang.
For operators and contributors
live-endpointsretries each endpoint three times before calling it dead. Two shipped defaults
once failed a scheduled run and answered fine two days later, and the job sat red throughout. A red
nobody trusts is a red nobody reads.- The review-signature gate is real. It checked that a signature was non-blank, which the literal
stringpendingsatisfies; it could not detect a self-signature; and it was required on no
branch, so even a red changed nothing. All three are fixed: a committed roster of who may sign, and
the context is now enforced. - The config drift gate reads
queries.toml's structs. A key added toCeilingorNamedQuery
used to ship undocumented, invisible to the agent skill that is supposed to describe it. - The stranded-work check no longer fails open. It reported "Nothing. Every PR is reachable from
main" when its query failed - a clean bill of health from a check that never ran.
v2.6.2
The front door. No new capability - a feature freeze runs to the end of 2026
(docs/roadmap-2027.md) - and this release is the first instalment of what
replaces it: making the parts a stranger meets first actually good.
Drop-in. Swap the binary and restart; no config changes, no data migration.
--help no longer opens with a fleet orchestrator
Twenty-four subcommands, ordered by when each was added, with worker and control - a writer pool
and a control-plane API - ranking fifth and sixth, above sql. A solo dev reading top to bottom
met cursor leases before they met querying.
They are grouped now. CORE is init, add, dev, sql, mcp - the whole happy path and the
AI hook, and nothing else. Then OPERATING, SCALED, and the compliance pack. Nothing was removed and
no capability changed; the enterprise breadth is still there, one screen further down, which is
where RFC-0015 always said it should be.
init auto-detects all seven chains, not three
chains::lookup() knew seven chains. chains::all() - the list init walks when you omit
--chain - returned three. So a contract on Optimism, Polygon, BSC or Gnosis could be indexed by
name and never found by auto-detect, while the README stated in bold that omitting --chain probes
all seven.
Two lists of the same set, one of them silent when it fell behind. The existing test could not see
it: it asserted every chain in all() resolves via lookup, which is trivially true of any
subset. One direction of a two-list invariant, and the missing direction is always the one that
rots. The new test derives its expectation from the alias table that already enumerates all seven,
so there are two lists in that file rather than three.
Fixing it immediately failed a second test, which is the more interesting part: BSC ships one
endpoint where RFC-0030 §4 requires two, and that rule had been passing only because the chain
breaking it was not in the list the rule iterates. Three candidate replacements were probed with
nuthatch doctor on 2026-08-21 and none is usable - two refuse the getLogs probe outright, the
third is tip-following only. So the exception is recorded in the test with its evidence, and written
to fail the day BSC gains a second endpoint, so the carve-out gets deleted rather than left behind.
Polygon can actually backfill
Polygon shipped in 2.6.0 with polygon-bor-rpc.publicnode.com listed first. By the next day that
endpoint had no archive depth and its getLogs probe failed outright, so a from-deployment backfill
could not use it at all, and log_window was 2,000 against an endpoint that serves 40.
The archive endpoint leads now, log_window is 40, and Polygon joins the live-endpoints probe so
this is measured on a schedule rather than once.
Worth stating plainly, because it is the more useful lesson: chains.rs recorded "Measured
2026-08-19: gives a 5,120-block window". That was true when written and false one day later. A
recorded measurement is a snapshot presented as a property. Run nuthatch doctor --rpc <url>
before trusting a long backfill to any endpoint, ours included.
doctor had its advice the wrong way round
doctor --rpc <url> with no --address probes a range nothing matches, which measures the
provider's raw block-range ceiling and says nothing about what a real nest sustains. The number it
printed was being read as a ceiling. It is a floor: every measurement on record has
address-filtered limits coming in under the raw range, so the range-only figure is a conservative
lower bound, not a cap.
Also: doctor --dir <nest> now derives --address from the nest's first declared contract, so the
probe reflects the filter the nest will actually issue rather than an empty one.
A correct nest no longer starts with 38 warnings
Rename a contract's alias in nuthatch.toml and every table under the old alias became its own
semantic.toml describes table X, which the registry has no decoder for line. On a real nest that
was 38 warnings for one edit, which is the number at which people stop reading warnings.
Whole-alias orphans - where the alias prefix is gone from the registry entirely, meaning a rename -
collapse into one warning per alias, carrying the real table count and naming the edit that
clears it. Genuine per-table drift still reports per table, because that is a different fault.
Smaller
init's output is clean. The ABI resolver's name moved out of a rawtracing::info!line -
ISO timestamp, log level, ANSI codes, straight through the middle of the→/✓prose - and into
the tick where it belonged.- Four security tests stopped being red on macOS. The admin-exposure suite bound
127.0.0.2,
which Linux aliases on loopback and macOS does not, so they failed at the socket before any
nuthatch code ran. They bind a real::1now. A local suite that is red by default is a local
suite people stop reading, and these are the ones you least want scrolled past.
Under the hood, for contributors
Two CI gates that exist because of faults nobody would have found by reading:
- A sprint whose issues are all closed must be reachable from
main. Six PRs of good work once
merged into a sprint branch, closed their issues, showed 6/6 on the dashboard, and shipped in
nothing. - The RFC index is now gated against each RFC's own
Statusheader. It found thirteen rows that
disagreed with the documents they indexed.
Known: the second gate's stranded-PR rule reports "Nothing" when its query fails rather than "not
judged" (#699). Contributors only.
v2.6.1
A fix release. Two faults found by running 2.6.0 against production data rather than by reading it,
one of which will bite anyone who takes up the [[calls]] feature 2.6.0 introduced.
No new capability, no config changes, no data migration. Swap the binary and restart.
A transient RPC failure no longer kills a nest (#651)
The one to upgrade for. On 2.6.0, a single dropped connection during a pinned eth_call ends the
process:
INFO backfilling 87.6% - block 440066152 of 496262427, 1178466 events, 248 ev/s
Error: pinned eth_call batch at block 440200000
Caused by:
transport error: error sending request for url (...)
That is a real run of ours: eight hours in, at 87.6%, discarded by one blip.
Every RPC fetch on the sealed-history path has had never-give-up retry with capped backoff for
months. The four fetches that arrived with 2.6.0's parity features did not - the tier-3 call batch,
the IPFS document path's block-hash lookup, the tier-3 block-hash lookup, and the top_level_calls
block-bodies fetch. All four now use the same helper as their neighbours.
This matters more than one lost run because [[calls]] implies a long backfill by construction:
--seal-direct is refused while calls are declared, so tier-3 nests run the slower hot-store path and
are exposed for hours at a time. The feature and the fragility shipped together.
If you hit it on 2.6.0, restart rather than re-index. Nests resume from where they stopped; ours
caught up in 17 minutes.
A nest refuses to serve rows a different config produced (#653)
Add an event to a running nest's nuthatch.toml, restart, and 2.6.0 would start, notice it was
already at tip, index nothing, and serve - stamping the new registry_hash onto data the old
configuration produced. Every query then reported provenance under a registry that never ran.
The only visible symptom was a view failing to load, and only if that view happened to reference one
of the new tables. A change touching tables no view references said nothing at all.
A nest now compares the stored data's registry against the configuration and refuses to start when
they differ, naming both hashes and the remedy. blob.rs already did this for packed nests at mount
time; the ordinary dev --dir path did not, so the guarantee held for mounted nests and quietly did
not for the path everyone starts with.
Upgrading an existing nest is unaffected. A store written before this release has no recorded
hash, and refusing those would break every running deployment for a fault they may not have, so the
hash is adopted and a line is logged saying it was recorded rather than verified. If a nest's config
has changed since it was indexed, re-index it.
Known issue, unchanged in this release
A first run on Ethereum mainnet against the default public endpoints is much slower than it should
be, and behind a provider that refuses over-wide ranges with a rate limit rather than a width
complaint it can stall entirely. Tracked in #672,
with a deterministic reproduction. Passing --window sized by nuthatch doctor --rpc <url> avoids
the worst of it in the meantime. Chains other than mainnet are not materially affected.
v2.6.0
The release that closes subgraph parity. A nuthatch nest can now do the two things a subgraph could
and a nest could not: call a contract, and read a document off IPFS. Four new chains, and two
silent-wrong-answer fixes found by running the new code against production data rather than against
fixtures.
Derived from git diff v2.5.0..main and from the live runs described below, not from PR titles.
Contract calls: [[calls]] (RFC-0023 tier 3, #262)
A nest can pin a contract read to a block and store the result as a table.
[[calls]]
name = "grt_total_supply"
contract = "0x9623063377AD1B27544C965cCd7342f7EA7e88C7"
calldata = "0x18160ddd"
every = 100000Calls may also be parameterised from the row that triggered them, so the shape a subgraph expresses
as contract.balanceOf(event.params.user) has an equivalent here. Needs --state-rpc pointed at an
archive node, because a pinned read at an old block is a state query and a pruned node cannot answer
it.
Determinism is preserved by the pinning: the block is fixed, so the answer is fixed, and two
operators re-executing the same nest get the same bytes. The result is content-addressed on
(chain_id, block, contract, calldata).
Verified in the field, not in a fixture. On a seven-contract Arbitrum nest: 2,725 pinned reads
across blocks 42,500,000 to 314,900,000, zero reverts. Seven of those were checked value-for-value
against an archive node and matched exactly, to the wei - including GRT totalSupply at block
315,000,000, which is 3,204,166,815.30 GRT.
One cost to know before you turn it on. Tier 3 resolves inside process_window, and --seal-direct
is refused while calls are declared - a seal-direct run would sail past every sampled block and seal
the range with the table silently absent. Measured on a real nest, adding a single [[calls]] entry
took the same 454M-block backfill from ~12 minutes to ~66. The guard is right and stays; teaching
the seal-direct paths to resolve calls is the follow-up (RFC-0038 §6e).
IPFS documents: [[ipfs]] (RFC-0037)
A column holding a content address becomes a table of resolved, verified documents.
[[ipfs]]
name = "subgraph_metadata"
on = "gns__subgraph_metadata_updated"
cid_column = "subgraphMetadata"Every fetched body is re-hashed and checked against the CID it claims to be, so a gateway serving
the wrong document yields no row rather than a plausible one. The CID is extracted from whatever
shape the contract stored - a bare CID, an ipfs:// URI, a path gateway URL, a subdomain gateway
URL, a Kubo ?arg= API URL, or a raw 32-byte digest - and the host is discarded. That last part
is deliberate: the string comes from a log, so honouring the host it names would let whoever emitted
the event choose what your indexer connects to.
bytes32 columns resolve as of this release. The Graph's own GNS stores subgraph metadata that way,
because 32 bytes is what a sha2-256 digest actually is and Qm… is merely one encoding of it. A
CIDv0 is base58btc(0x12 0x20 || digest), so the digest is re-framed rather than re-hashed.
Value::Hash32 stays refused: that variant holds keccak(value) for an indexed dynamic parameter,
so those 32 bytes have exactly the shape of a digest and are not one, and accepting them would mint a
well-formed address for a document that has never existed.
Verified against a live gateway on a live chain, not a stub: a nest over Arbitrum GNS resolved
5 of 5 documents, all verified, with real displayName values in the content. Three of the five
CIDs had been computed by hand beforehand and matched, so this is agreement with the network rather
than the code agreeing with itself.
Gateways are configured with --ipfs, never in nuthatch.toml. A gateway is an access path, and two
operators resolving the same CID through different gateways must get the same bytes, so it must not
enter the nest's content address. Point it at your own node if you want no third party in the path.
Four new chains (#646)
BSC, Polygon, Gnosis and Optimism, each with measured endpoints and FinalizedTag finality. BSC
ships with a single endpoint because no keyless archive was found for it, which is stated rather than
papered over.
Fixes
Rows descending from no log could overwrite each other (#642). Block rows, call results and IPFS
documents have no log_index of their own, and enough of them shared a synthetic one that they
collided in the hot store - a silent overwrite, visible only as missing rows. There is now a reserved
band: 500_000..=624_999 for pinned calls, 625_000..=749_999 for IPFS documents, 750_000.. for
transaction-level calls, and 999_999 for block rows.
Worth noting how this nearly escaped: the first attempt to reproduce it used the seal-direct path,
which passed, because Parquet is append-only and an overwrite cannot happen there. The bug only
appears through process_window.
Call and IPFS tables were queryable but invisible. registry.schema() lists event tables only,
and is treated as the catalogue nearly everywhere. Measured on two running nests: a call table
holding 3,509 rows appeared in neither /tables nor /schema, /table/<name> answered 404,
and the semantic.toml drift check warned that a correctly-described table "has no decoder" - a
warning that fires on a correct config, which is how you teach operators to ignore warnings. One
omission, three symptoms, one fix: the catalogue is now composed from the registry plus calls plus
IPFS.
Also
nuthatch init --from-subgraphverifies the manifest it fetches against its CID, and reports
file/ipfstemplates as pointing at[[ipfs]]rather than at[[factories]].nuthatch-decodeextracted as its own crate - deterministic ABI decode with no dbsp dependency -
and its 18 tests now run in CI, which they had stopped doing when they moved.- The fuzz smoke job no longer passes by not running:
continue-on-erroris gone, ASan and
debug-assertions are back.
Upgrading
Drop-in. Swap the binary and restart; no data migration, no flag changes. Existing nests are
unaffected - [[calls]] and [[ipfs]] are opt-in, and a nest declaring neither behaves exactly as
it did on 2.5.0.