Repository navigation
v4.9.0
Nuthatch 4.9.0 lets a nest start from another operator's published history instead of backfilling it
over RPC. It is a minor because of the new seed subcommand and because the data identity no longer
counts hidden files, which moves the identity, and so the mirror prefix, of any nest that holds one.
Seeding from a mirror
A backfill is most of what a nest costs to start: about a million GraphOps credits for the Lodestar
allocations nest. Somebody running the same nest may already have that history in a mirror, but
nothing could start a nest from one (#1903).
nuthatch seed --from <mirror> --dir <nest>fills a nest that has not indexed.<mirror>is a
path,s3://bucket/prefix, or thehttps://address of a public bucket. Every file is checked
against its catalogue hash, an interrupted run resumes, and the nextdevfollows the chain from the
block after the mirror ends.- A seed snapshot under
<dataset>/_seed/carries what the reader's mirror withholds: each table's
provisional tail, a digest of that list, and the block the publisher's history begins at. A running nest's
--publish-targetwrites it once caught up, andpublish syncon a stopped nest. A mirror whose
history begins after the nest's declared start block is refused. - An interrupted seed leaves the nest marked, and
devrefuses it untilseedis run again, so a
half-installed catalogue is never indexed over. - The hashes show each file is the one the mirror's catalogue names. They do not show who wrote the
catalogue or that it is true to the chain; seed from an operator you would trust to run the nest.
Seeding a clean clone of graph-gns-nest from a public R2 bucket holding production's data: 23
segments, 18.9 MB, 2.4 s, then at the chain tip after six RPC calls and identical to production on all
eight tables.
Data identity
- Hidden files and directories no longer count toward the data identity (#1904). A stray
.duckdb/
extension cache,.cache/or.gitignoremade two copies of one nest compute different identities,
so neither could find the other's mirror. The data identity of a nest holding hidden files moves
once, which moves its mirror prefix; its NID, and so its mounts, stay as they are.
Engine
- Burrmill f7c225d brings the QoS statements back to 4.5.0 times within the cost of the exact double
sum (#1902): a group holds its first two values as doubles before it allocates buckets, and a
dropped join filter no longer stays in the scan's predicate.indexer_dayon the QoS copy, median of
15 fresh servers: 1,461 ms on 4.8.0, 1,315 ms now, 1,275 ms on 4.5.0.
Operations
- The release gate waits for a quiet box and refuses to measure on a loaded one (#1900, #1898): the
4.8.0 gate ran at a load average of 26 and reported the noise as a regression.
Documentation
- The operator guide said losing
nuthatch.redbcosts only the unsealed window. It costs the whole
backfill, because every resume marker lives there; back the two up together.
Compatibility and upgrade
Replace the binary and restart. There is no storage migration or re-index. A nest that holds hidden
files publishes under a new dataset prefix after upgrading; nothing stores the data identity, so only
an existing mirror's prefix name moves.