Skip to content

v4.9.0

Choose a tag to compare

@github-actions github-actions released this 05 Oct 15:57
· 561 commits to main since this release
8e114ef

Nuthatch 4.9.0 lets a nest start from another operator's published history instead of backfilling it
over RPC. It is a minor because of the new seed subcommand and because the data identity no longer
counts hidden files, which moves the identity, and so the mirror prefix, of any nest that holds one.

Seeding from a mirror

A backfill is most of what a nest costs to start: about a million GraphOps credits for the Lodestar
allocations nest. Somebody running the same nest may already have that history in a mirror, but
nothing could start a nest from one (#1903).

  • nuthatch seed --from <mirror> --dir <nest> fills a nest that has not indexed. <mirror> is a
    path, s3://bucket/prefix, or the https:// address of a public bucket. Every file is checked
    against its catalogue hash, an interrupted run resumes, and the next dev follows the chain from the
    block after the mirror ends.
  • A seed snapshot under <dataset>/_seed/ carries what the reader's mirror withholds: each table's
    provisional tail, a digest of that list, and the block the publisher's history begins at. A running nest's
    --publish-target writes it once caught up, and publish sync on a stopped nest. A mirror whose
    history begins after the nest's declared start block is refused.
  • An interrupted seed leaves the nest marked, and dev refuses it until seed is run again, so a
    half-installed catalogue is never indexed over.
  • The hashes show each file is the one the mirror's catalogue names. They do not show who wrote the
    catalogue or that it is true to the chain; seed from an operator you would trust to run the nest.

Seeding a clean clone of graph-gns-nest from a public R2 bucket holding production's data: 23
segments, 18.9 MB, 2.4 s, then at the chain tip after six RPC calls and identical to production on all
eight tables.

Data identity

  • Hidden files and directories no longer count toward the data identity (#1904). A stray .duckdb/
    extension cache, .cache/ or .gitignore made two copies of one nest compute different identities,
    so neither could find the other's mirror. The data identity of a nest holding hidden files moves
    once, which moves its mirror prefix; its NID, and so its mounts, stay as they are.

Engine

  • Burrmill f7c225d brings the QoS statements back to 4.5.0 times within the cost of the exact double
    sum (#1902): a group holds its first two values as doubles before it allocates buckets, and a
    dropped join filter no longer stays in the scan's predicate. indexer_day on the QoS copy, median of
    15 fresh servers: 1,461 ms on 4.8.0, 1,315 ms now, 1,275 ms on 4.5.0.

Operations

  • The release gate waits for a quiet box and refuses to measure on a loaded one (#1900, #1898): the
    4.8.0 gate ran at a load average of 26 and reported the noise as a regression.

Documentation

  • The operator guide said losing nuthatch.redb costs only the unsealed window. It costs the whole
    backfill, because every resume marker lives there; back the two up together.

Compatibility and upgrade

Replace the binary and restart. There is no storage migration or re-index. A nest that holds hidden
files publishes under a new dataset prefix after upgrading; nothing stores the data identity, so only
an existing mirror's prefix name moves.