Skip to content

v3.0.0-alpha.1

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 28 Aug 15:30
· 36 commits to main since this release
7996c39

A nest can declare a relation in entities.toml and nuthatch maintains it as blocks arrive, instead
of recomputing it on every query. That is RFC-0041, and it is what the major bump is about.

This is an alpha, and it is an alpha for a reason. The endurance evidence is thin and still
accumulating: at the time of this cut an entity had followed a chain for twenty hours without falling
behind, and no process carrying one had been restarted after days of uptime.
docs/releases/3.0.0-alpha-stresstest.md says what is still unanswered and in what order it gets
answered; docs/bench/3.0.0-alpha-live.md carries the figures as they stand, including the two items
that are not finished and the one the run cannot test at all.

Drop-in, with one sentence of small print

Replace the binary and restart. No config migration, no data migration, no re-index. A nest that
declares no entities behaves exactly as it did on 2.7.x.

The 2.x to 3.0 jump says this is significant, not this will break you.

The small print: adding an entities.toml to a nest that already has history means its next
restart seeds the entity from the whole sealed corpus.
Measured, so it takes a figure rather than a
warning: on the Lodestar nest that is 249,979 rows across 733 sealed segments in 1.9 seconds, and
on a larger Horizon corpus 346,288 rows across 2,985 segments in 2.4 seconds. A restart gets
slower by about that much, once, per entity.

What an entity buys

The Lodestar indexer_rewards panel, which was an authored view over 733 sealed segments and is now
a maintained relation. Same nest, same data, same query, p50 over 25 runs:

p50 p99
authored view, recomputed per query 2.15 s 2.20 s
maintained relation 87.7 ms 92.1 ms

It returns the same 82 rows the view does, exactly.

And it costs what it should to keep current: one block's update against a relation of 309,548 groups
is 285 microseconds, and the same update against 61 groups is 195. Flat, which is the whole point,
and not true until late in the work.

Faster for every nest, entities or not

A /sql request was defining a view for every table in the nest's manifest whether or not the
statement named it, at roughly 62 microseconds per sealed segment. SELECT 1 cost 2.5 seconds on
a 38,428-segment nest before reading a row. Views are now defined for what a statement can reach.

The nest-wide corruption signal moved with it. A query reports degraded segments for tables it
read
; the whole-nest sweep runs separately rather than on the caller's time. If you parse
degraded_tables out of a /sql response, that field has narrowed in meaning, and it is the one
behaviour change on an existing surface here.

Four defects the measurements found

None was visible from the test suite. Each needed a real nest.

A restart silently kept only groups mod 10,000 of a relation. DBSP splits a transaction into
internal steps and each writes the output handle separately; nuthatch read it once, after. On a real
corpus 309,549 distinct keys went in and 9,549 groups came out - nothing faulted, nothing logged, and
every group it did keep held the right value.

One block's update cost 72 ms and grew with history. The published relation was copied in full
after every batch, so a block cost the size of the answer rather than the size of the change.

/explain answered differently for the same query depending on whether another request had
warmed the pooled DuckDB connection: 400 table does not exist cold, 200 valid warm.

A tip race was classified as a provider cap and killed a nest. Asking for a block the provider has
not served yet is normal at tip; the error carried the words block range, matched the cap marker
list, and the chunker narrowed a window that was never too large until it gave up. Found by an
overnight run, three and a half hours in.

For watching it

Six new Prometheus series per entity - applied_through, current, rows, faulted,
unavailable, seconds_since_progress - because a maintained relation previously had no presence on
/metrics at all, and the only way to ask whether it was keeping up was to poll /ready and parse
JSON.

[[alerts]] gains format = "discord" for sinks that want {"content": …} rather than our payload.
Default is unchanged and every existing sink is byte-identical.

Also in here

--seal-direct refuses a nest that declares an entity rather than completing with an empty relation.
The DuckDB connection cache keys on content rather than on a clock it cannot trust. A zero-width
backfill window is refused instead of spun on. /ready answers for a seal-direct that has stopped
sealing.