Skip to content

v3.6.1

Latest

Choose a tag to compare

@github-actions github-actions released this 07 Sep 21:24
· 129 commits to main since this release
ed86e47

One fix, on a defect that had two production nests reporting perfect health while the watermark their
receipts pin had not moved in two days. A bug fix; the freeze stands. The implementation landed in
#1200, on issue #1199.

A sparse nest stopped sealing, and nothing said so (#1199)

Segments are cut once enough rows have finalised. On a nest whose contracts are quiet that threshold
can take months to reach, and until 3.6.1 the sealed watermark simply waited: a finalized range
carrying too few rows was held, the range only grew, and the shortcut that carries the watermark
across a stretch with no rows at all stopped applying the moment there was one. A single finalized
row could freeze sealed_through until twenty thousand had accumulated.

Measured on the Graph protocol nests behind the Lodestar dashboard, which see roughly 95
event-carrying blocks a day: the watermark had not moved in 739,192 blocks, about 51 hours, while the
cursor sat exactly at tip. It is also why a third nest on the same box, same binary, same minute of
restart, was unaffected - a range with no rows at all still advanced normally. The difference between
a nest that sealed and one that did not was one row.

Query answers stayed correct throughout, because the hot tier serves the unsealed tail. What this
broke was provenance rather than data: sealed_through is the block a tattler receipt pins, so a
receipt issued from a nest in this state promised an answer computed from history two days older than
the one the caller was actually given.

Since 3.6.1 a held range is also cut once it has spanned a bounded number of blocks, whatever its row
count, and the watermark is carried across a leading stretch that carries nothing before the bound is
measured. The bound is per chain and is about six hours of that chain's block time, from block
times measured against the shipped endpoints on 2026-09-07: mainnet 1,800 blocks, Arbitrum One 86,400,
Base and Optimism 10,800, BSC 48,000, Polygon 14,400, Gnosis 4,250, Monad 72,000, Robinhood Chain
216,000. A chain outside the registry gets 10,800, six hours at the modal two-second L2 cadence. One
number could not have served both ends: six hours on Arbitrum is eleven weeks on mainnet, and six
hours on mainnet is ninety seconds on Robinhood Chain.

Six hours is a judgement rather than a measurement - long enough not to cut near-empty segments all
day, short enough that a receipt pins a watermark from within the same working morning.

Nothing already sealed changes, and no operator action is required. A nest in the stuck state
catches up on its next few polls after the upgrade. Segment identity is unaffected: a segment is
addressed by the hash of its Parquet bytes, which are built from its rows alone, and the cut stays a
function of the data rather than of when finality happened to arrive - so two operators indexing the
same chain still produce identical segments.

A dead Optimism default, and a probe that could not have seen it (#1196)

Also in this release. optimism-rpc.publicnode.com had moved archive requests behind a token: it
served an address-filtered eth_getLogs within about 20 blocks of tip and refused anything ~200 back,
so it could not serve a backfill at all - and it was listed second, which meant round-robin handed
it real backfill traffic. It is replaced by op-pokt.nodies.app, measured against RFC-0030 §4 on
2026-09-07: archive state a million blocks back, eth_getLogs at 5,000 and at 1,000,000 behind tip,
batch-of-5, finalized. The official sequencer RPC stays first.

It was invisible for at least nineteen days because the weekly live-endpoints probe carried its own
copy of the endpoint list. That copy had gone stale in both directions: it was still probing three
endpoints removed from the registry weeks earlier, and probing none of the five that BSC, Optimism and
Gnosis ship. The probe now reads the list out of src/chains.rs, and fails if any shipped chain has
no probe at all.

/ready can tell a stalled seal from a healthy one

Every seal field on /ready described the bulk --seal-direct backfill, which on these nests was
never running - so the endpoint reported no stall because no pass had started, rather than because one
was keeping up, and lag_blocks, which describes following, read 0 throughout. A cron health check
ran against that for days and passed.

  • seal_lag_blocks - how far the sealed watermark trails what the cursor has indexed. null for a
    serve-only role, and null before anything has sealed at all, so sealed_through: 0 beside it
    reads as "nothing sealed yet" rather than as a seal that has caught up.
  • tip_seal_stalled, and a readiness verdict on it at twelve hours. A seal that has caught up to
    its cursor has nothing to do and is not stalled, the same allowance a cursor sitting at tip already
    had. Twelve hours is chain-independent because the new bound is sized in time: it is written in
    blocks, but what it buys is a duration.
  • seconds_since_seal_progress is no longer confined to the bulk backfill. It was null on
    precisely the nests that needed it.

A healthy nest should show seal_lag_blocks oscillating rather than climbing. Climbing without bound
over a day is the shape of this defect.