Repository navigation
Server
-
Tier2 skips loading dependency stores and compiling WASM modules for a segment once its
cached block index proves every module excludes all of the segment's blocks. It still writes
the same (empty) output files as before. -
Bumped
dauth: with continuous authentication, each continuousAuthenticateresponse now refreshes the stream's trusted headers, and metering reads them on every event (#980). A stream whose first authentication returned no identity is metered with it from the next continuous call on. -
Add
Tier1Config.AllowPrunedChain, forwarded to tier2. It lets
requests keep running Substreams whose modules (mappers and stores) start before the chain's first streamable block,
after that block was raised (the chain was pruned). Those modules run from the first streamable block on,
and their stores resume from the snapshots taken before the pruning, which keep their original names
(ex:0400100000-0350000000.kv). A request is refused when a store has no snapshot at the first streamable block
to resume from, and the error says which store and how to fix it.
When it is enabled, a request using a store without an initial block is refused: such a store would be rebuilt from the new
first streamable block each time it is raised, while cached outputs still reflect its older state. Every store must
set an initial block, at or above the first streamable block. -
Fix requests failing with
snapshot "…" does not existafter the chain's first streamable block was raised, for
stores without an initial block. Their snapshots written before are named after the previous first streamable block
and were counted as present, then failed to load. They are now ignored and the store is rebuilt. -
Fix dev-mode requests reading blocks below the first streamable block when that block is not a multiple of the
segment size: the linear handoff was rounded down below it. It now stays at or above the first streamable block. -
Fix requests with a cursor being refused with
invalid start block …, must be >= …when their start block is below
the first streamable block: the cursor replaces the start block, so the block it resolves to is what is checked now. -
Fix the live back filler (started after a quickload) stopping for the rest of the request when it rewound below the
first streamable block, or to a snapshot named after another initial block: its first job failed. It now starts at
or above the first streamable block and skips those snapshots. -
Fix wasm calls that fail because the host ran out of file descriptors or memory (e.g.
Too many open fileswhile
creating an instance) being reported as deterministic errors, which returnedInvalidArgumentand were cached for 1h.
They now returnInternal, and the shared cache runs the call again instead of handing the same error to every
stream on that block. A store backend failure is no longer kept in the shared cache either. -
Fix tier1 leaking a gRPC connection each time a job failed to reach a tier2. The error now also says what failed
instead of justconnection refused. -
Production-mode requests disconnected for being more than 2 segments behind the last final block no longer get
theendpoint is shutting down, please reconnecterror. They getback-processing ended too far behind the last final block of the chain, please reconnect to process the gap in parallelwhen back-processing finishes, and
stream fell too far behind the last final block of the chain, please reconnect to process the gap in parallel
while streaming final blocks. Both are stillUnavailable.
Sink
- Sinks: the
substreams stream statslog line reports backprocessing progress again until the sink is live, as a
backprocessing_progressobject with the same figures as thesubstreams runprogress view: overall percentage,
rate, ETA, running jobs, blocks to process and blocks cached, and for each stage its percentage, first block,
squashed-up-to block, jobs and segments waiting to be squashed. It replaces theprogress_block_rate,
progress_last_block,progress_running_jobsandprogress_total_processed_blocksfields, which were only logged
once the first block of data had arrived. - SQL sink (DatabaseChanges): fix undo leaving changes from orphaned blocks in the database. Changes to the same
row in different reversible blocks of one batch were merged into one operation recorded under a single block, so
undoing the later block missed them (permanently forADDfields), and pending changes survived an undo. They are
now kept as separate operations within the batch, and an undo drops the pending changes of reverted blocks and
writes the rest with the cursor. Batching is unchanged. - SQL sink (DatabaseChanges, PostgreSQL): once the sink has read or written its cursor, each flush and revert
transaction starts by locking its cursor row and checking it still holds the cursor the sink last read or wrote.
Otherwise nothing is written and the sink stops withthe stored cursor is not the one this sink last read or wrote,
to be restarted from the stored cursor. This stops a second sink of the same module writing to the same database, a
sink writing on top of a database that lost commits (restore, failover with asynchronous replication), and a flush
retry applying a batch twice after a commit whose reply was lost (ADDfields were doubled). A conflict is not
retried. Other cursor rows are not checked, and the cursor update stays the last statement of the transaction. - SQL sink (DatabaseChanges, PostgreSQL): add
--standby-lockto run standby sinks. A sink takes a session advisory
lock on the cursors table before streaming and writes through that session; the others wait, checking every
--standby-lock-poll-interval, and the first to see the lock released takes over from the stored cursor. A sink
whose locked session is gone stops instead of retrying its flush, and standbys serve the metrics endpoint while they
wait. The active sink queries its locked session every--standby-lock-poll-intervalwhile not flushing, so a proxy
oridle_session_timeoutdoesn't end it during long stretches without writes (no failover ping-pong on sparse data).
See Standby Sinks. - SQL sink (DatabaseChanges): add
--stall-timeout, which stops the sink when the stream delivers no block or undo for
that long once live (blocks held by an undo buffer count). On PostgreSQL a flush now honours its context, so the stop
cancels a database call in progress (the transaction rolls back); ClickHouse flushes still always run to the end.
The sink library's newSinker.LastBlockMessageAtreports when the stream last delivered a block or undo. - SQL sink (DatabaseChanges): an undo whose transaction fails is now rolled back.
- Sink library: a handler error is never taken for the end of the stream anymore. One wrapping
io.EOF(a lost database
connection, say) used to end the run successfully, as if the stop block was reached, so a live sink exited with
status 0. When the handler fails while the sinker is stopping, the completion handler no longer runs on top of the
block it did not finish (the failure is taken as caused by the stop and not reported). The completion handler
now gets the cursor of the last block the handler accepted rather than of the last block received, which with
--undo-buffer-sizewas ahead of the blocks still in the buffer (Sinker.LastReceivedCursorstill gives the
latter). - Sink library: add
Sinker.AwaitRun(ctx), which waits forRunto return and gives its first failure. A stop
terminates the sinker right away whileRunstill finishes its completion handler, so a caller about to exit can
wait for it there, and get a failure that the termination error, set by the stop, cannot carry. - SQL sink (DatabaseChanges): fix the final flush when the stream ends (stopped, or at the stop block) treating the
stream's latest block as final. It pruned the undo history of blocks that were still reversible, so a reorg across
a sink restart left orphaned rows, and with--undo-buffer-sizeit saved a cursor past the blocks still in the
buffer, which were then skipped. It now writes the handled blocks with the last handled block's cursor and the
stream's final block height, and writes nothing if a block failed to be handled. The CLI now waits (up to 45 s) for
that last flush before exiting: on Kubernetes, raise the pod'sterminationGracePeriodSecondsabove it (30 s by
default), otherwise a pod killed first re-processes the blocks since the last stored cursor on restart. - Add
substreams sink pubsub, which publishes any module output to a Google Cloud Pub/Sub topic as the
webhook sink's batch JSON:{"manifest": {...}, "blocks": [{"clock": {...}, "data": {...}}]}. While
Substreams is not live,--pubsub-batch-max-blocksgroups blocks into one message. A live block is
published in that same JSON with only that block.--pubsub-undopublishes reorg notifications on that
topic, and message ordering uses the module name as the ordering key. A publish that still fails after
retries stops the process with status 75 and keeps the payload for the next start.
--pubsub-on-failure=skipis the opt-in that drops a block and continues. - Publish a module that emits
sf.substreams.sink.pubsub.v1.Publishin thesubstreams-sink-pubsub
format: one Pub/Sub message perPublish.Message, with that message's bytes and attributes, attribute
Cursor, and an ordering key of the zero-padded block number and message index. A reorg on that module
is published with attributesLastValidBlock,Step=Undo, andCursor. Any other module is published
as the webhook sink's batch JSON. - Fix
substreams sink pubsubpanicking when a block has no module output, including under
--noop-mode. - Count a Pub/Sub message in the sink totals only after it is published.
CLI
substreams packwrites map entries in key order, so packing the same inputs twice gives a byte-identical
.spkg. Before, a package with a multi-entry map, such asnetworks, got a different digest from one pack to
the next.- Commands that take a package (
substreams run,gui,info, ...) accept an organization package reference
<org>/<package>@<version>, fetched from the registry's/v1/packages/<org>/<package>/<version>. Without a
version, or with@latest, they fetch the package's latest release. An existing file or directory of the
same name is still read locally; prefix a path with./to keep it local in any case. - Manifest
importsaccept<org>/<package>@<version>. An organization package import needs a specific
version:<org>/<package>and<org>/<package>@latestare refused, as<package>@latestalready is.
Dependencies
golang.org/x/netis at v0.60.0, which clears CVE-2026-78663 (CRITICAL) and CVE-2026-78669, CVE-2026-78660 and
CVE-2026-78659 (HIGH), all HTTP/2 issues reported by Docker Scout against the image.- Bumped OpenTelemetry to v1.47.0 and
github.com/streamingfast/dhttpto v0.2.0 to fix known vulnerabilities. - Bumped
github.com/yuin/goldmarkto v1.7.17 to fix an XSS (GO-2026-5320) in the CLI's markdown rendering. - Bumped
github.com/go-jose/go-jose/v4to v4.1.6 to fix CVE-2026-116587 (HIGH, inefficient algorithmic complexity). - Go 1.27 or newer is now required to build.
github.com/go-jose/go-jose/v4is at v4.1.6, which clears CVE-2026-116587 (HIGH), reported by Docker Scout against the image.