Skip to content

v1.26.0

Latest

Choose a tag to compare

@sduchesneau sduchesneau released this 30 Oct 14:48
fc6b603

Server

  • Tier2 skips loading dependency stores and compiling WASM modules for a segment once its
    cached block index proves every module excludes all of the segment's blocks. It still writes
    the same (empty) output files as before.

  • Bumped dauth: with continuous authentication, each continuous Authenticate response now refreshes the stream's trusted headers, and metering reads them on every event (#980). A stream whose first authentication returned no identity is metered with it from the next continuous call on.

  • Add Tier1Config.AllowPrunedChain, forwarded to tier2. It lets
    requests keep running Substreams whose modules (mappers and stores) start before the chain's first streamable block,
    after that block was raised (the chain was pruned). Those modules run from the first streamable block on,
    and their stores resume from the snapshots taken before the pruning, which keep their original names
    (ex: 0400100000-0350000000.kv). A request is refused when a store has no snapshot at the first streamable block
    to resume from, and the error says which store and how to fix it.
    When it is enabled, a request using a store without an initial block is refused: such a store would be rebuilt from the new
    first streamable block each time it is raised, while cached outputs still reflect its older state. Every store must
    set an initial block, at or above the first streamable block.

  • Fix requests failing with snapshot "…" does not exist after the chain's first streamable block was raised, for
    stores without an initial block. Their snapshots written before are named after the previous first streamable block
    and were counted as present, then failed to load. They are now ignored and the store is rebuilt.

  • Fix dev-mode requests reading blocks below the first streamable block when that block is not a multiple of the
    segment size: the linear handoff was rounded down below it. It now stays at or above the first streamable block.

  • Fix requests with a cursor being refused with invalid start block …, must be >= … when their start block is below
    the first streamable block: the cursor replaces the start block, so the block it resolves to is what is checked now.

  • Fix the live back filler (started after a quickload) stopping for the rest of the request when it rewound below the
    first streamable block, or to a snapshot named after another initial block: its first job failed. It now starts at
    or above the first streamable block and skips those snapshots.

  • Fix wasm calls that fail because the host ran out of file descriptors or memory (e.g. Too many open files while
    creating an instance) being reported as deterministic errors, which returned InvalidArgument and were cached for 1h.
    They now return Internal, and the shared cache runs the call again instead of handing the same error to every
    stream on that block. A store backend failure is no longer kept in the shared cache either.

  • Fix tier1 leaking a gRPC connection each time a job failed to reach a tier2. The error now also says what failed
    instead of just connection refused.

  • Production-mode requests disconnected for being more than 2 segments behind the last final block no longer get
    the endpoint is shutting down, please reconnect error. They get back-processing ended too far behind the last final block of the chain, please reconnect to process the gap in parallel when back-processing finishes, and
    stream fell too far behind the last final block of the chain, please reconnect to process the gap in parallel
    while streaming final blocks. Both are still Unavailable.

Sink

  • Sinks: the substreams stream stats log line reports backprocessing progress again until the sink is live, as a
    backprocessing_progress object with the same figures as the substreams run progress view: overall percentage,
    rate, ETA, running jobs, blocks to process and blocks cached, and for each stage its percentage, first block,
    squashed-up-to block, jobs and segments waiting to be squashed. It replaces the progress_block_rate,
    progress_last_block, progress_running_jobs and progress_total_processed_blocks fields, which were only logged
    once the first block of data had arrived.
  • SQL sink (DatabaseChanges): fix undo leaving changes from orphaned blocks in the database. Changes to the same
    row in different reversible blocks of one batch were merged into one operation recorded under a single block, so
    undoing the later block missed them (permanently for ADD fields), and pending changes survived an undo. They are
    now kept as separate operations within the batch, and an undo drops the pending changes of reverted blocks and
    writes the rest with the cursor. Batching is unchanged.
  • SQL sink (DatabaseChanges, PostgreSQL): once the sink has read or written its cursor, each flush and revert
    transaction starts by locking its cursor row and checking it still holds the cursor the sink last read or wrote.
    Otherwise nothing is written and the sink stops with the stored cursor is not the one this sink last read or wrote,
    to be restarted from the stored cursor. This stops a second sink of the same module writing to the same database, a
    sink writing on top of a database that lost commits (restore, failover with asynchronous replication), and a flush
    retry applying a batch twice after a commit whose reply was lost (ADD fields were doubled). A conflict is not
    retried. Other cursor rows are not checked, and the cursor update stays the last statement of the transaction.
  • SQL sink (DatabaseChanges, PostgreSQL): add --standby-lock to run standby sinks. A sink takes a session advisory
    lock on the cursors table before streaming and writes through that session; the others wait, checking every
    --standby-lock-poll-interval, and the first to see the lock released takes over from the stored cursor. A sink
    whose locked session is gone stops instead of retrying its flush, and standbys serve the metrics endpoint while they
    wait. The active sink queries its locked session every --standby-lock-poll-interval while not flushing, so a proxy
    or idle_session_timeout doesn't end it during long stretches without writes (no failover ping-pong on sparse data).
    See Standby Sinks.
  • SQL sink (DatabaseChanges): add --stall-timeout, which stops the sink when the stream delivers no block or undo for
    that long once live (blocks held by an undo buffer count). On PostgreSQL a flush now honours its context, so the stop
    cancels a database call in progress (the transaction rolls back); ClickHouse flushes still always run to the end.
    The sink library's new Sinker.LastBlockMessageAt reports when the stream last delivered a block or undo.
  • SQL sink (DatabaseChanges): an undo whose transaction fails is now rolled back.
  • Sink library: a handler error is never taken for the end of the stream anymore. One wrapping io.EOF (a lost database
    connection, say) used to end the run successfully, as if the stop block was reached, so a live sink exited with
    status 0. When the handler fails while the sinker is stopping, the completion handler no longer runs on top of the
    block it did not finish (the failure is taken as caused by the stop and not reported). The completion handler
    now gets the cursor of the last block the handler accepted rather than of the last block received, which with
    --undo-buffer-size was ahead of the blocks still in the buffer (Sinker.LastReceivedCursor still gives the
    latter).
  • Sink library: add Sinker.AwaitRun(ctx), which waits for Run to return and gives its first failure. A stop
    terminates the sinker right away while Run still finishes its completion handler, so a caller about to exit can
    wait for it there, and get a failure that the termination error, set by the stop, cannot carry.
  • SQL sink (DatabaseChanges): fix the final flush when the stream ends (stopped, or at the stop block) treating the
    stream's latest block as final. It pruned the undo history of blocks that were still reversible, so a reorg across
    a sink restart left orphaned rows, and with --undo-buffer-size it saved a cursor past the blocks still in the
    buffer, which were then skipped. It now writes the handled blocks with the last handled block's cursor and the
    stream's final block height, and writes nothing if a block failed to be handled. The CLI now waits (up to 45 s) for
    that last flush before exiting: on Kubernetes, raise the pod's terminationGracePeriodSeconds above it (30 s by
    default), otherwise a pod killed first re-processes the blocks since the last stored cursor on restart.
  • Add substreams sink pubsub, which publishes any module output to a Google Cloud Pub/Sub topic as the
    webhook sink's batch JSON: {"manifest": {...}, "blocks": [{"clock": {...}, "data": {...}}]}. While
    Substreams is not live, --pubsub-batch-max-blocks groups blocks into one message. A live block is
    published in that same JSON with only that block. --pubsub-undo publishes reorg notifications on that
    topic, and message ordering uses the module name as the ordering key. A publish that still fails after
    retries stops the process with status 75 and keeps the payload for the next start.
    --pubsub-on-failure=skip is the opt-in that drops a block and continues.
  • Publish a module that emits sf.substreams.sink.pubsub.v1.Publish in the substreams-sink-pubsub
    format: one Pub/Sub message per Publish.Message, with that message's bytes and attributes, attribute
    Cursor, and an ordering key of the zero-padded block number and message index. A reorg on that module
    is published with attributes LastValidBlock, Step=Undo, and Cursor. Any other module is published
    as the webhook sink's batch JSON.
  • Fix substreams sink pubsub panicking when a block has no module output, including under
    --noop-mode.
  • Count a Pub/Sub message in the sink totals only after it is published.

CLI

  • substreams pack writes map entries in key order, so packing the same inputs twice gives a byte-identical
    .spkg. Before, a package with a multi-entry map, such as networks, got a different digest from one pack to
    the next.
  • Commands that take a package (substreams run, gui, info, ...) accept an organization package reference
    <org>/<package>@<version>, fetched from the registry's /v1/packages/<org>/<package>/<version>. Without a
    version, or with @latest, they fetch the package's latest release. An existing file or directory of the
    same name is still read locally; prefix a path with ./ to keep it local in any case.
  • Manifest imports accept <org>/<package>@<version>. An organization package import needs a specific
    version: <org>/<package> and <org>/<package>@latest are refused, as <package>@latest already is.

Dependencies

  • golang.org/x/net is at v0.60.0, which clears CVE-2026-78663 (CRITICAL) and CVE-2026-78669, CVE-2026-78660 and
    CVE-2026-78659 (HIGH), all HTTP/2 issues reported by Docker Scout against the image.
  • Bumped OpenTelemetry to v1.47.0 and github.com/streamingfast/dhttp to v0.2.0 to fix known vulnerabilities.
  • Bumped github.com/yuin/goldmark to v1.7.17 to fix an XSS (GO-2026-5320) in the CLI's markdown rendering.
  • Bumped github.com/go-jose/go-jose/v4 to v4.1.6 to fix CVE-2026-116587 (HIGH, inefficient algorithmic complexity).
  • Go 1.27 or newer is now required to build.
  • github.com/go-jose/go-jose/v4 is at v4.1.6, which clears CVE-2026-116587 (HIGH), reported by Docker Scout against the image.