Skip to content

Releases: subsquid-labs/portal-ponder

@subsquid/ponder 0.17.6-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 12 Aug 11:04
b614e6e

@subsquid/ponder@0.17.6-sqd.1 — the Portal fork of ponder, grafted onto ponder@0.17.6.

What this is

The Portal layer (backfill from SQD Portal) grafted onto upstream ponder@0.17.6, published as a drop-in ponder bin. Routine upstream-parity: tracks a new upstream patch line at parity, no fork behavior or public-surface change.

Upstream delta 0.17.4 → 0.17.6

  • The graft seam (HistoricalSync.syncBlockRangeData / syncBlockData, interval-scoped) is unchanged.
  • The only wiring re-derivation is rpc/actions.ts: upstream moved address-chunking into eth_getLogs, so the Portal helper block re-anchors after the now-chunked eth_getLogs (11/12 hunks verbatim, 1 re-anchored). eth_getFilteredBlockLogs depends only on eth_getLogs's unchanged in/out contract, so the chunking is a transparent internal detail.
  • Upstream's null-topic / zeroHash / address-chunking changes are confined to the RPC-mode plane, off the Portal /stream path.

Gate

Both-version scripts/sync-upstream.sh --test green (0.17.6 graft + supported lines); biome clean; no exported-type / config / signature drift vs 0.17.4-sqd.1. CI fork-seam(0.17.6) green on the tracking PR (#216).

Published with --tag latest (numeric-max) — the default npm install @subsquid/ponder. 0.17.4-sqd.1 remains installable by exact version.

@subsquid/ponder 0.17.4-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 31 Jul 23:18
68728ae

@subsquid/ponder@0.17.4-sqd.1

ponder@0.17.4 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is an upstream-parity release: 0.17.3 → 0.17.4 is four upstream indexing-performance patch PRs, every one off-seam and grep-proven inert for the fork, so they are passively inherited with no Portal-layer behavior change and no wiring change. It installs by default (--tag latest), superseding 0.17.3-sqd.1 (which now has no dist-tag — installable by exact version, npm i @subsquid/ponder@0.17.3-sqd.1).

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.17.4-sqd.1.

What changed upstream — ponder@0.17.3 → 0.17.4

Four upstream patch-level indexing-performance PRs (all @kyscott18), passively inherited:

  • ponder-sh/ponder#2353 (a0a91c3) — batch internal metrics updates.
  • #2356 (13d34ee) — schema-aware copying.
  • #2355 (d34cf7e) — normalize only changed row fields on updates.
  • #2352 (0183ecc) — reduce copy-on-write row memory.

The entire packages/core/src delta is nine files, all in ponder's indexing plane: indexing-store/{cache,index,utils}.ts, internal/metrics.ts, utils/copy.ts, indexing/index.ts (two call-sites swapped to a batched metrics helper), and a one-line common.metrics.flushIndexingMetrics() added at the tail of each runtime/{isolated,multichain,omnichain}.ts indexing loop.

Three of the nine changed files (runtime/isolated|multichain|omnichain.ts) are among the 12 wiring files, but each has a single wiring hunk at its run* entrypoint (L152 / L174 / L177) — above and non-overlapping the perf additions (L401 / L454 / L473). So portal/wiring/0.17.4.patch — a verbatim, byte-identical copy of portal/wiring/0.17.3.patch (same 12 files; the verbatim lineage now runs 0.17.0 → 0.17.1 → 0.17.2 → 0.17.3 → 0.17.4) — applies with zero rejects. The six load-bearing wiring files (build/config, config/index, internal/types, rpc/actions, rpc/index, runtime/historical — the historical seam) are byte-identical 0.17.3 → 0.17.4.

Grep-proven inert for the fork. The Portal layer (portal/*.ts) references none of the three changed metrics symbols (ponder_indexing_completed_events / incrementIndexingCompletedEvents / flushIndexingMetrics), and the upstream perf changes are confined to the indexing-store / metrics / copy internals that the Portal fetch/assembly layer never touches. Dependencies are unchanged except the @ponder/client self-version bump. The graft seam (HistoricalSync.syncBlockRangeData / syncBlockData, interval-scoped) is unchanged; 0.17.4 touches only the indexing plane.

Compatibility

  • Built + tested against ponder@0.17.4: scripts/sync-upstream.sh 0.17.4 --test — patch applies clean (zero rejects), tsc builds, 416 tests / 24 files green; both-version confirmed (the identical suite is also green on ponder@0.16.6, the pnpm-9 path). CI runs the fork-seam(0.17.4) job (in compat.tested).
  • No fresh RPC byte-diff is claimed on 0.17.4 — compatibility rests on seam identity + verbatim patch + the full suite, exactly as for 0.17.3-sqd.1. The §3 / §5 byte-diff and A/B soak evidence remains on the 0.16.6 graft and transfers by seam identity.
  • Seam verified identical across 0.15.170.17.4 (compat.tested in versions.json).

@subsquid/ponder 0.17.3-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 30 Jul 11:46
440f950

@subsquid/ponder@0.17.3-sqd.1

ponder@0.17.3 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is an upstream-parity release: 0.17.2 → 0.17.3 is a single upstream memory-retention bugfix, off-seam and grep-proven inert for the fork, so it is passively inherited with no Portal-layer change and no wiring change. It installs by default (--tag latest), superseding 0.17.2-sqd.1 (which now has no dist-tag — installable by exact version, npm i @subsquid/ponder@0.17.2-sqd.1).

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.17.3-sqd.1.

What changed upstream — ponder@0.17.2 → 0.17.3

One upstream patch-level bugfix, passively inherited:

  • ponder-sh/ponder#2350 (@kyscott18, commit 80dd936) — fixes realtime sync retaining unfiltered block transactions in memory until finalization (upstream reports >10% live-indexing memory improvement). The entire source delta is one line in src/sync-realtime/index.ts's reconcileBlock async generator (~L1140): the yielded block's transactions array is repointed from the unfiltered set to the already-filtered top-level transactions array, so the unfiltered array is released for garbage collection.

src/sync-realtime/index.ts is one of the 12 wiring files, but the #2350 edit (~L1140) does not overlap the wiring patch's two hunks (the import block and the #23 filtered-logs shouldRequestLogs fallback, ~L308–347), so portal/wiring/0.17.3.patch — a verbatim copy of portal/wiring/0.17.2.patch (sha256-equal, same 12 files; the verbatim lineage now runs 0.17.0 → 0.17.1 → 0.17.2 → 0.17.3) — applies with zero rejects.

Grep-proven inert for the fork. The swap changes what a yielded realtime event's block.transactions contains (unfiltered → filtered), but nothing in the fork reads it: .block.transactions is referenced zero times in the Portal layer (portal/*.ts) and in the wiring patch. Realtime transactions are persisted from the top-level transactions field (filtered, identical logic in 0.17.2 and 0.17.3), and the #27 access_list path maps from the top-level Portal transactions, never from block.transactions. So the change persists and emits no different data on either the RPC or Portal /stream path — it only frees memory. The graft seam (HistoricalSync.syncBlockRangeData / syncBlockData, interval-scoped) is unchanged; 0.17.3 touches only src/sync-realtime/index.ts.

Compatibility

  • Built + tested against ponder@0.17.3: scripts/sync-upstream.sh 0.17.3 --test — patch applies clean (all 12 files "Applied … cleanly"), tsc builds, 408 tests / 23 files green; both-version confirmed (the identical suite is also green on ponder@0.17.2). CI runs the fork-seam(0.17.3) job (in compat.tested).
  • No fresh RPC byte-diff is claimed on 0.17.3 — compatibility rests on seam identity + verbatim patch + the full suite, exactly as for 0.17.2-sqd.1. The §3 / §5 byte-diff and A/B soak evidence remains on the 0.16.6 graft and transfers by seam identity.
  • Seam verified identical across 0.15.170.17.3 (compat.tested in versions.json).

@subsquid/ponder 0.17.2-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 27 Jul 18:15
7c8d9da

@subsquid/ponder@0.17.2-sqd.1

ponder@0.17.2 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is an upstream-parity release: 0.17.1 → 0.17.2 is two upstream patch-level bugfixes, both off-seam or non-overlapping with the graft, so the fork passively inherits them with no Portal-layer change and no wiring change. It installs by default (--tag latest), superseding 0.17.1-sqd.2 (which now has no dist-tag — installable by exact version, npm i @subsquid/ponder@0.17.1-sqd.2).

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.17.2-sqd.1.

What changed upstream — ponder@0.17.1 → 0.17.2

Two upstream patch-level bugfixes, both passively inherited:

  • ponder-sh/ponder#2341 (@hellowodl) — fixes rows returned by context.db.find() / insert() / update() being mutated by a subsequent write to the same row (adds a defensive copy at the get-sites). Implemented purely in src/indexing-store/index.ts, which is off the graft surface (the indexing store, like the 0.16.9 bytea-COPY fix) — the seam is untouched.
  • ponder-sh/ponder#2347 (@kyscott18) — fixes realtime-discovered factory child addresses being omitted from ponder_sync.factory_addresses when a factory is shared across multiple event handlers. Implemented in src/sync-realtime/index.ts — one of the 12 wiring files, but in regions that do not overlap the wiring patch's two hunks (the import block and the #23 filtered-logs shouldRequestLogs fallback), so portal/wiring/0.17.2.patch applies with zero rejects and the fork passively inherits the upstream completeness fix. This is a realtime runtime-state delta (a strictly larger persisted factory_addresses set for shared factories), inherited verbatim from upstream — not a behavior change introduced by the Portal layer.

portal/wiring/0.17.2.patch is a verbatim copy of portal/wiring/0.17.1.patch (sha256-equal, same 12 files — itself a verbatim copy back to 0.17.0). The graft seam (HistoricalSync.syncBlockRangeData / syncBlockData, interval-scoped) is unchanged; 0.17.2 touches only src/indexing-store/index.ts and src/sync-realtime/index.ts, neither of which is the historical seam.

Compatibility

  • Built + tested against ponder@0.17.2: scripts/sync-upstream.sh 0.17.2 --test — patch applies clean (all 12 files "Applied … cleanly"), tsc builds, 396 tests / 22 files green; both-version confirmed (the identical suite is also green on ponder@0.16.6).
  • No fresh RPC byte-diff is claimed on 0.17.2 — compatibility rests on seam identity + verbatim patch + the full suite, exactly as for 0.17.1-sqd.2. The §3 / §5 byte-diff and A/B soak evidence remains on the 0.16.6 graft and transfers by seam identity.
  • Seam verified identical across 0.15.170.17.2 (compat.tested in versions.json).

@subsquid/ponder 0.17.1-sqd.2

Choose a tag to compare

@dzhelezov dzhelezov released this 21 Jul 08:53
191e191

@subsquid/ponder@0.17.1-sqd.2

ponder@0.17.1 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is a fork-side fix release on the same ponder@0.17.1 — the wiring patch and upstream version are unchanged from 0.17.1-sqd.1; only the Portal layer gained the fix below. It installs by default (--tag latest), superseding 0.17.1-sqd.1 (which now has no dist-tag — installable by exact version, npm i @subsquid/ponder@0.17.1-sqd.1).

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.17.1-sqd.2.

What changed since 0.17.1-sqd.1 — factory query-sharding (#194, PR #195)

A factory whose discovered child-address set is large (~5.8k+) overflowed Portal's 256 KiB raw-query cap (MAX_RAW_QUERY_SIZE) in a single query body → Portal 400 → the fork's fail-loud turned that into a permanent hard stop for the chain. Batching + merge don't reduce the total body size, so above the wall the chain could not progress.

The fix — byte-budgeted log-query sharding. logQueryShards() partitions the merged log-request address union into multiple query bodies, each kept below the cap, streamed sequentially and unioned:

  • Completeness preserved by construction (the Portal layer's INV-1 / INV-3 / INV-11 filter-completeness invariants): the shard loop lives inside the chunk's runStreams, so the chunk is cached/marked-done only after every shard has drained — no partial-commit window — and any shard error rejects the whole chunk and retries the full plan.
  • Byte-identical no-op below the wall: when the whole body fits one shard the result is exactly one shard equal to the un-sharded query, so the entire existing validated corpus (≤ 872 children) is provably unaffected.
  • A single merged element that alone exceeds the budget is now an attributable, up-front error rather than an opaque 400 at stream time.

Mutation-verified: a mutation that drops all-but-the-first shard (the silent-loss failure mode) turns the cross-shard completeness tests red.

Scope: historical-plane log/factory-child sharding. Realtime /stream sharding, factory→tx-filter sharding, and shard concurrency are tracked follow-ups (#196) — each retains a safe fail-loud (or the RPC-realtime fallback) today.

Compatibility

  • Built + tested against ponder@0.17.1: scripts/sync-upstream.sh 0.17.1 --test — patch applies clean, tsc builds, 379 tests / 20 files green (adds portal/portal-shard.test.ts).
  • Wiring patch and graft seam (HistoricalSync.syncBlockRangeData / syncBlockData) unchanged from 0.17.1-sqd.1; the fix is off-seam Portal-layer code (portal/portal-filters.ts, portal/portal.ts).
  • Seam verified identical across 0.15.170.17.1 (compat.tested in versions.json).

Previous revision — 0.17.1-sqd.1 (upstream-parity bump)

No Portal-layer change0.17.1-sqd.1 was a pure upstream-parity bump. 0.17.0 → 0.17.1 is a single upstream performance PR (ponder-sh/ponder#2338, @kyscott18: "Improved sync performance by removing redundant block range constraints from event queries"), which narrows Ponder's internal trace/log read-back query (sync-store/index.ts, getEventData) to the block ranges of the filters active in each interval. That change sits downstream of the Portal fetch/insert and off the graft seam, so the rows the Portal writes are unchanged. portal/wiring/0.17.1.patch is a verbatim copy of portal/wiring/0.17.0.patch (sha256-equal, same 12 files).

@subsquid/ponder 0.17.0-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 20 Jul 12:49
dba6bbb

@subsquid/ponder@0.17.0-sqd.1

ponder@0.17.0 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is the
first cut on ponder@0.17.0 and the newest ponder version in the fork, so it installs by
default. It carries the same Portal layer as 0.16.10-sqd.1 — the full invariant-first
re-architecture and correctness wave — grafted onto ponder@0.17.0, plus the fork-side fixes
below that landed since the last release note (0.16.8-sqd.1). It supersedes 0.16.9-sqd.1
as latest (that cut had no release note; it tracked ponder@0.16.9's upstream bytea-COPY
fix — see Compatibility).

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.17.0-sqd.1.

Highlights (new since 0.16.8-sqd.1)

  • Realtime finalize is write-side idempotent on resume — no duplicated factory-child rows
    (INV-17, #175).
    A resumed single writer that re-finalizes the same factory-child creation
    block used to insert the ponder_sync.factory_addresses row twice (once by the pre-kill
    finalize transaction, once by the resume finalize — two sequential finalize transactions).
    The read→dedupe guard (read the store's persisted children, keep a child only if it is absent
    or re-discovered at a strictly lower block — LEAST semantics) previously ran only at the
    historical call site; it now also runs on the realtime finalize path, so both sync modes are
    byte-identically idempotent on resume. App data was already unaffected (the only consumer
    min-merges to a set) — this closes the store-identity/table-growth defect under repeated
    realtime crash/resume.
  • inserted.logs reports the store-inserted count, not the raw-streamed count (#143, #177).
    The inserted.logs metric (and the completion line's logs=) now counts the assembled logs
    actually written to the store, instead of the raw pre-re-match streamed count. A window whose
    over-returned logs are all re-match-dropped now reports inserted.logs=0, consistent with
    inserted.blocks=0. This is a metric-accuracy fix — no change to the data that is stored.

Portal layer

  • Write-side idempotence (portal-child-dedupe.ts, runtime/realtime.ts wiring hook): the
    shared read→dedupe core (dedupeChildAddressesAgainstStore) now runs at BOTH factory-child
    insert call sites — the historical path (portal.ts persistPendingChildren) and the
    realtime finalize path (runtime/realtime.ts handleRealtimeSyncEvent, via
    dedupeFinalizeChildAddresses), each inside its own store transaction (INV-17, #175).
  • Metrics (portal-metrics.ts, portal.ts): inserted.logs is mapped to a new
    insertedLogs counter taken from assembled.logs.length at the insertLogs site; the raw
    logs count is retained for the progress fingerprint and mb_streamed keeps raw-I/O
    semantics (#143, #177).
  • Wiring patch: portal/wiring/0.17.0.patch. It is re-derived, not verbatim: 0.16.10 → 0.17.0 is a tree-wide mechanical reformat (import-list reorder, static readonly
    static override readonly, @ts-ignore@ts-expect-error) plus dep bumps (@ponder/utils
    0.2.18 → 0.3.0, pg ^8.11 → ^8.16). Applying the 0.16.10 patch left 29 of 34 hunks landing
    by offset and 5 rejected — all 5 pure import-block additions where the reformat reordered the
    surrounding import lists; each rejected hunk's added lines were re-placed byte-for-byte into
    the reformatted import block (comm-verified: the set of added wiring lines is identical to
    0.16.10.patch, zero delta). The graft seam
    (HistoricalSync.syncBlockRangeData/syncBlockData, interval-scoped) survives untouched.

Fixes

Everything in 0.16.8-sqd.1 (the full phantom-coverage / stream-realtime+reorg / factory-and-
account-source / bounded-warmup / HTTP-client correctness wave, plus the accessList
honest-NULL fix and the default-level Portal logging), plus the two items above: INV-17
realtime-finalize write-side idempotence (#175) and the inserted.logs store-inserted metric
(#143, #177). The fork also passively inherits two off-seam upstream fixes over this range:
ponder@0.16.9's bytea-COPY encoding fix (ponder-sh/ponder#2323, in indexing-store/cache.ts)
and ponder@0.16.10's commit-path cache-invalidation resilience (runtime/isolated.ts /
multichain.ts / omnichain.ts) — both off the graft surface, inherited unchanged.

Compatibility

  • Built + tested against ponder@0.17.0. The full Portal property/regression suite is green:
    370/370 across 19 files on ponder@0.17.0 (scripts/sync-upstream.sh 0.17.0 --test) —
    and identically on every tracked version (0.15.17, 0.16.6, 0.16.7, 0.16.8, 0.16.9,
    0.16.10, 0.17.0).
  • ponder@0.17.0 bumps its build toolchain (engines.pnpm >=11, packageManager pnpm@11.0.0)
    and its deps (@ponder/utils 0.3.0, pg ^8.16); the fork inherits those upstream deps with no
    fork-side dep change. The sync-upstream.sh harness reads each clone's own packageManager and
    builds with the pnpm it declares (falling back to the historical pnpm@9.10.0 pin when the field
    is absent), so 0.16.10 still builds with pnpm 9.10.0 and 0.17.0 with pnpm 11 — both green at
    370 tests.
  • No fresh RPC byte-diff on 0.17.0; compat rests on seam-identity (the wiring patch re-derives
    but re-places every added wiring line byte-for-byte, so the graft is unchanged) plus the full
    suite, as for 0.16.10 / 0.16.9 / 0.15.17.
  • Seam verified against: 0.17.0, 0.16.10, 0.16.9, 0.16.8, 0.16.7, 0.16.6, 0.15.17
    (compat.tested). The seam is stable and identical in shape across 0.15.17 – 0.17.0.

@subsquid/ponder 0.16.10-sqd.2

Choose a tag to compare

@dzhelezov dzhelezov released this 20 Jul 12:49
dba6bbb

@subsquid/ponder@0.16.10-sqd.2

Supported-lines backport wave (2026-07-28). -sqd.2 re-cuts ponder@0.16.10 with the current Portal layer — the same portal/ tree shipped on latest (0.17.2-sqd.1). It keeps 0.16.10 a first-class supported line so a user pinned to ponder-0.16.10 runs the same fixes as everyone else.

Published under dist-tag ponder-0.16.10latest is not clobbered (it stays 0.17.2-sqd.1, the newest tracked ponder version). Install: npm i @subsquid/ponder@0.16.10-sqd.2 (or npm i @subsquid/ponder@ponder-0.16.10).

New since 0.16.10-sqd.1

  • Factory log-query byte-budget sharding (#194, #195). A factory whose discovered child-address IN-list union overflows the Portal 256 KiB query cap now splits the over-cap child set into byte-sized shards, streamed sequentially and unioned — completeness-preserving by construction, a byte-identical no-op below the wall.
  • Account-tx (from/to) byte-budget sharding (#196, #199). The same over-cap sharding applied to account-source transaction (from/to) queries.
  • INV-26 store-state partition-invariance. The assembled store state is codified as invariant to how the Portal range- and shard-partitions a request (cross-shard duplicate logs collapse via logs_pkey + onConflictDoNothing); adversarial-partition fuzz coverage added. Invariant catalog is now INV-1…INV-26.

Everything from 0.16.10-sqd.1 is carried forward (INV-17 realtime-finalize write-side idempotence #175, the inserted.logs store-inserted metric #143, the accessList honest-NULL fix, the full invariant-first correctness wave).

Compatibility

  • Built + tested green on ponder@0.16.10: 400/400 across 23 files (CI=true scripts/sync-upstream.sh 0.16.10 --test, pnpm@9.10.0) — verified both by a pre-publish gate and by the release workflow's own in-CI build+test.
  • The graft seam (HistoricalSync.syncBlockRangeData/syncBlockData) is untouched; every added fix lives in the Portal fetch/assembly layer, off the seam. wiring/0.16.10.patch is unchanged (a verbatim copy of wiring/0.16.9.patch). No fresh RPC byte-diff; compat rests on seam-identity + the full suite.

@subsquid/ponder 0.16.8-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 10 Jul 15:11
fcca962

@subsquid/ponder@0.16.8-sqd.1

ponder@0.16.8 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is the
first cut on ponder@0.16.8 and the newest ponder version in the fork, so it installs by
default. It carries the same Portal layer as 0.16.7-sqd.1 — the full invariant-first
re-architecture and correctness wave — grafted onto ponder@0.16.8, plus the fixes below that
landed since the 0.16.7-sqd.1 cut.

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.16.8-sqd.1.

Highlights (new since 0.16.7-sqd.1)

  • accessList correctness — store NULL, never a fabricated []. Some Portal datasets
    (e.g. arbitrum, avalanche) don't carry the access_list column. The transform previously
    filled an empty [] for those rows; it now stores SQL NULL, byte-matching what an RPC node
    reports for a transaction whose access list is genuinely absent — and the field is gated to the
    exact EIP typed-transaction set {1,2,3,4} rather than "type ≥ 1", so a legacy tx is never
    handed a spurious access list (#110, #111).
  • Positive Portal confirmation at default log level (closes #115). A backfill that routes
    through the Portal now emits an INFO banner — Portal backfill active for <chain>: <url> — a
    periodic progress ticker (blocks scanned / discovery chunks), and a completion line, and the
    upstream "fetching backfill data" line is reworded to name SQD Portal on the Portal path. You
    can now confirm from default logs that the Portal — not an RPC — served the history (#119).
  • Completion line reports Portal data-plane provenance (#131). The per-chain completion line
    now carries the data-plane totals — logs/txs/receipts counts and an rpc_fallback count —
    and closes with served entirely by the SQD Portal (0 JSON-RPC for history) exactly when
    rpc_fallback is 0 (otherwise N block range(s) fell back to JSON-RPC). The claim is scoped to
    the historical sync data plane; user-land readContract still uses your configured RPC.
  • Actionable persistent-throttle error instead of an unhandledRejection crash (for #116).
    A full backfill throttled (HTTP 429) by the free public Portal past its retry budget now
    surfaces a clear, actionable error and holds a prefetch invariant lock, rather than exhausting
    retries and crashing via an unhandled promise rejection (#118).

Portal layer

  • Transform (portal-transform.ts): the accessList honest-NULL fix (#110/#111) — see the
    Highlights above.
  • Observability (portal.ts, portal-metrics.ts): the default-level Portal banner, progress
    ticker, and completion line (#119); the completion line reports data-plane provenance —
    logs/txs/receipts counts and an rpc_fallback count, with a 0 JSON-RPC for history
    clause gated on rpcFallback === 0 (#131).
  • HTTP client (portal-client.ts): persistent-throttle actionable error + prefetch lock (#118).
  • Wiring patch: portal/wiring/0.16.8.patch. Byte-identical to the 0.16.6/0.16.7
    patches (sha256 ddd12dd5…, same 10/10 files); it applies cleanly (zero rejects), builds, and
    passes the full Portal suite on ponder@0.16.8. 0.16.7 → 0.16.8 is a realtime-only upstream
    change — it adds isAsyncExecutionChain (chainId 143/10143) in sync-realtime/index.ts +
    utils/finality.ts so async-execution chains can surface blocks before logsBloom is
    execution-ready. Neither file is on the graft surface (sync-historical, runtime/historical,
    sync-store, rpc, sync), so the seam is untouched and the 81 Portal realtime tests pass
    unchanged.

Fixes

Everything in 0.16.7-sqd.1 (the full phantom-coverage / stream-realtime+reorg / factory-and-
account-source / bounded-warmup / HTTP-client correctness wave), plus the three items above:
accessList honest-NULL (#110, #111), default-level Portal logging (#119, closes #115), the
persistent-throttle actionable error + prefetch lock (#118, for #116), and the completion line's
Portal data-plane provenance (#131).

Compatibility

  • Built + tested against ponder@0.16.8. The full Portal property/regression suite is green:
    312/312 across ponder@0.15.17, ponder@0.16.6, ponder@0.16.7, and ponder@0.16.8
    (scripts/sync-upstream.sh <ver> --test) — grown from 302 at the 0.16.7-sqd.1 cut as the
    correctness wave added coverage (incl. the #131 completion-line provenance tests).
  • No fresh RPC byte-diff on 0.16.8; compat rests on seam-identity (the wiring patch is
    byte-identical to 0.16.6/0.16.7) plus the full suite, as for 0.16.7/0.15.17.
  • Seam verified against: 0.16.8, 0.16.7, 0.16.6, 0.15.17 (compat.tested). The seam is
    stable and identical in shape across 0.15.17 – 0.16.8.

@subsquid/ponder 0.16.7-sqd.1

Choose a tag to compare

@dzhelezov dzhelezov released this 08 Jul 14:48
4453802

@subsquid/ponder@0.16.7-sqd.1

ponder@0.16.7 + the SQD Portal backfill layer, drop-in (bin stays ponder). This is the
first cut on ponder@0.16.7 and the newest ponder version in the fork, so it installs by
default. It carries the same Portal layer as 0.16.6-sqd.2 — the full invariant-first
re-architecture and correctness wave — grafted onto ponder@0.16.7.

Install: npm i @subsquid/ponder, or pin exact: npm i @subsquid/ponder@0.16.7-sqd.1.

Highlights

  • Silent-gap / phantom-coverage elimination. The known paths that could mark a block range
    "synced" without actually delivering its data now fail loud (or retry) instead of recording
    phantom coverage. This covers the
    unbounded-source backfill floor (a source with no fromBlock starts at genesis, not at the
    min of the defined floors — INV-16), head-truncated frontier chunks (extended before being
    served, never served stale — INV-13), a bounded backfill whose toBlock runs past the
    finalized head (head-clamped, not phantom-covered — INV-9/INV-13), and the mid-range
    /finalized-stream 204: a load-balanced replica serving only part of a requested range and
    204-ing the tail is now a transient incomplete-range that retries on a fresher replica or
    fails loud, never recorded as clean completion (issue #47).
  • Bounded time-to-first-durable-commit. The finalized-range fetch starts from a small warmup
    span and grows geometrically, and on a restart resumes from the persisted watermark, so a dense
    or wide-range source lands its first durable commit promptly instead of blocking on a full-range
    scan — regardless of range size; discovery follows the same geometric slow-start on the
    watermark-following path (#50).
  • Stream-realtime correctness + reorg handling. The unfinalized window is kept
    strictly-increasing and parent-hash-linked; an unknown-parent gap is fatal (never a silent
    fork switch), finality is verified by hash at the exact probe height, and a 1-block tip
    orphan heals via the Portal's HTTP 409 fork-negotiation protocol instead of fataling
    (issue #33). Same-block factory-child log redelivery is reconciled without dropping or
    duplicating blocks (INV-10).
  • Factory / account-source coverage. Discovered factory children are persisted to the sync
    store in the same transaction that marks the interval synced — including on the RPC
    finality-delegation path — so a restart never silently drops a factory's events
    (INV-15), with write-side idempotence so a re-flush never durably duplicates rows (INV-17).
    Every transaction matched by a tx filter gets a receipt inserted (account sources).
  • Invariant-first architecture. portal/ is a pure functional core behind an imperative
    shell; 18 named invariants (INV-1…INV-18, portal/INVARIANTS.md) are asserted at runtime and
    proven by property-based + regression tests. A violation is a loud InvariantViolation, not
    silent corruption.

Portal layer

  • Backfill / transform (portal.ts, portal-transform.ts, portal-client.ts,
    portal-chunks.ts, portal-discovery.ts, portal-filters.ts, portal-assemble.ts): the
    chunked finalized-range fetch, discovery-before-data ordering (INV-3), interval-exact
    assembly (INV-2), full-tree trace ranking (INV-5), request merging (INV-11), AIMD concurrency
    control + bounded buffered rows (INV-7/INV-8), and byte-identical RPC-shape transforms
    (INV-6).
  • Realtime wiring (portal-realtime.ts, portal-realtime-wire.ts): finalized-stream resume
    with parentBlockHash, 409 fork negotiation, reorg reconciliation, and the
    historical→realtime cutover guard (portal-cutover-guard.ts) that skips the refetch for
    end-capped chains (INV-18).
  • Wiring patch: portal/wiring/0.16.7.patch. The runtime/historical.ts cutover hunks are
    byte-identical to the 0.16.6 patch; the patch applies cleanly (zero rejects) because the
    grafted seam — HistoricalSync.syncBlockRangeData / syncBlockData, interval-scoped — is
    identical in shape from 0.15.17 through 0.16.7. 0.16.6 → 0.16.7 is a single upstream
    DB-layer change (live-query notification batching, ponder-sh/ponder#2314) that touches none
    of the graft surfaces.

Fixes

The Portal-layer correctness fixes since the first fork cuts, grouped by theme:

  • Silent-gap / phantom-coverage: genesis backfill floor for unbounded sources (#8, #10),
    head-truncated frontier extension (#5), zero-line-200 gap (#66), bounded-backfill head clamp
    and stream-mode delegation fatals (#44), and the mid-range 204 transient-incomplete-range
    close (#47).
  • Stream-realtime + reorg: same-block child logs, finality anchor, reorg prune, head pin,
    parent txs (#26); parentBlockHash on /stream resume + 1-block-orphan 409 healing (#33);
    stream-mode finality floor at persisted finality (restart regression); omnichain
    foreign-checkpoint timestamp mapping (#57); and the bounded historical→realtime cutover skip
    (INV-18).
  • Factory / account-source coverage: persist discovered factory children (#2), flush on the
    RPC-delegation path + survive core's transaction retry (INV-15), write-side idempotence
    (#53), the factory-range gate pin (#21), and a receipt for every tx-filter-matched
    transaction (#24); the extend-path needed-field fatal scoped to a genuine per-filter need — no
    false-fatal on client-unmatched traces (#20) or a shared filter's receipt requirement (#94).
  • Liveness / bounded warmup: bounded time-to-first-durable-commit — the finalized-range
    fetch (and the watermark-following discovery scan) start from a small warmup span and grow
    geometrically, resume-seeded from the persisted watermark on restart (#50).
  • HTTP-client hardening: retry truncated response bodies + loud dataset-start skips (#25),
    fail loud when a provider omits accessList on typed transactions (#27), and zero-backoff /
    timeout hardening (#9 follow-ups, #16).
  • Invariant-first re-architecture: functional core + imperative shell (#6), with the
    head-clamp / discovery-floor / trace-receipt correctness wave (#15) folded in.

Compatibility

  • Built + tested against ponder@0.16.7. The full Portal property/regression suite is green:
    302/302 across ponder@0.15.17, ponder@0.16.6, and ponder@0.16.7
    (scripts/sync-upstream.sh <ver> --test).
  • The dense-source / factory-child discovery paths are additionally re-accepted under a Poisson
    kill/resume chaos campaign: the store stays
    logically byte-identical to a clean baseline across 33 verified resumes with zero invariant
    violations (VALIDATION.md §4.4).
  • Seam verified against: 0.16.7, 0.16.6, 0.15.17 (compat.tested). The seam is stable
    and identical in shape across 0.15.17 – 0.16.7.

@subsquid/ponder 0.16.6-sqd.3

Choose a tag to compare

@dzhelezov dzhelezov released this 08 Jul 14:48
4453802

@subsquid/ponder@0.16.6-sqd.3

Supported-lines backport wave (2026-07-28). -sqd.3 re-cuts ponder@0.16.6 with the current Portal layer — the same portal/ tree shipped on latest (0.17.2-sqd.1). It keeps 0.16.6 a first-class supported line so a user pinned to ponder-0.16.6 runs the same fixes as everyone else.

Published under dist-tag ponder-0.16.6latest is not clobbered. Install: npm i @subsquid/ponder@0.16.6-sqd.3 (or npm i @subsquid/ponder@ponder-0.16.6).

New since 0.16.6-sqd.2

0.16.6-sqd.2 was a 2026-07-08 layer (the correctness wave through the #47 mid-range-204 close, the #50 warmup, and the #20/#94 needed-field scoping). -sqd.3 brings the line onto the current layer, adding:

  • accessList honest-NULL fix (#110, #111) — Portal chains that omit access_list (e.g. arbitrum/avalanche, an upstream 400) now store NULL instead of a fabricated [], byte-identical to geth ground truth.
  • Realtime body-cap filtered fallback (#23).
  • INV-17 realtime-finalize write-side idempotence (#175) — no duplicated factory-child rows on realtime crash/resume.
  • inserted.logs store-inserted metric (#143).
  • Factory log-query byte-budget sharding (#194, #195) and account-tx (from/to) byte-budget sharding (#196, #199) — over-cap IN-list unions split into byte-sized shards, completeness-preserving, byte-identical no-op below the wall.
  • INV-26 store-state partition-invariance codification (invariant catalog now INV-1…INV-26).

The full invariant-first re-architecture and earlier correctness wave carry forward unchanged.

Compatibility

  • Built + tested green on ponder@0.16.6: 400/400 across 23 files (CI=true scripts/sync-upstream.sh 0.16.6 --test, pnpm@9.10.0) — verified both by a pre-publish gate and by the release workflow's own in-CI build+test.
  • The graft seam (HistoricalSync.syncBlockRangeData/syncBlockData) is untouched; every added fix lives in the Portal layer, off the seam. No fresh RPC byte-diff; compat rests on seam-identity + the full suite.