Skip to content

@subsquid/pipes v1.0.0-alpha.18

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 24 Jul 13:17
· 22 commits to main since this release

Prerelease. Install with npm i @subsquid/pipes@alpha.

Parquet target

  • Pluggable segment-writer engine (#124). settings.engine on parquetTarget now accepts any ParquetEngine implementation; the SDK ships parquetjsEngine as the built-in, zero-config default. The target owns staging, coverage-window naming, output verification, publication and crash recovery; an engine owns only the rows→Parquet-file step, so alternative encoders (e.g. a DuckDB-backed engine) can plug in without touching durability or recovery.
  • Encoding options moved to the engine factory. settings.rowGroupSize and settings.compression are removed — pass them to the engine instead: parquetjsEngine({ rowGroupSize, compression }). Zero-config behaviour is unchanged (same 100k / SNAPPY defaults).
  • Files are named for the block window they cover (#123), <from>-<to>.parquet, not for the min/max block of the rows inside them. A range absent from a table's filenames now means "not indexed" rather than "indexed, no data". Locate a block's rows via the row-group footer statistics, not the filename. This changes on-disk layout; existing datasets are read back correctly, but contiguity begins from the upgrade point.

EVM

  • Portal trace shapes accept nulls that occur on-chain: a reverted CREATE whose result.address is null, and suicide traces carrying a null address / balance. Previously these made trace validation throw.

Solana

  • A decoder now covers exactly one program/ABI, making multi-program decode wiring explicit.

Decoding (EVM + Solana)

  • Unified decode-error hook policy across EVM and Solana, so both surface decode failures through the same mechanism.

Breaking changes

  • parquetTarget: settings.rowGroupSize / settings.compression removed — use the engine factory. The exported ParquetStore constructor now takes an engine.
  • Parquet on-disk file naming changed to coverage windows (see above).

What to test

Parquet backfills end to end — file naming, rollover, and crash-recovery — plus any EVM traces pipeline that touches reverted CREATE / self-destruct traces.