feat: add parquet receipt store with DuckDB range queries by jewei1997 · Pull Request #2861 · sei-protocol/sei-chain

jewei1997 · 2026-02-11T18:22:55Z

Summary

This PR adds a parquet-based receipt storage backend with DuckDB for efficient range queries on logs, enabling fast eth_getLogs queries across block ranges.

Add parquet backend option (Backend: "parquet" in config)
Parquet files rotate every 500 blocks
DuckDB queries across closed parquet files for efficient log filtering
WAL for crash recovery of in-progress parquet files
Pruning of old parquet files based on KeepRecent config
Build tag support: use -tags duckdb to enable parquet backend

The parquet backend supports the new FilterLogs range query API introduced in #2788, enabling efficient cross-block log queries without falling back to per-receipt fetching.

Dependencies

Depends on feat: add ledger cache layer for receipt store #2788 (ledger cache layer)

Test plan

Receipt store unit tests pass (without duckdb tag)
Parquet store tests pass with -tags duckdb
Integration testing with full node using parquet backend

github-actions · 2026-02-11T18:26:01Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Feb 17, 2026, 5:44 PM

codecov · 2026-02-11T18:54:11Z

Codecov Report

❌ Patch coverage is 71.16858% with 301 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.32%. Comparing base (ccf65fa) to head (fc71aad).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/ledger_db/parquet/store.go	66.44%	56 Missing and 43 partials ⚠️
sei-db/ledger_db/receipt/parquet_store.go	65.53%	62 Missing and 29 partials ⚠️
sei-db/ledger_db/parquet/reader.go	73.26%	48 Missing and 25 partials ⚠️
sei-db/ledger_db/receipt/receipt_cache.go	79.54%	10 Missing and 8 partials ⚠️
sei-db/ledger_db/parquet/wal.go	75.38%	8 Missing and 8 partials ⚠️
sei-db/ledger_db/receipt/cached_receipt_store.go	95.91%	1 Missing and 1 partial ⚠️
sei-db/ledger_db/receipt/receipt_store.go	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2861       +/-   ##
===========================================
+ Coverage   48.36%   57.32%    +8.95%     
===========================================
  Files         671     2097     +1426     
  Lines       50621   172767   +122146     
===========================================
+ Hits        24485    99032    +74547     
- Misses      23988    64850    +40862     
- Partials     2148     8885     +6737

Flag	Coverage Δ
sei-chain	`52.82% <71.16%> (?)`
sei-cosmos	`48.19% <ø> (+<0.01%)`	⬆️
sei-db	`68.72% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
evmrpc/filter.go	`69.07% <ø> (ø)`
sei-db/ledger_db/receipt/cached_receipt_store.go	`92.30% <95.91%> (ø)`
sei-db/ledger_db/receipt/receipt_store.go	`76.96% <80.00%> (ø)`
sei-db/ledger_db/parquet/wal.go	`75.38% <75.38%> (ø)`
sei-db/ledger_db/receipt/receipt_cache.go	`81.11% <79.54%> (ø)`
sei-db/ledger_db/parquet/reader.go	`73.26% <73.26%> (ø)`
sei-db/ledger_db/receipt/parquet_store.go	`65.53% <65.53%> (ø)`
sei-db/ledger_db/parquet/store.go	`66.44% <66.44%> (ø)`

... and 1543 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sei-db/ledger_db/receipt/receipt_cache.go

+		for blockNum, logs := range chunk.logs {
+			if blockNum < fromBlock || blockNum > toBlock {
+				continue
+			}
+			for _, lg := range logs {
+				if matchLog(lg, crit) {
+					logCopy := *lg
+					result = append(result, &logCopy)
+				}
+			}
+		}


sei-db/ledger_db/parquet/reader.go

+	"database/sql"
+	"fmt"
+	"path/filepath"
+	"runtime"


sei-db/ledger_db/parquet/store.go

+	go func() {
+		for {
+			latestVersion := s.latestVersion.Load()
+			pruneBeforeBlock := latestVersion - s.config.KeepRecent
+			if pruneBeforeBlock > 0 {
+				pruned := s.pruneOldFiles(uint64(pruneBeforeBlock))
+				if pruned > 0 && s.log != nil {
+					s.log.Info(fmt.Sprintf("Pruned %d parquet file pairs older than block %d", pruned, pruneBeforeBlock))
+				}
+			}
+
+			// Add jitter to avoid thundering herd
+			jitter := time.Duration(float64(pruneIntervalSeconds)*0.5) * time.Second
+			sleepDuration := time.Duration(pruneIntervalSeconds)*time.Second + jitter
+
+			select {
+			case <-s.pruneStop:
+				return
+			case <-time.After(sleepDuration):
+				// Continue to next iteration
+			}
+		}
+	}()


sei-db/ledger_db/parquet/store.go

go.mod

sei-db/ledger_db/parquet/reader.go

sei-db/ledger_db/receipt/cached_receipt_store.go

sei-db/ledger_db/receipt/receipt_cache.go

sei-db/ledger_db/parquet/reader.go

sei-db/ledger_db/parquet/store.go

+			}
+
+			// Add random jitter (up to 50% of base interval) to avoid thundering herd
+			jitter := time.Duration(rand.Float64()*float64(pruneIntervalSeconds)*0.5) * time.Second


sei-db/ledger_db/parquet/store.go

+			}
+
+			// Add random jitter (up to 50% of base interval) to avoid thundering herd
+			jitter := time.Duration(rand.Float64()*float64(pruneIntervalSeconds)*0.5) * time.Second


## Summary This PR adds a parquet-based receipt storage backend with DuckDB for efficient range queries on logs, enabling fast `eth_getLogs` queries across block ranges. - Add parquet backend option (`Backend: "parquet"` in config) - Parquet files rotate every 500 blocks - DuckDB queries across closed parquet files for efficient log filtering - WAL for crash recovery of in-progress parquet files - Pruning of old parquet files based on `KeepRecent` config - Build tag support: use `-tags duckdb` to enable parquet backend The parquet backend supports the new `FilterLogs` range query API introduced in #2788, enabling efficient cross-block log queries without falling back to per-receipt fetching. ## Dependencies - Depends on #2788 (ledger cache layer) ## Test plan - Receipt store unit tests pass (without duckdb tag) - Parquet store tests pass with `-tags duckdb` - Integration testing with full node using parquet backend

This PR adds a parquet-based receipt storage backend with DuckDB for efficient range queries on logs, enabling fast `eth_getLogs` queries across block ranges. - Add parquet backend option (`Backend: "parquet"` in config) - Parquet files rotate every 500 blocks - DuckDB queries across closed parquet files for efficient log filtering - WAL for crash recovery of in-progress parquet files - Pruning of old parquet files based on `KeepRecent` config - Build tag support: use `-tags duckdb` to enable parquet backend The parquet backend supports the new `FilterLogs` range query API introduced in #2788, enabling efficient cross-block log queries without falling back to per-receipt fetching. - Depends on #2788 (ledger cache layer) - Receipt store unit tests pass (without duckdb tag) - Parquet store tests pass with `-tags duckdb` - Integration testing with full node using parquet backend

jewei1997 added 19 commits February 2, 2026 09:55

feat: add ledger cache layer for receipt store

de009ae

feat: add parquet receipt store with DuckDB range queries

215a286

fix: update fakeReceiptStore.FilterLogs to new signature

982cf16

fix: update fakeReceiptStore.FilterLogs to new signature

432bd51

rename flushBatch to fillCache

3da7c1c

Merge branch 'main' into ledger-cache-layer

3f64159

Merge branch 'main' into ledger-cache-layer

eb54490

fix test

6aa9c6b

make parquet its own package

a9dd371

Merge branch 'ledger-cache-layer' into parquet-receiptdb

4fe5492

fix build issue

e744ecd

remove duckdb stub

9d1d18f

remove parquet enabled

2c364bc

fix go lint

e807617

feat: optimize WAL write path with binary encoding and batched writes

4b294d9

feat: add store helpers and pebble pre-marshaled bytes support

69423d9

fix

83fe0a7

check for errors in db.exec calls duckdb

dd95c8b

Merge branch 'main' into parquet-receiptdb2

9c93d6f

fix make install

f909611

github-advanced-security bot found potential problems Feb 11, 2026

View reviewed changes

remove dead code + other fixes

ef8d773

jewei1997 added the non-app-hash-breaking label Feb 11, 2026

jewei1997 added 2 commits February 11, 2026 13:48

fix go lint and fsync issue

ff07a1e

fix go.mod

18e487c

yzang2019 approved these changes Feb 11, 2026

View reviewed changes