Skip to content

db: SQLite pragma tuning + chunk GetDepositTxsByIndexes#677

Merged
pk910 merged 2 commits into
ethpandaops:masterfrom
barnabasbusa:bbusa/sqlite-hygiene-and-deposit-chunk
May 6, 2026
Merged

db: SQLite pragma tuning + chunk GetDepositTxsByIndexes#677
pk910 merged 2 commits into
ethpandaops:masterfrom
barnabasbusa:bbusa/sqlite-hygiene-and-deposit-chunk

Conversation

@barnabasbusa
Copy link
Copy Markdown
Collaborator

Two small, independent SQLite-backend fixes. Postgres is unaffected.

1. Pragma tuning alongside WAL

WAL was already enabled, but synchronous, cache_size, and temp_store were left at their defaults — so every commit still fsyncs the WAL and the 2 MB default page cache churns on any write-heavy block. Set standard values:

  • synchronous = NORMAL — safe under WAL (only the most recent commit is at risk on power loss; the DB stays consistent), avoids one fsync per commit.
  • cache_size = -262144 — 256 MB page cache.
  • temp_store = MEMORY — keeps planner-internal tables off disk.
  • busy_timeout = 5000 — wait up to 5s for a competing writer instead of failing immediately.

No behavior change beyond throughput and contention.

2. Chunk GetDepositTxsByIndexes to stay under SQLITE_MAX_VARIABLE_NUMBER

GetDepositTxsByIndexes builds a single WHERE deposit_index IN ($1, $2, ..., $N) with one placeholder per requested index. SQLite's default SQLITE_MAX_VARIABLE_NUMBER is 999 in older builds (32766 in newer), so once the caller passes more than ~999 indexes the query fails outright with too many SQL variables.

This is a correctness bug, not a perf bug — the query crashes. It's reachable on any network with a long pending-deposit queue or any future API path that walks a large index list. It surfaced in practice on a high-deposit-rate devnet:

Error while fetching deposit txs by indexes:
  SQL logic error: too many SQL variables (1)

Chunk the input into batches of 900 (well under the lower SQLite limit) and concatenate results. Behavior is unchanged for inputs below the threshold.

What this PR does not claim to fix

It does not fix the heavier issue of dora's tx-indexer falling behind on blocks with thousands of DepositEvent logs (e.g. an EIP-8254-style 8192-deposit-per-block stress test). That root cause is the volume of rows inserted per block (8192 events + ~70K internal-call rows per block × multiple secondary indexes) hitting the throughput ceiling of the pure-Go SQLite engine. Real fixes there would require either restructuring the per-block insert pattern, or accepting the CGO complexity of mattn/go-sqlite3, or moving the hot path off SQL — all out of scope here. Production deployments use Postgres and don't have this issue.

Test plan

  • Existing tests still pass on SQLite and Postgres backends.
  • GetDepositTxsByIndexes returns the same set whether called with 100, 999, 1000, or 5000 indexes (manual sanity check on a populated DB).

WAL mode was already enabled, but synchronous, cache, and temp-store
settings were left at their defaults. With WAL alone, every commit
still fsyncs the WAL and the 2 MB default page cache churns on any
write-heavy block. Tune to standard values:

- synchronous=NORMAL: safe under WAL (only the most recent commit is
  at risk on power loss; the database stays consistent), avoids one
  fsync per commit.
- cache_size=-262144: 256 MB page cache, removes most page churn on
  realistic indexer workloads.
- temp_store=MEMORY: keeps planner-internal tables off disk.
- busy_timeout=5000: wait up to 5s for a competing writer instead of
  failing immediately when the writer mutex is briefly held.

No behavior change beyond throughput and contention; the durability
guarantee under WAL+NORMAL is the same one CockroachDB, Geth, etc.
already rely on.
GetDepositTxsByIndexes builds a single 'WHERE deposit_index IN
($1, $2, ..., $N)' with one placeholder per requested index. SQLite's
default SQLITE_MAX_VARIABLE_NUMBER is 999 in older builds (32766 in
newer), so once the caller passes more than ~999 indexes the query
fails outright with 'too many SQL variables'.

This is reachable on any network with a long pending-deposit queue or
any future API path that walks a large index list, and it surfaced in
practice on a high-deposit-rate devnet:

  Error while fetching deposit txs by indexes:
    SQL logic error: too many SQL variables (1)

Chunk the input into batches of 900 (well under the lower SQLite
limit) and concatenate results. Behavior is unchanged for inputs
below the threshold.
@pk910 pk910 merged commit add787b into ethpandaops:master May 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants