Skip to content

fix: bound and scope SQLite's refreshStatistics() ANALYZE#226

Merged
pdlug merged 1 commit into
mainfrom
fix/sqlite-analyze-scaling
Jul 5, 2026
Merged

fix: bound and scope SQLite's refreshStatistics() ANALYZE#226
pdlug merged 1 commit into
mainfrom
fix/sqlite-analyze-scaling

Conversation

@pdlug

@pdlug pdlug commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a scaling bug discovered while running the LDBC SNB benchmark (#225): refreshStatistics() on the SQLite backend ran a bare, unscoped ANALYZE (no table argument), which does two things wrong:

  1. Unscoped: it re-analyzes every table in the database file, not just TypeGraph's own tables. The Postgres backend already scopes its ANALYZE to TypeGraph-managed tables (coreAnalyzeStatements in postgres.ts) — SQLite's never did.
  2. Unbounded: it does a full table/index scan per call. Postgres's ANALYZE always examines a bounded, fixed-size sample of rows (governed by default_statistics_target) regardless of table size, so it never showed this problem. SQLite's ANALYZE scans the whole table unless bounded by PRAGMA analysis_limit (a feature that exists for exactly this reason, per SQLite's own docs).

bulkCreate/bulkInsert on nodes and edges auto-trigger this refresh once a single call's row count crosses AUTO_REFRESH_STATISTICS_ROW_THRESHOLD (1000 rows, added in #212). #212's own PR description says the design deliberately did not cover "loops of small batches that never individually reach the threshold" — but never considered loops of large batches (each already over the threshold), which is exactly what any real streaming bulk loader does for a multi-million-row dataset (and what this repo's own backend-setup.md docs recommend: for (const batch of batches) { await store.nodes.Document.bulkCreate(batch); }). Each such batch independently re-triggers the refresh, and with unbounded per-call cost growing with total table size, total load time integrates to O(n²).

Discovered via a real LDBC SNB SF1 load (packages/benchmarks, #225): the comments stage (~2M rows) ran for over 4.5 hours without finishing, versus 46 minutes for the ~1M-row posts stage just before it — a ratio far beyond what row-count alone explains. Isolated the cause with controlled reproductions (60k-200k row loads), confirmed it disappears with a single node kind and no edges, persists without any ontology involved, and correlates with total graph size rather than any one table's size.

Fix

refreshStatistics() on the SQLite backend now:

  • Sets PRAGMA analysis_limit = 1000 (SQLite's own suggested value for large databases) before running ANALYZE, bounding per-call cost regardless of table size.
  • Scopes ANALYZE to TypeGraph's own tables (typegraph_nodes, typegraph_edges, typegraph_node_uniques, the fulltext table, plus the recorded_* tables when present), matching the Postgres backend.

Verification

  • New test file tests/backends/sqlite/refresh-statistics-scope.test.ts (4 tests): pins the analysis_limit value, asserts an unrelated table sharing the same connection is never touched, asserts no throw when recorded_* tables are absent, and bounds batch-to-batch growth in a repeated-large-bulkInsert loop. Confirmed these tests fail against the pre-fix code (reverted the fix, re-ran — 2 of 4 failed as expected) before restoring the fix.
  • Existing tests/auto-refresh-statistics.test.ts (perf: auto-refresh planner statistics after large bulk writes #212's suite): all 8 tests still pass unchanged — no change to the trigger semantics, only to what refreshStatistics() itself does on SQLite.
  • Full packages/typegraph suite: 229 files / 4525 tests passed (898 skipped, Postgres-dependent — see below), 0 failures.
  • Property-based suite: 345 passed / 1 skipped, 0 failures.
  • Postgres suite (pnpm test:postgres, unaffected by this SQLite-only change but run per this repo's convention for backend changes): 67 files / 1816 tests passed, 0 failures.
  • Manual repro: 100k-row load (with the Message ontology) completes in ~8.4s with only ~2.1x growth from first batch to last (consistent with normal B-tree logarithmic growth); 200k-row load (no ontology) completes in ~22.9s with ~2.7x growth. Both previously would have shown the ~5x-per-58k-row trend that led to the multi-hour SF1 hang.

refreshStatistics() ran a bare, unscoped `ANALYZE` on the SQLite
backend: it touched every table in the database file (not just
TypeGraph's own — the Postgres backend already scopes to its own
tables), and it did a full, unbounded table/index scan per call, unlike
Postgres's ANALYZE, which always samples a fixed-size set of rows
regardless of table size.

bulkCreate/bulkInsert auto-trigger this refresh once a single call's
row count crosses AUTO_REFRESH_STATISTICS_ROW_THRESHOLD (#212). A
caller streaming a bulk load through repeated bulkInsert() calls — the
only practical pattern for a multi-million-row load, and the pattern
this repo's own docs recommend — re-triggers the refresh on every
batch; with unbounded per-call cost growing with total table size,
total load time integrated to O(n^2) (discovered via a real LDBC SNB
SF1 benchmark load that didn't finish after 4.5+ hours).

Scopes SQLite's ANALYZE to TypeGraph-managed tables (matching Postgres)
and sets PRAGMA analysis_limit first, bounding each call's cost the way
Postgres's already was. A 100k-200k row reproduction of the original
shape now completes in single-digit seconds with load time growing
log-ishly with table size, not quadratically.
@pdlug pdlug merged commit 4cd6b4c into main Jul 5, 2026
12 checks passed
@pdlug pdlug deleted the fix/sqlite-analyze-scaling branch July 5, 2026 06:16
@github-actions github-actions Bot mentioned this pull request Jul 5, 2026
pdlug added a commit that referenced this pull request Jul 5, 2026
ANALYZE), which unblocks the SF1 benchmark run documented as a finding
in reports/snb-lane1-results.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant