fix: bound and scope SQLite's refreshStatistics() ANALYZE#226
Merged
Conversation
refreshStatistics() ran a bare, unscoped `ANALYZE` on the SQLite backend: it touched every table in the database file (not just TypeGraph's own — the Postgres backend already scopes to its own tables), and it did a full, unbounded table/index scan per call, unlike Postgres's ANALYZE, which always samples a fixed-size set of rows regardless of table size. bulkCreate/bulkInsert auto-trigger this refresh once a single call's row count crosses AUTO_REFRESH_STATISTICS_ROW_THRESHOLD (#212). A caller streaming a bulk load through repeated bulkInsert() calls — the only practical pattern for a multi-million-row load, and the pattern this repo's own docs recommend — re-triggers the refresh on every batch; with unbounded per-call cost growing with total table size, total load time integrated to O(n^2) (discovered via a real LDBC SNB SF1 benchmark load that didn't finish after 4.5+ hours). Scopes SQLite's ANALYZE to TypeGraph-managed tables (matching Postgres) and sets PRAGMA analysis_limit first, bounding each call's cost the way Postgres's already was. A 100k-200k row reproduction of the original shape now completes in single-digit seconds with load time growing log-ishly with table size, not quadratically.
Merged
pdlug
added a commit
that referenced
this pull request
Jul 5, 2026
ANALYZE), which unblocks the SF1 benchmark run documented as a finding in reports/snb-lane1-results.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a scaling bug discovered while running the LDBC SNB benchmark (#225):
refreshStatistics()on the SQLite backend ran a bare, unscopedANALYZE(no table argument), which does two things wrong:ANALYZEto TypeGraph-managed tables (coreAnalyzeStatementsinpostgres.ts) — SQLite's never did.ANALYZEalways examines a bounded, fixed-size sample of rows (governed bydefault_statistics_target) regardless of table size, so it never showed this problem. SQLite'sANALYZEscans the whole table unless bounded byPRAGMA analysis_limit(a feature that exists for exactly this reason, per SQLite's own docs).bulkCreate/bulkInserton nodes and edges auto-trigger this refresh once a single call's row count crossesAUTO_REFRESH_STATISTICS_ROW_THRESHOLD(1000 rows, added in #212). #212's own PR description says the design deliberately did not cover "loops of small batches that never individually reach the threshold" — but never considered loops of large batches (each already over the threshold), which is exactly what any real streaming bulk loader does for a multi-million-row dataset (and what this repo's ownbackend-setup.mddocs recommend:for (const batch of batches) { await store.nodes.Document.bulkCreate(batch); }). Each such batch independently re-triggers the refresh, and with unbounded per-call cost growing with total table size, total load time integrates to O(n²).Discovered via a real LDBC SNB SF1 load (packages/benchmarks, #225): the comments stage (~2M rows) ran for over 4.5 hours without finishing, versus 46 minutes for the ~1M-row posts stage just before it — a ratio far beyond what row-count alone explains. Isolated the cause with controlled reproductions (60k-200k row loads), confirmed it disappears with a single node kind and no edges, persists without any ontology involved, and correlates with total graph size rather than any one table's size.
Fix
refreshStatistics()on the SQLite backend now:PRAGMA analysis_limit = 1000(SQLite's own suggested value for large databases) before runningANALYZE, bounding per-call cost regardless of table size.ANALYZEto TypeGraph's own tables (typegraph_nodes,typegraph_edges,typegraph_node_uniques, the fulltext table, plus therecorded_*tables when present), matching the Postgres backend.Verification
tests/backends/sqlite/refresh-statistics-scope.test.ts(4 tests): pins theanalysis_limitvalue, asserts an unrelated table sharing the same connection is never touched, asserts no throw whenrecorded_*tables are absent, and bounds batch-to-batch growth in a repeated-large-bulkInsert loop. Confirmed these tests fail against the pre-fix code (reverted the fix, re-ran — 2 of 4 failed as expected) before restoring the fix.tests/auto-refresh-statistics.test.ts(perf: auto-refresh planner statistics after large bulk writes #212's suite): all 8 tests still pass unchanged — no change to the trigger semantics, only to whatrefreshStatistics()itself does on SQLite.packages/typegraphsuite: 229 files / 4525 tests passed (898 skipped, Postgres-dependent — see below), 0 failures.pnpm test:postgres, unaffected by this SQLite-only change but run per this repo's convention for backend changes): 67 files / 1816 tests passed, 0 failures.Messageontology) completes in ~8.4s with only ~2.1x growth from first batch to last (consistent with normal B-tree logarithmic growth); 200k-row load (no ontology) completes in ~22.9s with ~2.7x growth. Both previously would have shown the ~5x-per-58k-row trend that led to the multi-hour SF1 hang.