perf(sqlite): bulk-ingest write-path defaults — 64 MiB page cache + 50k WAL autocheckpoint - #950
Merged
Conversation
SQLite's default page cache is 2 MiB per connection. Once the search_index B-trees outgrow it — immediately, on any real dataset — every index INSERT and batch commit evicts and rewrites hot pages, so bulk ingest spends its time shuffling the cache instead of building the trees. Measured on the real 31 GB bulk-submit manifest (single worker, identical 7-minute windows on a fresh database): 232/s -> 289/s (+25%), with index-row INSERT cost falling 1.71 -> 1.24 ms per entry and batch commit 1.95 -> 1.72 ms. A larger batch size was also tried and measured slower (205/s at 5,000 entries): bigger transactions overflow the cache and grow the commit-time checkpoint, which is exactly what the larger cache absorbs. The limit is per pooled connection (pool of 10), but a connection's cache only grows with the pages it actually touches, so idle and read-only connections stay small.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A bulk-ingest batch writes far more than the default 4 MiB wal_autocheckpoint threshold, so every batch commit also ran a checkpoint and paid the WAL-to-database copy inline — commit cost was 1.72 ms per entry, ~50% of the remaining ingest budget. Fewer, larger checkpoints move the same bytes sequentially: measured on the same real-manifest window, 289/s -> 382/s (+32%), with batch-commit cost falling to 0.96 ms per entry. The -wal file now grows to ~40 MiB between checkpoints.
angela-helios
changed the title
Measured stepwise on the same real-manifest window: 10,000 pages took ingest from 289/s to 382/s (batch commit 1.72 -> 0.96 ms per entry); 50,000 pages takes the multi-row branch from 417/s to 501/s (commit -> 0.55 ms per entry). The -wal file grows to ~200 MiB under sustained writes, which a server ingesting tens of gigabytes can afford; light workloads rarely reach the threshold between their own commits.
angela-helios
changed the title
smunini
approved these changes
Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two internal SQLite defaults on the write path, measured independently on identical 7-minute windows of the real 31 GB bulk-submit manifest (single worker, fresh database, on top of #949):
1. Page cache: 2 MiB → 64 MiB per connection
SQLite's default page cache is 2 MiB. The
search_indexB-trees outgrow that immediately, so bulk ingest spends its time evicting and rewriting hot pages.A larger batch size was tried first and measured slower (205/s at 5,000 entries): bigger transactions overflow the small cache and grow the commit-time checkpoint. Recorded so it isn't re-run.
2. WAL autocheckpoint: 1,000 pages → 50,000 pages
A bulk batch writes far more than the default 4 MiB threshold, so every batch commit also ran a checkpoint and paid the WAL→database copy inline.
Measured stepwise on identical windows:
Cost: the
-walfile grows to ~200 MiB under sustained writes (measured 207 MiB); light workloads rarely reach the threshold between their own commits.Memory ceiling
The cache limit is per pooled connection (pool of 10 → ~640 MiB theoretical worst case), but a connection's cache only grows with pages it actually touches — in practice the write connection fills it and idle/read-only connections stay small. No configuration is added; both are internal defaults in the same spirit as WAL mode itself.
Stacked
With #944 + #949 + this PR, the full 954,288-entry import measured 51 minutes end to end before the checkpoint change (comment on #947); the stacked increments here and in #951 project to ~32 minutes, single worker, out of the box. Full sqlite persistence suite green (344).