feat(sqlite): parallel SQLite readers in Electron (WAL + worker_threads)#282
Merged
crs48 merged 5 commits intoJun 26, 2026
Merged
Conversation
added 5 commits
June 26, 2026 05:45
… Electron Front the Electron better-sqlite3 adapter with the priority scheduler, add an optional read-only secondary connection, a read-only worker_threads reader pool for heavy parallel reads (the browser can't — opfs-sahpool holds an exclusive handle), cooperative yielding in applyNodeBatch, and WAL/scheduler diagnostics (exploration 0230).
Turn on the read-only secondary connection and an auto-sized reader-thread pool for the desktop data process, and expose a sqlite:diagnostics IPC returning scheduler depth, reader-pool occupancy, and WAL growth (exploration 0230).
Contributor
|
Preview removed for PR #282. |
crs48
deleted the
claude/0230-parallel-sqlite-readers-in-electron-wal-and-work
branch
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| * interleave. Whole-batch atomicity is traded for yield points — safe here | ||
| * because node-batch writes are idempotent LWW upserts. | ||
| */ | ||
| async applyNodeBatch(input: SQLiteNodeBatchApplyInput): Promise<SQLiteNodeBatchApplyResult> { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements exploration 0230.
Why
0228 found the browser can't run SQLite readers in parallel:
opfs-sahpoolholds an exclusive handle, so a second connection falls back to in-memory. That wall is browser-only. Electron uses nativebetter-sqlite3already in WAL mode, where SQLite supports one writer concurrent with many readers — each on its own connection. The bottleneck wasn't SQLite; it was that the whole desktop data layer ran on one synchronous thread.What landed
scheduler, default on) — interactive reads are served ahead of a queued write/import burst; transaction-internal ops run inline (today's semantics preserved).readonlyReadConnection) — a read-only secondary connection so plain reads don't contend with write locks.readerPoolSize: 'auto') — read-onlyworker_threadsbetter-sqlite3connections that serve heavy reads (FTS / aggregates / big scans) in parallel, with least-busy dispatch, anisHeavyReadheuristic so cheap reads stay inline, and graceful fallback to the inline connection if a worker can't boot.applyNodeBatch— chunked atomic transactions withsetImmediatebetween chunks so a long import no longer monopolizes the data-process thread.getDiagnostics()/getWalStats()/checkpointWal()on the adapter, plus asqlite:diagnosticsIPC; read-your-writes window (readYourWritesWindowMs).Proof
electron-reader-pool.test.tsspawns realworker_threadsand shows two heavy reads complete in ≈ max, not sum (ratio ~1.00) — genuine parallelism.applyNodeBatchyields so a read interleaves mid-import; cheap reads stay inline; WAL checkpoint bounds growth.worker-scheduler.tsreverted tomain(no changes) — the shared scheduler tests still pass.Deferred (gated on environment)
Three validation items need a signed/packaged desktop build + a soak run, which can't be exercised in CI here — so the exploration stays unchecked on those. The reader pool is best-effort, so a packaging failure degrades to inline reads rather than breaking the app.
🤖 Generated with Claude Code