perf(sqlite): worker-queue upgrades from the local-first field survey (0263) - #382
Merged
crs48 merged 14 commits intoJul 5, 2026
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
db.exec() re-parsed every statement; hot-path SQL now runs through a bounded LRU of oo1.Stmt handles. Multi-statement SQL bypasses the cache (prepare compiles only the first statement), exec() and close() finalize all cached handles (exploration 0263). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
New optional SQLiteAdapter.queryBatch(reads[]) implemented natively in the web adapter, exposed as a single scheduled job on SQLiteWorkerHandler, and proxied by WebSQLiteProxy and PortSQLiteAdapter; in-process adapters (memory/electron/expo) loop for parity. The data layer's multi-chunk node hydrate now rides ONE RPC instead of one per 450-id chunk, and getNodes no longer pre-chunks around it (exploration 0263). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
getNode issued a node-metadata queryOne then a properties query — two worker RPCs per node. It now rides the shared single-JOIN hydrate path (exploration 0263). All 1538 data tests pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…sce hits WorkerScheduler now keeps a ring-buffered per-lane accumulator (queue/exec p50/p95, max exec, op counts, coalesced-hit count), always on and exposed via getSchedulerOpStats()/resetSchedulerOpStats() on the worker handler and proxy. Adds a statement-cache micro-benchmark against the old exec path (exploration 0263). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Evidence: the initializing spinner renders synchronously on mount (App.tsx:718) before the useEffect that opens SQLite (App.tsx:360), and the WASM import + sqlite3InitModule() run inside the SQLite worker (web.ts open phases), never on the main thread — matching Notion's never-block-first-paint criterion. No code change needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…r routing The opfs-sahpool VFS holds exclusive handles, so a second tab used to lose the OPFS race and silently run on a non-durable :memory: database (0204). Tabs now elect a leader via navigator.locks; followers get a MessagePort into the leader's SQLite worker via a tiny SharedWorker ferry and speak the existing connectPort protocol. Leader loss rejects in-flight follower calls immediately and idempotent reads re-issue after reconnect/promotion; graceful close drains the scheduler, closes the adapter, then releases the lock; abandoned manual transactions roll back on next client connect; and :memory:-fallback sessions are now counted (exploration 0263, patterned on Notion / wa-sqlite #81 / PowerSync). Unsupported environments (no SharedWorker — Android Chrome) keep the previous per-tab behaviour; multiTab: false opts out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…otion Two real browser tabs share one durable OPFS database via the 0263 leadership layer: tab A elects leader, tab B follows through the SharedWorker ferry with cross-tab read-your-writes, and closing the leader promotes the follower (data survives on OPFS backends). Runs green on chromium, webkit (memory-backend variant — Playwright WebKit lacks OPFS sync handles), and mobile-chromium. Harness excludes sqlite-wasm from vite pre-bundling, matching apps/web. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…validate cache Store changes for schemas no cached query observes now drop before any grouping work; bulk changes (>250) reload only SUBSCRIBED entries and flag unwatched ones stale — served stale-while-revalidate and re-queried on the next subscription (correctness reloads like optimistic-revert stay forced). QueryCache gains row-weight accounting (200 entries / 50k rows, LRU weight eviction that never touches subscribed entries) plus hit/miss/ eviction stats exposed via bridge.getQueryCacheStats() (explorations 0262 Phase C + 0263 P3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…rker-queue-vs-the-local-first-field # Conflicts: # packages/sqlite/src/adapters/web-worker.ts
…ield Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Contributor
🖼️ UI changes in this PRNo visual differences detected in the changed UI. |
Contributor
|
Preview removed for PR #382. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
|
|
||
| /** This tab won the leadership lock after the previous leader went away. */ | ||
| private async promoteToLeader(): Promise<void> { |
| } | ||
|
|
||
| /** Total sessions on this device that fell back to `:memory:` storage. */ | ||
| export function getMemoryFallbackSessionCount(): number { |
crs48
deleted the
claude/0263-sqlite-worker-queue-vs-the-local-first-field
branch
July 5, 2026 18:11
crs48
added a commit
that referenced
this pull request
Jul 5, 2026
…ies, planner stats (0264) (#384) Implements exploration `0264_[x]_QUERY_MODEL_READ_SPEED_THE_REMAINING_LEVERS.md` (10/10 implementation + 6/6 validation items; sequel to #382). ## Wave 1 — mechanical - **ANALYZE hygiene**: `analysis_limit=400` + `PRAGMA optimize=0x10002` at open (web + electron) and a periodic bootSettled-cadence `PRAGMA optimize` — the planner finally has `sqlite_stat1` (skip-scan precondition); previously optimize only ran on close, which browser tabs rarely reach. - **Fused single-RPC queries**: pushed-down descriptors compile a candidate CTE feeding the hydrate join — one worker round-trip instead of id-select→hydrate, with `COUNT(*) OVER ()` folding `count:'exact'` into the same statement. - **Arity padding**: hydrate `VALUES` and IN lists pad to fixed buckets so repeated lookups share one SQL string and hit the 0263 statement cache (different sizes → byte-identical SQL, proven by test). ## Wave 2 — measured - **Aggregated hydration (the big one)**: `json_group_object` collapses EAV rows in SQL — one row per node. Real-WASM benchmark at 450 nodes × 8 props: **8× fewer boundary rows, 4.9× faster hydrate SQL (55.5→11.0ms/chunk), ~10× cheaper structured clone, 4.5× faster end-to-end**; 12.6ms/chunk at a 10k-row table (no O(table) regression). Now the DEFAULT (`aggregatedHydration:false` opts out); both modes verified node-identical; full suite + parity audit green. - **Adaptive indexes behind `xnet:adaptive-indexes`**: creation defers to a bootSettled maintenance scheduler; with the flag on, single custom-property sorts push down to SQL pagination (hydrate one page, not the whole schema; null placement matches the JS comparator exactly). - **Worker-runtime input-latency telemetry**: Event Timing API per-runtime rolling history — the measurement `data-runtime.ts` was waiting on. Default stays `'main'` until field numbers show worker p95 ≤ main p95 (the documented flip criterion). ## Wave 3 — evidence-gated - **JSONB props column: gate RESOLVED against the migration** — the benchmark showed the per-row WASM→JS extraction dominated (not the EAV walk), and SQL aggregation already captures it; a dual-write migration would buy only the residual ~11ms. - **Persisted warm-start snapshots**: the prewarm working set persists at idle (identity-DID + schema-version stamped, 400KB budget) and re-seeds next boot as STALE bridge entries — first paint renders yesterday's rows while the live query revalidates (seeding stale IS the snapshot-vs-live race; Notion's never-cache-first lesson). Live data is never overwritten. ## Validation - 10,128 tests green (incl. 1,554 data tests with the parity audit enabled in both hydration modes), typecheck + lint clean, benchmark numbers recorded in the doc. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements exploration
0263_[x]_SQLITE_WORKER_QUEUE_VS_THE_LOCAL_FIRST_FIELD.md(all 12 implementation + 7 validation items checked; also carries exploration 0262 and checks off its overlapping Phase B/C items).P1 — worker-internal efficiency
db.exec()re-parsed every call; multi-statement SQL bypasses the cache sinceprepare()compiles only the first statement; handles finalize onexec()/close).queryBatch(reads[])across adapter/worker/proxy/port — several reads in ONE worker round-trip; the data layer's multi-chunk hydrate (450 ids/chunk) andgetNode()(was 2 RPCs) now ride it.getSchedulerOpStats(), plus a micro-benchmark against the old exec path.P2 — multi-tab leadership (the field-standard fix; Notion / wa-sqlite #81 / PowerSync)
navigator.locks; followers get a MessagePort into the leader's SQLite worker through a tiny SharedWorker ferry — the second tab no longer silently falls back to a non-durable:memory:database (0204).:memory:-fallback sessions counted.multiTab: falseopts out; environments without SharedWorker (Android Chrome) keep the previous behaviour.tests/e2e/src/multitab-sqlite.spec.ts): two tabs share one durable OPFS DB with cross-tab read-your-writes; killing the leader promotes the follower. Green on chromium, webkit, mobile-chromium.P3 — read tier (0262 Phase C)
QueryCache: row-weight-aware LRU eviction (200 entries / 50k rows, never evicts subscribed entries) + hit/miss/eviction stats.Validation
🤖 Generated with Claude Code