perf(data): stop query-debug diagnostics convoying the SQLite worker - #388
Merged
Conversation
With xnet:query:debug enabled, plan diagnostics issued EXPLAIN QUERY PLAN + PRAGMA schema_version + one PRAGMA index_info per index (~29) as separate round-trips per query on the single serial SQLite worker. The #351 schema_version-keyed cache only populated after a build finished, so concurrent boot queries all missed it and each enqueued its own full build — hundreds of identical index_info round-trips convoying real query results by 18-20s (2026-07-05 capture: loadDoc readMs 19637 with worker execMs 0). - getIndexInfo shares one in-flight probe+build across concurrent callers, and fetches all index metadata in ONE batched pragma_index_info join (per-index fallback for runtimes without table-valued pragmas): a cold diagnostic run is 2 round-trips, warm 1. - The storage adapter memoizes plan diagnostics per unique compiled SQL shape per session (bound values are params, so keyset pages share an entry), cleared when adaptive indexes are created/dropped, capped at 512 entries. - Regression tests count worker round-trips per diagnostic run at the diagnostics layer, verify concurrent dedupe + fallback + a real sql.js engine pass, and assert one EXPLAIN per SQL shape at the adapter layer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Contributor
|
Preview removed for PR #388. |
crs48
added a commit
that referenced
this pull request
Jul 6, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @xnetjs/abuse@0.1.1 ### Patch Changes - Updated dependencies []: - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 ## @xnetjs/cli@0.0.5 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/sqlite@0.1.1 - @xnetjs/data@0.1.1 - @xnetjs/plugins@0.1.1 - @xnetjs/runtime@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/crypto@0.1.1 ### Patch Changes - Updated dependencies []: - @xnetjs/core@0.1.1 ## @xnetjs/data@0.1.1 ### Patch Changes - [#388](#388) [`2ab72a9`](2ab72a9) Thanks [@crs48](https://github.com/crs48)! - Query-plan debug diagnostics no longer convoy the SQLite worker. With `xnet:query:debug` enabled, every query used to issue EXPLAIN QUERY PLAN + PRAGMA schema_version + one PRAGMA index_info per index as separate serial worker round-trips — hundreds per boot, delaying real query results by 18-20s. `getIndexInfo` now dedupes concurrent callers onto one in-flight build and fetches all index metadata in a single batched `pragma_index_info` join (with a per-index fallback for runtimes without table-valued pragmas), and the storage adapter collects plan diagnostics once per unique compiled SQL shape per session instead of per execution (invalidated when adaptive indexes are created or dropped). - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/sqlite@0.1.1 - @xnetjs/storage@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/data-bridge@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/sqlite@0.1.1 - @xnetjs/data@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/history@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/identity@0.1.1 ### Patch Changes - Updated dependencies []: - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/plugins@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/abuse@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/react@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/data-bridge@0.1.1 - @xnetjs/history@0.1.1 - @xnetjs/plugins@0.1.1 - @xnetjs/runtime@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/runtime@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/data-bridge@0.1.1 - @xnetjs/storage@0.1.1 - @xnetjs/history@0.1.1 - @xnetjs/plugins@0.1.1 - @xnetjs/sync@0.1.1 - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/sqlite@0.1.1 ### Patch Changes - [#388](#388) [`2ab72a9`](2ab72a9) Thanks [@crs48](https://github.com/crs48)! - Query-plan debug diagnostics no longer convoy the SQLite worker. With `xnet:query:debug` enabled, every query used to issue EXPLAIN QUERY PLAN + PRAGMA schema_version + one PRAGMA index_info per index as separate serial worker round-trips — hundreds per boot, delaying real query results by 18-20s. `getIndexInfo` now dedupes concurrent callers onto one in-flight build and fetches all index metadata in a single batched `pragma_index_info` join (with a per-index fallback for runtimes without table-valued pragmas), and the storage adapter collects plan diagnostics once per unique compiled SQL shape per session instead of per execution (invalidated when adaptive indexes are created or dropped). ## @xnetjs/storage@0.1.1 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/sqlite@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/sync@0.1.1 ### Patch Changes - Updated dependencies []: - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 - @xnetjs/core@0.1.1 ## @xnetjs/core@0.1.1 ## xnet-cloud@0.0.3 ### Patch Changes - Updated dependencies []: - @xnetjs/cloud@0.0.1 - @xnetjs/crypto@0.1.1 ## @xnetjs/brain@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/vectors@0.0.1 ## @xnetjs/comms@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/crypto@0.1.1 ## @xnetjs/dashboard@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/plugins@0.1.1 - @xnetjs/react@0.1.1 - @xnetjs/social@0.0.4 ## @xnetjs/labs@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/plugins@0.1.1 ## @xnetjs/licenses@0.0.4 ### Patch Changes - Updated dependencies []: - @xnetjs/crypto@0.1.1 ## @xnetjs/maps@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 ## @xnetjs/server@0.0.3 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/data-bridge@0.1.1 - @xnetjs/identity@0.1.1 - @xnetjs/crypto@0.1.1 ## @xnetjs/social@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 - @xnetjs/crypto@0.1.1 ## @xnetjs/unreal@0.0.4 ### Patch Changes - Updated dependencies [[`2ab72a9`](2ab72a9)]: - @xnetjs/data@0.1.1 ## xnet-desktop@0.1.1 Desktop shell release riding the @xnetjs/core 0.1.1 train. Desktop-specific changes are not tracked here; see the core packages' changelogs for what shipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
xnet:query:debugenabled, the query-plan diagnostics path issued per query:EXPLAIN QUERY PLAN+PRAGMA schema_version+ onePRAGMA index_info()per index (~29), each a separate round-trip to the single serial SQLite worker. A live capture (2026-07-05, post-#381, fast 8MB database, worker execMs 0-2ms) showed hundreds of index_info round-trips per boot; the aggregate convoy delayed real query results by 18-20s (loadDocreadMs 19637 with worker exec 0ms) while boot infra finished in 1.4s. Debug mode was distorting the very thing it measures.Why #351's cache missed
The schema_version-keyed cache added in #351 is correct for sequential calls, but it only populates after a full build finishes. Boot fires dozens of debug-instrumented queries concurrently — every one checked the still-empty cache and enqueued its own full
sqlite_master+ 29×index_infobuild onto the serial worker. Hence identical index_info calls repeating dozens of times in the capture.Fix
@xnetjs/sqlite(diagnostics.ts)getIndexInfonow shares ONE in-flight probe+build across concurrent callers (the resolved cache stays keyed onPRAGMA schema_version).pragma_index_infotable-valued join (SQLite ≥ 3.16), with the per-index loop kept as a fallback for runtimes without table-valued pragmas. Cold diagnostic run = 2 worker round-trips, warm = 1.@xnetjs/data(sqlite-adapter.ts)?params, so keyset pages share an entry) and served from a memo after that — instead of per execution. The memo is cleared when the adapter creates/drops an adaptive index (plans may change), skips caching failed collections, and is capped at 512 entries (IN-list binds mint one shape per list length).Regression tests
EXPLAIN QUERY PLANper compiled SQL shape across sequential, value-varied, and 8-way concurrent executions, with plan diagnostics still present on every result.Full suite: 924 files / 10,133 tests green locally. Changeset (patch
@xnetjs/sqlite+@xnetjs/data) and changelog fragment (performance,devtools) included.🤖 Generated with Claude Code