perf(sync): run cold-open maintenance off the boot path (0260) - #366
Merged
Conversation
Background maintenance passes (change-log compaction, one-time VACUUM, presence-blob cleanup) were scheduled off `requestIdleCallback`, which measures MAIN-thread idle — and the main thread is idle exactly while the single SQLite worker is saturated serving the cold-open landing read. So those idle callbacks fired mid-boot and their heavy DELETE/VACUUM ops queued behind (and added to) first paint on the one serial connection. `bootSettled()` resolves when `query:first-rows` is marked (first landing data rendered). `runWhenBootSettled(task)` waits for that, then a settle delay, then a real idle slot — with a fallback timer so a stalled boot still eventually runs the work. This is the gate the maintenance passes now hang off, keeping them strictly off the boot critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
… (0260) #360 scheduled compaction on `requestIdleCallback` and pruned up to 250 k rows in one pass. On the single serial SQLite worker that ~doubled the cold-open (31.5 s): the idle callback fired mid-boot and ~40 back-to-back DELETE chunks queued ahead of the landing read. Rework compaction to never touch the boot critical path: - Gate `scheduleChangeLogCompaction` behind `runWhenBootSettled` (first paint + settle delay + real idle) instead of a bare idle callback. - Prune in small CHUNK=2000 passes, capped at 25 chunks/session, with an idle yield between each so a landing read or interaction always preempts, looping-until-dry across boots. The web adapter has no priority scheduler (ops are FIFO to the one worker), so small chunks + yield ARE the in-flight guard. - Bail the pass on the kill switch or a hidden tab (throttled idle / possible tab discard); resume next boot. - Stop re-arming the full VACUUM every prune (`xnet:db-vacuumed:v1` is no longer cleared) — that made the NEXT boot pay a whole-file rewrite. Compaction stays default-on with an opt-out kill switch (`localStorage['xnet:compact:changes'] = 'off'`); the emergency default-off "Hotfix A" is unnecessary now that the pass is boot-safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…ed (0260) `scheduleOneTimeVacuum` and `scheduleStalePresenceCleanup` had the same `requestIdleCallback` flaw as compaction: a heavy DELETE + whole-file VACUUM fired mid-boot, queuing behind the cold-open landing read on the single serial worker. Both now run via `runWhenBootSettled` (first paint + idle), so file reclaim happens strictly after the workspace is interactive. The one-shot VACUUM is the boot-settled space-reclaim step that pairs with compaction no longer re-arming it every prune. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…260) On a cold boot against a reset hub, `node-sync-response` reported `highWaterMark 0` while the local confirmed cursor was ~318 k. The rollback guard read that as a hub rollback and re-offered the ENTIRE local change log via `getChangesSince(0)` — two ~3 s full-table scans on the single worker (adding to cold-open), every change then rejected INVALID_HASH, tripping the outbound breaker anyway. Gate the guard: only re-offer for a genuine PARTIAL rollback (`0 < highWaterMark < cursor`) and only while the outbound breaker is not already halted. A fresh/empty/reset hub (`highWaterMark === 0`) and a tripped INVALID_HASH breaker both short-circuit, so there is no unbounded `getChangesSince(0)` re-offer flood. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
…e it off the boot path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Contributor
🖼️ UI changes in this PRNo UI changes detected in this PR. |
Contributor
|
Preview removed for PR #366. |
crs48
added a commit
that referenced
this pull request
Jul 4, 2026
…-vacuum (0260) (#369) ## Why The durable half of the cold-open fix that [#366](#366) set up but deferred (the "REMAINING" item in exploration 0260). Change-log compaction (0254) DELETEs superseded history, but the OPFS database opened with the default `auto_vacuum = NONE`, so those pages only re-entered SQLite's **freelist** — the file never shrank. Since the ~16 s cold-open read is the OPFS page-fault of that file's working set, pruning rows did nothing for cold-open until the file physically shrank. The only thing that returned space to the OS was the **one-time boot VACUUM**, which latches after a single run and so typically rewrote a still-fat file and never ran again. ## What Convert the OPFS working set to **incremental auto-vacuum**, so freed pages are handed back to the filesystem per boot: - **[`web.ts`](packages/sqlite/src/adapters/web.ts)** — open OPFS databases with `PRAGMA auto_vacuum = INCREMENTAL`. Fresh databases are incremental from birth (never bloat); the mode only *converts* an existing `NONE` database at a `VACUUM`. - **[`db-vacuum.ts`](apps/web/src/lib/db-vacuum.ts)** — the existing one-time boot-settled VACUUM now doubles as that conversion for pre-existing databases (comment only — a `VACUUM` applies the pending mode change automatically). - **[`change-log-compaction.ts`](apps/web/src/lib/change-log-compaction.ts)** — after each prune pass, run `PRAGMA incremental_vacuum` to return the freed pages to the OS. A harmless no-op until the DB is converted; per-boot reclaim afterwards. The file shrinks a little every idle boot as the log drains, instead of only on the single one-time VACUUM. ## Convergence - First boot after deploy (existing fat DB): one-time VACUUM converts `NONE → INCREMENTAL` and reclaims whatever compaction has deleted so far. - Each subsequent idle boot: compaction deletes ≤50k rows and `incremental_vacuum` reclaims them — the file shrinks steadily until the ~318k-row backlog is drained (~7 boots), and it **keeps** reclaiming for the workspace's whole life instead of latching once. ## Validation New `packages/sqlite/src/adapters/auto-vacuum-reclaim.test.ts` proves the exact SQLite semantics against a real engine (better-sqlite3): 1. `NONE` (default) → a DELETE does **not** shrink the file and `incremental_vacuum` is a no-op. 2. A fresh `INCREMENTAL` DB → `incremental_vacuum` drops both `page_count` **and** on-disk bytes. 3. A pre-existing `NONE` DB → one `VACUUM` converts it (and compacts), then `incremental_vacuum` reclaims per call. Plus `change-log-compaction.test.ts` gains cases asserting `PRAGMA incremental_vacuum` fires only when rows were actually pruned (not on a dry log, kill-switch, or memory DB). Local: `pnpm typecheck` (91/91), `pnpm lint` (0 errors), sqlite unit suite (158 pass / 3 skip), apps/web lib (216 pass). Follow-up to 0260 / [#366](#366). Note: pairs with deploying #366 — that stops the compaction-on-boot regression and the reset-hub flood; this makes the pruning actually shrink the file that gates cold-open. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 4, 2026
…y Site (#370) ## Why the live site is still serving old code Every **Deploy Site to GitHub Pages** run since 2026-07-03 18:06 UTC has **failed** — 7 in a row — at the "Install site dependencies" step: ``` ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with package.json * 2 dependencies were added: @types/qrcode@^1.5.6, qrcode@^1.5.4 ``` PR #363 (Run on Mobile / Expo Go QR page) added `qrcode` + `@types/qrcode` to `site/package.json` without regenerating `site/pnpm-lock.yaml` (the site is a standalone `--ignore-workspace` install with its own lockfile, so the root lockfile check didn't catch it). Result: the production site — including the `/app/` PWA bundle — has been pinned at `1ed5ab82`, which predates the cold-open fixes #366 and #369. That is exactly why every cold-boot capture since still shows the pre-#366 signature (`hub high-water mark 0 … re-offering`). ## Fix Regenerate `site/pnpm-lock.yaml` (Node 23 / pnpm 10). Diff is 220 additions, zero removals: `qrcode`, `@types/qrcode`, their transitive deps (`dijkstrajs`, `pngjs`, yargs CLI chain), plus inert `libc` metadata pnpm 10 adds. ## Verified locally - `pnpm install --ignore-workspace --frozen-lockfile` in `site/` — passes (CI's exact failing command). - Full Astro site build — passes, 110 pages (the QR page has never made it past install in this workflow, so the build step was proven too). Merging touches `site/**`, which triggers the production deploy — shipping everything accumulated since #363, including #366 + #369. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 4, 2026
## The second deploy-pipeline bug (stacked on #370) After #370 unblocked Deploy Site, run `28709576907` went green — but the live app **still served pre-#366 code**. The publish commit (`e99fa1de`) tells the story: all the chunk renames landed (`index-DppD45_e.js → index-BuveDUTb.js`, `web-worker-BIYC6Nw7.js → web-worker-BFqMZy2X.js`, …) but **`app/index.html`, `app/404.html`, and `app/sw.js` are absent from the diffstat** — gh-pages ended up with a stale `index.html` pointing at a deleted entry chunk. ## Root cause `publish-gh-pages` syncs with `rsync -a`, whose quick check skips files whose **size and mtime** both match: - **Size never changes for these files.** A rebuild substitutes 8-char content hashes for 8-char content hashes, so `index.html`/`404.html`/`sw.js` are byte-for-byte the *same length* across builds. - **mtime carries no signal.** Both sides are freshly generated — `cp -R` build output (prepare step) vs a just-`git worktree add`-ed checkout (publish step) — every mtime is "now". This run's prepare-copy and checkout landed in the **same second**, so the quick check called them identical. Every prior deploy dodged this by a one-second timing margin; it's a latent race in every publish (production, PR previews, branch previews — all use this action). ## Fix `rsync -a --checksum` — decide by content, deterministically. `git add -A` already ignores genuinely unchanged files, so the only cost is reading both trees. ## After merge `.github/**` isn't in Deploy Site's path filters, so I'll `workflow_dispatch` a production deploy, then verify `gh-pages:app/index.html` references the live entry chunk and the deployed bundle carries the #366/#369 markers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 5, 2026
…um (0260) (#380) ## Problem Live capture from the newly-deployed build (post #366/#369) showed change-log compaction running but the OPFS file never shrinking. Root cause: the one-time VACUUM from exploration 0233 latched its `xnet:db-vacuumed:v1` localStorage flag on long-lived profiles **before** incremental auto-vacuum existed (#369). For those profiles: - the database is still `auto_vacuum = NONE` (the pending `INCREMENTAL` mode set at open in web.ts only applies to a fresh DB or at a VACUUM), - so every per-prune `PRAGMA incremental_vacuum` (#369) is a **silent no-op**, - and the file can never shrink, no matter how many rows compaction deletes. ## Fix Gate the one-time VACUUM on the database's **actual** `PRAGMA auto_vacuum` mode, not the flag alone: - flag latched **and** mode = INCREMENTAL (2) → skip (steady state: one idle PRAGMA read per boot) - flag latched **but** mode = NONE (0) → run the conversion VACUUM anyway (the 0233-era profile case) - no flag → vacuum + latch as before Scheduling is unchanged — still boot-settled + idle via `runWhenBootSettled` (#366), so the whole-file rewrite never races the cold-open read burst. Bonus: the conversion VACUUM immediately returns the freelist accumulated by prior prunes (~250k rows) to the OS. ## Testing - `db-vacuum.test.ts` extended: latched+INCREMENTAL skips, latched+NONE converts, unflagged vacuums+latches, memory mode skips — 4/4 pass - full `apps/web/src/lib/` suite: 217/217 pass - `pnpm --filter xnet-web typecheck` + eslint clean - full `pnpm test`: 10,020 passed (pre-push devkit git-integration failures are hook-env-only; suite is green outside the hook) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 5, 2026
…page per pass (#381) ## How this was found Local end-to-end validation of the cold-open fixes (#366/#369/#380): seeded a **464 MB replica** of the user's pathological profile straight into a real browser's OPFS database (300k-row `changes` log, 250k rows deleted into the freelist → 81% dead pages, devolved to `auto_vacuum=NONE`, latched 0233-era `xnet:db-vacuumed:v1` flag), then booted the shipped code against it repeatedly. **What validated clean:** - Boot 1: #380's conversion VACUUM fired post-settle — **464 MB → 80 MB**, `auto_vacuum` 0 → 2, first paint 1823 ms → 1090 ms on the following boot. - Boot 2: vacuum correctly skipped (flag + mode 2); compaction pruned **exactly the 47,000 prunable rows** (K1/K2/K3 predicate verified by dry-run count beforehand). - Boot 3: steady state — compaction dry, vacuum skip, one idle PRAGMA read. **What didn't:** boot 2 ended with `{deleted: 47000, reclaimed: true}` — and the file still at 80 MB with 9,248 freelist pages. Direct probing showed every `exec('PRAGMA incremental_vacuum')` (with or without a page-count argument) freed **exactly one page**. ## Root cause A WASM *binding* quirk, not engine semantics: SQLite frees one freelist page per `sqlite3_step` of `PRAGMA incremental_vacuum`, and sqlite-wasm's oo1 `exec` steps a row-less statement exactly once. better-sqlite3's `pragma()` steps to completion — which is why #369's engine-level test passed while the production reclaim was a near-no-op (at one page per boot, a 75 MB backlog would take ~9,000 boots). ## Fix - `SQLiteAdapter.incrementalVacuum(maxPages?)` — new **optional** interface method returning pages freed. - Web adapter: `stepIncrementalVacuumToCompletion` steps the pragma until done (or the cap); wired through the worker (write lane) + proxy. - Electron adapter: via `db.pragma` + freelist delta (better-sqlite3 already steps fully). - `change-log-compaction.ts` prefers the method (falls back to `exec` on adapters without it), and a **dry pass now reclaims a stranded freelist backlog** (≥1024 pages ≈ 8 MiB) — healing profiles that pruned under the buggy build or bailed mid-pass on a hidden tab. ## Re-validation (same seeded profile, fixed build) - Backlog branch: first boot with the fix returned all 9,244 stranded pages — **80 MB → 8 MB** in one pass. - Integrated path: seeded 6,000 fresh prunable rows → pass reported `{deleted: 5400, freedPages: 403, reclaimed: true}` (5,400 = exact predicted count) and the file returned to baseline. ## Testing - NEW `incremental-vacuum-stepping.test.ts` — pins the one-page-per-exec bug **against the real @sqlite.org/sqlite-wasm oo1 API** and the fix (drains freelist, honours cap): 3/3. - `change-log-compaction.test.ts` extended (method preferred, exec fallback, dry-small-freelist no-op, backlog reclaim): 9/9. - Full suite 10,025 passed; typecheck 91/91; eslint clean. - Changeset: `@xnetjs/sqlite` minor. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem (exploration 0260)
After #360 shipped change-log compaction, cold-open got worse (~31.5 s vs the ~15.8 s baseline), not better. Root cause: the single OPFS SQLite worker is strictly serial (one exclusive sync-access handle → no second connection, ever), and the maintenance passes were scheduled off bare
requestIdleCallback. That callback measures main-thread idle — which is idle exactly while the worker is saturated serving the cold-open landing read. So compaction fired mid-boot and its ~40 back-to-back 250 k-row DELETE chunks queued ahead of first paint. A reset hub compounded it: the rollback guard re-offered the entire 318 k-row log via two ~3 sgetChangesSince(0)scans, every change then rejectedINVALID_HASH.The fix — Phase 1.1 (the full scheduling fix)
bootSettled()/runWhenBootSettled()(boot-timeline.ts): a gate that resolves atquery:first-rows(first landing data painted), then waits a settle delay + a real idle slot, with a fallback timer for stalled boots.CHUNK=2000passes (≤25/session) with an idle yield between each, loops-until-dry across boots, bails on the kill switch or a hidden tab, and no longer re-arms the full VACUUM every prune. Default-on with an opt-out kill switch (localStorage['xnet:compact:changes']='off'); the emergency default-off "Hotfix A" is unnecessary now that the pass is boot-safe.runWhenBootSettled(they shared the identical flaw).@xnetjs/runtime) only re-offers on a genuine partial rollback (0 < highWaterMark < cursor) and only while the outbound breaker is not already halted — no unboundedgetChangesSince(0)flood into a reset/skewed hub.Validation
New/updated unit tests stand in for each validation-checklist item:
change-log-compaction.test.ts(new): no prune before first paint; small-chunk loop-until-dry; kill-switch and hidden-tab bail; memory/no-floor skips.boot-timeline.test.ts:bootSettled()stays pending untilquery:first-rows, resolves for late awaiters.db-vacuum.test.ts/presence-blob-cleanup.test.ts: run only after the boot gate releases.node-store-sync-provider.test.ts: no re-offer to ahighWaterMark 0hub or while the breaker is halted; partial rollback still re-offers.pnpm typecheck(91/91),pnpm lint(0 errors), fullapps/web/src/libsuite (214 passing) and the runtime sync provider suite (22 passing) all green locally.Implements
docs/explorations/0260_[x]_COMPACTION_STARVES_THE_COLD_OPEN_SCHEDULE_IT_OFF_THE_BOOT_PATH.md.🤖 Generated with Claude Code