perf(sqlite): reclaim compacted change-log pages via incremental auto-vacuum (0260) - #369
Merged
Merged
Conversation
…-vacuum (0260) The durable cold-open fix that #366 set up but deferred. Change-log compaction (0254) DELETEs superseded history, but under the default `auto_vacuum = NONE` those pages only re-enter SQLite's freelist — the OPFS file never shrinks, so the ~16 s cold read that faults the file's working set stays bloat-priced no matter how many rows are pruned. And the one-time boot VACUUM (which is the only thing that returned space to the OS) latches after a single run, so it typically rewrote a still-fat file and never ran again. Convert the working set to incremental auto-vacuum instead: - `web.ts`: open OPFS databases with `PRAGMA auto_vacuum = INCREMENTAL`. Fresh databases are incremental from birth and never bloat; the mode only converts an existing NONE database at a `VACUUM`. - `db-vacuum.ts`: the existing one-time boot-settled VACUUM now doubles as that conversion for pre-existing databases (comment only — a VACUUM applies the pending mode change automatically). - `change-log-compaction.ts`: after each prune pass, run `PRAGMA incremental_vacuum` to hand the freed pages back to the filesystem — a harmless no-op until the DB is converted, and per-boot reclaim afterwards. So the file shrinks a little every idle boot as the log drains, instead of only on the single one-time VACUUM. Proven with a better-sqlite3 semantics test (`auto-vacuum-reclaim.test.ts`): NONE → DELETE doesn't shrink and `incremental_vacuum` is a no-op; INCREMENTAL → page_count and on-disk bytes both drop; existing NONE databases convert via one VACUUM and reclaim per call thereafter. Plus compaction unit tests asserting the reclaim fires only when rows were pruned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Contributor
🖼️ UI changes in this PRNo UI changes detected in this PR. |
Contributor
|
Preview removed for PR #369. |
crs48
added a commit
that referenced
this pull request
Jul 4, 2026
…y Site (#370) ## Why the live site is still serving old code Every **Deploy Site to GitHub Pages** run since 2026-07-03 18:06 UTC has **failed** — 7 in a row — at the "Install site dependencies" step: ``` ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with package.json * 2 dependencies were added: @types/qrcode@^1.5.6, qrcode@^1.5.4 ``` PR #363 (Run on Mobile / Expo Go QR page) added `qrcode` + `@types/qrcode` to `site/package.json` without regenerating `site/pnpm-lock.yaml` (the site is a standalone `--ignore-workspace` install with its own lockfile, so the root lockfile check didn't catch it). Result: the production site — including the `/app/` PWA bundle — has been pinned at `1ed5ab82`, which predates the cold-open fixes #366 and #369. That is exactly why every cold-boot capture since still shows the pre-#366 signature (`hub high-water mark 0 … re-offering`). ## Fix Regenerate `site/pnpm-lock.yaml` (Node 23 / pnpm 10). Diff is 220 additions, zero removals: `qrcode`, `@types/qrcode`, their transitive deps (`dijkstrajs`, `pngjs`, yargs CLI chain), plus inert `libc` metadata pnpm 10 adds. ## Verified locally - `pnpm install --ignore-workspace --frozen-lockfile` in `site/` — passes (CI's exact failing command). - Full Astro site build — passes, 110 pages (the QR page has never made it past install in this workflow, so the build step was proven too). Merging touches `site/**`, which triggers the production deploy — shipping everything accumulated since #363, including #366 + #369. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 4, 2026
## The second deploy-pipeline bug (stacked on #370) After #370 unblocked Deploy Site, run `28709576907` went green — but the live app **still served pre-#366 code**. The publish commit (`e99fa1de`) tells the story: all the chunk renames landed (`index-DppD45_e.js → index-BuveDUTb.js`, `web-worker-BIYC6Nw7.js → web-worker-BFqMZy2X.js`, …) but **`app/index.html`, `app/404.html`, and `app/sw.js` are absent from the diffstat** — gh-pages ended up with a stale `index.html` pointing at a deleted entry chunk. ## Root cause `publish-gh-pages` syncs with `rsync -a`, whose quick check skips files whose **size and mtime** both match: - **Size never changes for these files.** A rebuild substitutes 8-char content hashes for 8-char content hashes, so `index.html`/`404.html`/`sw.js` are byte-for-byte the *same length* across builds. - **mtime carries no signal.** Both sides are freshly generated — `cp -R` build output (prepare step) vs a just-`git worktree add`-ed checkout (publish step) — every mtime is "now". This run's prepare-copy and checkout landed in the **same second**, so the quick check called them identical. Every prior deploy dodged this by a one-second timing margin; it's a latent race in every publish (production, PR previews, branch previews — all use this action). ## Fix `rsync -a --checksum` — decide by content, deterministically. `git add -A` already ignores genuinely unchanged files, so the only cost is reading both trees. ## After merge `.github/**` isn't in Deploy Site's path filters, so I'll `workflow_dispatch` a production deploy, then verify `gh-pages:app/index.html` references the live entry chunk and the deployed bundle carries the #366/#369 markers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 5, 2026
…um (0260) (#380) ## Problem Live capture from the newly-deployed build (post #366/#369) showed change-log compaction running but the OPFS file never shrinking. Root cause: the one-time VACUUM from exploration 0233 latched its `xnet:db-vacuumed:v1` localStorage flag on long-lived profiles **before** incremental auto-vacuum existed (#369). For those profiles: - the database is still `auto_vacuum = NONE` (the pending `INCREMENTAL` mode set at open in web.ts only applies to a fresh DB or at a VACUUM), - so every per-prune `PRAGMA incremental_vacuum` (#369) is a **silent no-op**, - and the file can never shrink, no matter how many rows compaction deletes. ## Fix Gate the one-time VACUUM on the database's **actual** `PRAGMA auto_vacuum` mode, not the flag alone: - flag latched **and** mode = INCREMENTAL (2) → skip (steady state: one idle PRAGMA read per boot) - flag latched **but** mode = NONE (0) → run the conversion VACUUM anyway (the 0233-era profile case) - no flag → vacuum + latch as before Scheduling is unchanged — still boot-settled + idle via `runWhenBootSettled` (#366), so the whole-file rewrite never races the cold-open read burst. Bonus: the conversion VACUUM immediately returns the freelist accumulated by prior prunes (~250k rows) to the OS. ## Testing - `db-vacuum.test.ts` extended: latched+INCREMENTAL skips, latched+NONE converts, unflagged vacuums+latches, memory mode skips — 4/4 pass - full `apps/web/src/lib/` suite: 217/217 pass - `pnpm --filter xnet-web typecheck` + eslint clean - full `pnpm test`: 10,020 passed (pre-push devkit git-integration failures are hook-env-only; suite is green outside the hook) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
crs48
added a commit
that referenced
this pull request
Jul 5, 2026
…page per pass (#381) ## How this was found Local end-to-end validation of the cold-open fixes (#366/#369/#380): seeded a **464 MB replica** of the user's pathological profile straight into a real browser's OPFS database (300k-row `changes` log, 250k rows deleted into the freelist → 81% dead pages, devolved to `auto_vacuum=NONE`, latched 0233-era `xnet:db-vacuumed:v1` flag), then booted the shipped code against it repeatedly. **What validated clean:** - Boot 1: #380's conversion VACUUM fired post-settle — **464 MB → 80 MB**, `auto_vacuum` 0 → 2, first paint 1823 ms → 1090 ms on the following boot. - Boot 2: vacuum correctly skipped (flag + mode 2); compaction pruned **exactly the 47,000 prunable rows** (K1/K2/K3 predicate verified by dry-run count beforehand). - Boot 3: steady state — compaction dry, vacuum skip, one idle PRAGMA read. **What didn't:** boot 2 ended with `{deleted: 47000, reclaimed: true}` — and the file still at 80 MB with 9,248 freelist pages. Direct probing showed every `exec('PRAGMA incremental_vacuum')` (with or without a page-count argument) freed **exactly one page**. ## Root cause A WASM *binding* quirk, not engine semantics: SQLite frees one freelist page per `sqlite3_step` of `PRAGMA incremental_vacuum`, and sqlite-wasm's oo1 `exec` steps a row-less statement exactly once. better-sqlite3's `pragma()` steps to completion — which is why #369's engine-level test passed while the production reclaim was a near-no-op (at one page per boot, a 75 MB backlog would take ~9,000 boots). ## Fix - `SQLiteAdapter.incrementalVacuum(maxPages?)` — new **optional** interface method returning pages freed. - Web adapter: `stepIncrementalVacuumToCompletion` steps the pragma until done (or the cap); wired through the worker (write lane) + proxy. - Electron adapter: via `db.pragma` + freelist delta (better-sqlite3 already steps fully). - `change-log-compaction.ts` prefers the method (falls back to `exec` on adapters without it), and a **dry pass now reclaims a stranded freelist backlog** (≥1024 pages ≈ 8 MiB) — healing profiles that pruned under the buggy build or bailed mid-pass on a hidden tab. ## Re-validation (same seeded profile, fixed build) - Backlog branch: first boot with the fix returned all 9,244 stranded pages — **80 MB → 8 MB** in one pass. - Integrated path: seeded 6,000 fresh prunable rows → pass reported `{deleted: 5400, freedPages: 403, reclaimed: true}` (5,400 = exact predicted count) and the file returned to baseline. ## Testing - NEW `incremental-vacuum-stepping.test.ts` — pins the one-page-per-exec bug **against the real @sqlite.org/sqlite-wasm oo1 API** and the fix (drains freelist, honours cap): 3/3. - `change-log-compaction.test.ts` extended (method preferred, exec fallback, dry-small-freelist no-op, backlog reclaim): 9/9. - Full suite 10,025 passed; typecheck 91/91; eslint clean. - Changeset: `@xnetjs/sqlite` minor. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The durable half of the cold-open fix that #366 set up but deferred (the "REMAINING" item in exploration 0260).
Change-log compaction (0254) DELETEs superseded history, but the OPFS database opened with the default
auto_vacuum = NONE, so those pages only re-entered SQLite's freelist — the file never shrank. Since the ~16 s cold-open read is the OPFS page-fault of that file's working set, pruning rows did nothing for cold-open until the file physically shrank. The only thing that returned space to the OS was the one-time boot VACUUM, which latches after a single run and so typically rewrote a still-fat file and never ran again.What
Convert the OPFS working set to incremental auto-vacuum, so freed pages are handed back to the filesystem per boot:
web.ts— open OPFS databases withPRAGMA auto_vacuum = INCREMENTAL. Fresh databases are incremental from birth (never bloat); the mode only converts an existingNONEdatabase at aVACUUM.db-vacuum.ts— the existing one-time boot-settled VACUUM now doubles as that conversion for pre-existing databases (comment only — aVACUUMapplies the pending mode change automatically).change-log-compaction.ts— after each prune pass, runPRAGMA incremental_vacuumto return the freed pages to the OS. A harmless no-op until the DB is converted; per-boot reclaim afterwards. The file shrinks a little every idle boot as the log drains, instead of only on the single one-time VACUUM.Convergence
NONE → INCREMENTALand reclaims whatever compaction has deleted so far.incremental_vacuumreclaims them — the file shrinks steadily until the ~318k-row backlog is drained (~7 boots), and it keeps reclaiming for the workspace's whole life instead of latching once.Validation
New
packages/sqlite/src/adapters/auto-vacuum-reclaim.test.tsproves the exact SQLite semantics against a real engine (better-sqlite3):NONE(default) → a DELETE does not shrink the file andincremental_vacuumis a no-op.INCREMENTALDB →incremental_vacuumdrops bothpage_countand on-disk bytes.NONEDB → oneVACUUMconverts it (and compacts), thenincremental_vacuumreclaims per call.Plus
change-log-compaction.test.tsgains cases assertingPRAGMA incremental_vacuumfires only when rows were actually pruned (not on a dry log, kill-switch, or memory DB).Local:
pnpm typecheck(91/91),pnpm lint(0 errors), sqlite unit suite (158 pass / 3 skip), apps/web lib (216 pass).Follow-up to 0260 / #366. Note: pairs with deploying #366 — that stops the compaction-on-boot regression and the reset-hub flood; this makes the pruning actually shrink the file that gates cold-open.
🤖 Generated with Claude Code