fix(sqlite): retry OPFS-SAHPool instead of silently falling to in-memory - #202
Merged
Conversation
…memory
Root cause of "the app renders nothing until the hub dot goes green": on a
page reload the new worker's installOpfsSAHPoolVfs runs before the previous
worker has released its OPFS sync access handles, so it throws
NoModificationAllowedError ("Access Handles cannot be created..."). The
adapter caught this once and silently fell back to an in-memory database —
so nothing persisted across reloads and the workspace looked empty until the
hub re-synced (getStorageMode() === 'memory').
Fix: withOpfsLockRetry retries only the SAH contention error with linear
backoff (5×, 150ms→600ms) so the previous worker can release its handles and
we stay on durable OPFS. Non-lock failures (OPFS unsupported, private window)
still fall through immediately. When we do land in-memory, emit one loud,
actionable console.error (close other tabs + reload) instead of a quiet warn
buried in init noise.
Retry/detection logic is extracted to opfs-retry.ts and unit-tested (8 tests);
web.ts WASM/OPFS paths can't run under node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Preview removed for PR #202. |
crs48
added a commit
that referenced
this pull request
Jun 19, 2026
…208) Implements [`docs/explorations/0206`](docs/explorations/0206_%5Bx%5D_SYNC_FLOOD_RATE_LIMIT_AND_CHANGE_LOG_REPLAY.md). ## Root cause After the OPFS fix (#202) made the local DB durable, a debug log showed the client publishing **thousands** of `node-change` messages on every load → `WebSocket closed, code: 1008 reason: Rate limit exceeded` → reconnect → re-flood. `NodeStoreSyncProvider.lastSyncedLamport` was an **in-memory cursor reset to 0 on every load**, so `syncLocalChanges()` replayed the **entire change log** (`getChangesSince(0)`), one WebSocket message per change. The OPFS fix is what exposed it — before, the in-memory DB was empty each load, hiding the replay. ## What shipped (doc order A → C → throttle → D → E) - **A — persisted cursor.** `getSyncCursor`/`setSyncCursor` in `sync_state` (key `nodeSync:hwm:<room>`, monotonic). The provider loads it on connect and advances/persists it **only** from the hub's `node-sync-response` `highWaterMark` (the durable confirmation); an in-memory `pushedThrough` handles within-session dedup. - **C — request-sync-first.** On connect we send `node-sync-request` and wait for the hub's response (4 s timeout fallback) before pushing, so when the hub is already ahead we push **nothing**. - **Throttle.** Outbound node-changes go through a rate-limited queue (≤40 / 1000 ms, deduped by hash) so even a genuine first-ever backlog can't exceed the hub's 100 msg/sec limit. The queue is dropped on disconnect and rebuilt from `pushedThrough` on reconnect (no loss, no double-send). - **D — 1008 backpressure.** A `1008` close now triggers a long jittered backoff (`rateLimitBackoffMs`, default 15 s) instead of the normal short cadence, so a rate-limit close can't drive a reconnect-and-reflood storm. - **E — robustness.** `deserializeChange` (client + hub) falls back to the redundant top-level `schemaId`; `applyRemoteChanges` (and the single-change path) skip+warn on one un-appliable change instead of aborting the batch; the client now handles `node-error` (log, don't re-flood). ## Tests 24 new/updated unit tests across `node-store-sync-provider`, `connection-manager`, `sqlite-adapter`, and `store`. Full `runtime` (112), data store (195), and hub (371) suites green; typecheck/eslint/prettier clean on the changed set. ## Deferred by design (in the doc) - A new `node-sync-batch` wire type + hub support — the client-side throttle already keeps a backlog under the limit without a protocol change. - Excluding `SocialEnrichment` from sync — with the persisted cursor it syncs once (not re-replayed); dropping cross-device enrichment is a separate product call. - Optional change-log pruning. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The actual root cause behind "renders nothing until the hub dot goes green"
Console logs from a real session pinned it down — this is more severe than the 0204 exploration's primary hypotheses:
On a page reload, the new worker's
installOpfsSAHPoolVfsruns before the previous worker has released its OPFS sync access handles, so it throwsNoModificationAllowedError. The adapter caught it once and silently fell back to a:memory:database — so nothing persisted across reloads, every cold load started empty, and data could only appear once the hub re-synced. That is "lags until synced." (TheMissing SharedArrayBuffer/COOP-COEP line is a separate, benign warning — OPFS-SAHPool doesn't need SAB.)Fix
withOpfsLockRetryretries only the SAH-contention error with linear backoff (5×, 150ms→600ms) so the previous worker can release its handles and we stay on durable OPFS. Non-lock failures (OPFS unsupported, private window, quota) still fall through immediately — no added latency there.console.error(close other tabs + reload) instead of a quiet warn lost in init noise.opfs-retry.tsand unit-tested (8 tests);web.ts's WASM/OPFS paths can't run under node.Tests
8 new unit tests (
opfs-retry.test.ts); full sqlite suite (94) green;@xnetjs/sqlitetypecheck/eslint/prettier clean.Separately observed in the same log (not fixed here — follow-ups)
Error: First change for node social:enrichment:… must include schemaId— an incoming remote change whose first change lacksschemaIdthrows and aborts the rest of thatnode-sync-responsebatch. Worth making defensive (skip+warn vs throw).SQLITE_CONSTRAINT_FOREIGNKEYonyjs_stateinsert — a knock-on of the above (doc content written for a node row that never got created).Happy to take those on next; this PR fixes the confirmed cause of the empty-until-synced symptom.
🤖 Generated with Claude Code