feat!: kv-store on SQLite-wasm over OPFS#22658
Merged
mverzilli merged 10 commits intomerge-train/fairiesfrom Apr 20, 2026
Merged
feat!: kv-store on SQLite-wasm over OPFS#22658mverzilli merged 10 commits intomerge-train/fairiesfrom
mverzilli merged 10 commits intomerge-train/fairiesfrom
Conversation
First phase of the PXE storage encryption work: a new browser-targeted kv-store backend that persists to OPFS via @sqlite.org/sqlite-wasm's opfs-sahpool VFS. Mirrors the indexeddb backend's schema and semantics (including sparse-multi-map keyCount) so consumer code is backend-agnostic. - Typed RPC over a dedicated Worker since opfs-sahpool requires Worker context - SerialQueue + #inTx flag to serialize transactions without deadlocking on nested container ops inside transactionAsync - ordered-binary key encoding + msgpackr value encoding (same as lmdb-v2) - Range queries preserve IndexedDB's forward/reverse inclusivity asymmetry - Browser-only: excluded from node mocha runs, added to vitest include set No encryption yet — phase 2 will add it at either the sqlite3mc level or the OPFS-VFS page-encryption level.
Two issues surfaced by the browser test suite: 1. map.ts: Buffer is not defined in the vitest browser stub unless the `buffer` module is explicitly imported (the stub's globalThis assignment runs as a side effect). Adding explicit import matches the pattern in foundation/src/crypto/aes128/index.ts. 2. store.ts: transactionAsync must join an outer transaction when called inside one. Without this, MultiMap.set (which opens its own transactionAsync for atomic hash-check + keyCount-read + insert) deadlocks when invoked via inherited Map.setIfNotExists (also wraps in transactionAsync). SQLite doesn't support nested BEGIN, and re-acquiring the SerialQueue while the outer holds it would block forever. All 131 kv-store browser tests now pass (indexeddb + sqlite-opfs combined).
…ool dir Two related changes that let multiple sqlite-opfs stores coexist in the same tab: 1. EmbeddedWalletOptions gains `walletDb.store` — parallel to the existing `pxe.store` hook — so callers can inject a custom wallet DB backend. Browser and Node entrypoints both honor it. 2. AztecSQLiteOPFSStore.open accepts a `poolDirectory` parameter forwarded to the worker's installOpfsSAHPoolVfs call. Needed because SAH Pool acquires an exclusive lock on its OPFS directory — a second store sharing the default directory would collide. Gregoswap will use one directory per store (pxe vs walletDb).
msgpackr returns plain Uint8Array in browsers for packed Buffer values,
but callers like walletDB store account type as Buffer.from(typeString)
and later call typeBuffer.toString('utf8'). Uint8Array.toString silently
ignores the encoding argument and returns comma-joined bytes instead of
the UTF-8 string, yielding errors like 'Unknown account type 115,99,104,...'
(which decodes to 'schnorr-initializerless').
Mirrors IndexedDBAztecMap.restoreBuffers — re-wrap at the storage boundary
so callers get Buffer-flavored behavior regardless of backend.
Wraps the SAH Pool's exportFile() so callers can retrieve a raw SQLite image for inspection/backup. OPFS SAH Pool stores data in opaque slot files (not regular .sqlite files), so this is the supported way to get an image that opens in any SQLite tool. Only works for non-ephemeral DBs since the pool must be initialized.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Extracts the existing LMDB-v2 map benchmark suite into a shared runner (shared_map_bench.ts) that takes any AztecAsyncKVStore and a reporter, then wires it up for three backends: - LMDB-v2 (Node, existing): writes results to BENCH_OUTPUT / BENCH_OUTPUT_MD env-gated files, unchanged behavior - IndexedDB (browser, new): runs in vitest-browser, logs to console - SQLite-OPFS (browser, new): runs in vitest-browser, logs to console Browser benches are discoverable via vitest's include pattern but self-skip unless VITE_BENCH=1, so `yarn test:browser` still returns 131 passing + 2 skipped (no new runtime cost for default runs). To run: yarn workspace @aztec/kv-store bench:browser # both browser backends yarn workspace @aztec/kv-store test:node -- --spec src/bench/map_bench.test.ts # LMDB Also bumps testTimeout to 300s when VITE_BENCH=1 since full-population + many-iteration tests exceed the default 30s.
Matches the IndexedDB backend's all-zeros return; SQLite can populate it cheaply via PRAGMAs when a caller actually needs real numbers.
Previously, close() and delete() called worker.terminate() without clearing the #pending map, leaving any in-flight #sendRequest promises hanging forever. In practice this didn't bite because those lifecycle methods run during shutdown when nothing is awaiting, but it was a real latent leak. Extracted the rejection logic into a #rejectPending(reason) helper and call it from close(), delete(), and the worker.onerror handler (which was already doing the same work inline).
CI flagged 7 ESLint issues in the sqlite-opfs backend and bench helper: - store.ts: transactionAsync/runAsync/allAsync were declared async but their bodies just returned existing promises; drop the keyword (sync throws from the body already convert to rejections via the caller's await, so behavior is unchanged). - worker.ts: handleExport genuinely needed an await on pool.exportFile (which returns Promise<Uint8Array>) — added. Also dropped the explicit printErr: console.error args from sqlite3InitModule; the underlying Emscripten default matches this behavior and avoids no-console flags. - shared_map_bench.ts: use type-only import for Logger.
Thunkar
reviewed
Apr 20, 2026
| @@ -1,7 +1,7 @@ | |||
| { | |||
Contributor
There was a problem hiding this comment.
We're not using mocha anymore, right? If so, we should delete this
Contributor
Author
There was a problem hiding this comment.
I think we still use it for the node tests, but fair observation that we could just unify it all under the more modern vitest harness
Collaborator
|
❌ Failed to cherry-pick to |
AztecBot
pushed a commit
that referenced
this pull request
Apr 20, 2026
5 tasks
mverzilli
added a commit
that referenced
this pull request
Apr 20, 2026
Backport of #22658 to v4-next. ## Original PR summary Experimental support for SQLite as a backend to kv-store, with the goal of eventually abandoning IndexedDB. ## Conflicts Single conflict in `yarn-project/kv-store/package.json` exports map: - v4-next keeps the `./config` export (still imported by archiver, aztec-node, bot, node-lib, p2p, etc.) whereas that export had been removed on `next`. - Resolved by keeping all three exports: `./sqlite-opfs`, `./stores`, and `./config`. The rest of the cherry-pick (new `sqlite-opfs/` module, wallet entrypoint wiring for `walletDb.store`, yarn.lock) applied cleanly. ## Commits 1. `cherry-pick: ...(with conflicts)` — original squash commit applied with the conflict marker preserved in `package.json`. 2. `fix: resolve cherry-pick conflicts` — keeps the `./config` export alongside the new `./sqlite-opfs` entry. No additional build-adaptation commit was required. ClaudeBox log: https://claudebox.work/s/3c079f039c55d141?run=1
dipkakwani
pushed a commit
to dipkakwani/aztec-packages
that referenced
this pull request
Apr 22, 2026
BEGIN_COMMIT_OVERRIDE feat!: kv-store on SQLite-wasm over OPFS (AztecProtocol#22658) fix(pxe): verify private event commitment matches content (AztecProtocol#22638) fix(pxe): propagate calldata count from nested private oracles (AztecProtocol#22642) END_COMMIT_OVERRIDE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experimental support for SQLite as a backend to kv-store, with the goal of eventually abandoning IndexedDB