feat: kv-store sqlite backend with page level encryption by mverzilli · Pull Request #22759 · AztecProtocol/aztec-packages

mverzilli · 2026-04-24T11:57:58Z

Replaces the sqlite3 wasm version we were using with a sqlite3mc wasm (mc -> multiple ciphers) that supports page level encryption via pragmas.

Some potential discussion points:

Uses ChaCha20 instead of AES due to performance issues of software implementations of the latter
Vendors the binary and glue code for sqlite3mc from upstream repo https://github.com/utelle/SQLite3MultipleCiphers. We include a vendor.sh script to verify integrity of the vendored files against SHA hashes. This is a quick way of providing an initial version. If we decide to stick to this solution, we might want to instead setup our own build pipeline, building upstream from source and publishing the results on npm.

Please don't be too scared by the line-diff ~46,000 out of the ~46,900 lines in this changeset are a consequence of vendoring.

…t, add tsBuildInfoFile Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replaces the placeholder src/index.ts with a real re-export of sqlite3mc's bundler-friendly ES module default and mirrors @sqlite.org/sqlite-wasm types. Adds a .d.mts companion for NodeNext module resolution and widens tsconfig rootDir to '.' so vendor/ is reachable without crossing the rootDir boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Remove @sqlite.org/sqlite-wasm from runtime dependencies; add @aztec/sqlite3mc-wasm (workspace:^) - Move @sqlite.org/sqlite-wasm to devDependencies (types only) - Update worker.ts import to @aztec/sqlite3mc-wasm - Fix TS narrowing of module-level `sqlite3` variable after ??= across await points by capturing the result in a local const - Fix pool! non-null assertion in ensurePool return (TS loses narrowing of module-level vars across await checkpoints) - Add types field to sqlite3mc-wasm exports map so TS resolves declarations at dest/src/index.d.ts correctly - Add ../sqlite3mc-wasm project reference to kv-store tsconfig.json All 70 browser tests pass (2 skipped/benchmarks). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…FSStore.open Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…phemeral+key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…terminate worker on init failure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…rong-key, no-key, ephemeral-rejected) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…array/multi_map Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds src/bench/sqlite-opfs-encrypted/map_bench.test.ts, an encrypted persistent-OPFS store bench that runs the same shared_map_bench workload as the plaintext sqlite-opfs variant. Wired into vitest.config.ts include glob. Set VITE_BENCH=1 to run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

socket-security · 2026-04-24T12:00:27Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/@aztec/sqlite3mc-wasm@0.0.0-use.local

View full report

…ne-store invariant postMessage previously structured-cloned the key into the worker, leaving an identical copy on the main thread until GC. Transfer the ArrayBuffer instead — detaches on the caller side, so the default path (one open(), one consumption) leaves zero key bytes on the main thread after postMessage returns. This intentionally surfaces key duplication: callers that want the same key in multiple stores must explicitly clone (new Uint8Array(savedKey)) per call. The clone is a visible, deliberate decision instead of an invisible structured- clone side effect. Tests in encrypted_store.test.ts add a cloneKey() helper for the reopen-same-db scenarios, where the test naturally needs two consumptions of the same key.

kv-store no longer imports @sqlite.org/sqlite-wasm directly — worker.ts imports sqlite3InitModule and the Database/SAHPoolUtil/Sqlite3Static types from @aztec/sqlite3mc-wasm, which re-exports them. The transitive type resolution works via workspace hoisting: @aztec/sqlite3mc-wasm keeps the devDep for its own type re-exports, and Yarn hoists it to yarn-project's root node_modules where kv-store's tsc walker finds it.

…cible vendor.sh script Adds provenance metadata (source URL, zip name, SHA256) to the package README so reviewers can independently verify the vendored binary matches the upstream release. Adds scripts/vendor.sh that downloads a given sqlite3mc release, hashes the zip against a caller-supplied expected SHA256, and refuses to extract on mismatch — making re-vendoring reproducible and tamper-evident. Closes the binary-lineage audit gap identified during PR review: previously, the vendored WASM's only integrity anchor was the single curl command in the plan document, with no post-merge way for a reviewer to verify.

… verification chain Review caught that 10 of the 11 vendored files had drifted from upstream — pre-commit prettier was reformatting the JS/MJS glue files on commit. The whitespace changes are semantically harmless (same runtime behavior) but cryptographically meaningful: the vendored bytes no longer matched upstream, breaking any hash-based integrity verification. Changes: 1. .prettierignore: add sqlite3mc-wasm/vendor/ so pre-commit hooks don't re-format the upstream files. 2. vendor/jswasm/: restore all 10 upstream .js/.mjs/.wasm files to their pristine release bytes (verified byte-for-byte against the v2.2.4 zip). 3. vendor/jswasm/SHA256SUMS: new file, per-file hashes for every file in vendor/jswasm/ including the 10 upstream files and the 1 locally-authored .d.mts. Enables 'sha256sum -c SHA256SUMS' for a fast local integrity check. 4. scripts/vendor.sh: now preserves our locally-authored .d.mts across re-vendoring and regenerates SHA256SUMS from the final file set. 5. README.md: documents the full verification chain a reviewer can walk: zip -> extracted files -> SHA256SUMS -> repo state. Calls out which file is locally authored and why. Closes an audit gap: previously the README's single zip-level SHA256 was the only integrity anchor, and it covered neither the per-file bytes nor the post-extraction state.

CI run cc2e1a64158b6f02 failed at the lint step with "ESLint couldn't find an eslint.config.(js|mjs|cjs) file" because the new package shipped without one. Mirrors kv-store's flat config (re-export foundation's config) and adds an explicit vendor/** ignore as defense- in-depth for any future tool that walks the package tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Thunkar

Not gonna block, but have you considered pulling the vendored files during build and not committing them to the repo?

mverzilli · 2026-05-04T08:11:29Z

Not gonna block, but have you considered pulling the vendored files during build and not committing them to the repo?

Yes, it makes me a little nervous to have online external dependencies, but other than that I would be fine with it. What stance do we usually take as an org in that case?

AztecBot · 2026-05-04T12:15:46Z

❌ Failed to cherry-pick to v4-next due to conflicts. (🤖) View backport run.

Cherry-pick of #22759 with conflict markers preserved in worker.ts. Conflict resolution and build fixes follow in subsequent commits. Original PR: #22759

Combines both sides of the merge in ensurePool(): - Keeps poolDirectory tracking from #22721 (kv-store browser test hangs fix) - Adopts the local 's = sqlite3' alias and sqlite3mc_vfs_create call from #22759 The handleDeleteDb codepath inherited from #22721 (async + ensurePool with remembered poolDirectory) is preserved as-is and is orthogonal to the encryption wrapper, which only kicks in via MC_SAH_POOL_VFS_NAME in handleInit.

## Summary Backport of #22759 (`feat: kv-store sqlite backend with page level encryption`) to `backport-to-v4-next-staging`. The auto-cherry-pick failed on `yarn-project/kv-store/src/sqlite-opfs/worker.ts`. The conflict was localized to the `ensurePool` function: the v4-next branch already contains #22721 (kv-store browser test hangs fix), which added a `poolDirectory = directory;` assignment, while #22759 introduced a local `s = sqlite3` alias and the sqlite3mc VFS registration call. Both changes are compatible and have been combined. ## Commit structure 1. `1944712b` — original cherry-pick committed as-is with the conflict marker preserved in `worker.ts`, so reviewers can see exactly what conflicted. 2. `0c397e2c` — conflict resolution: keeps `poolDirectory` tracking from #22721 + the local `s` alias and `sqlite3mc_vfs_create` call from #22759. No build-fixes commit was needed — `kv-store` and `sqlite3mc-wasm` both typecheck clean against the resolved tree. ## Verification - `yarn install` succeeded. - `tsgo --noEmit` is clean for both `kv-store` and `sqlite3mc-wasm`. - Diff stat matches the original PR: 26 files, +699/-14. ## Original PR - #22759 - Merge commit: `97c6e48fe9bf269d12fa0640e4e9303f5e7cbae5` ClaudeBox log: https://claudebox.work/s/6446d1f92380c25a?run=1 ClaudeBox log: https://claudebox.work/s/6446d1f92380c25a?run=1

…#22929) Follow up from #22759, this exclusion is needed for sqlite browser tests to boot properly (already applied on backport) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

BEGIN_COMMIT_OVERRIDE chore: fix kv-store browser tests hangs (AztecProtocol#22721) feat: kv-store sqlite backend with page level encryption (AztecProtocol#22759) fix: install node 22 for aztec-cli acceptance test (AztecProtocol#22917) fix(kv-store): exclude @aztec/sqlite3mc-wasm from vitest optimizeDeps (AztecProtocol#22929) feat(txe): add tx private logs to tx side effects oracle (AztecProtocol#22889) feat(aztec-nr): add call_self stubs for utility functions (AztecProtocol#22885) END_COMMIT_OVERRIDE

BEGIN_COMMIT_OVERRIDE docs: add map and state variable docs (#22824) fix: e2e compat should not fail for contracts added after legacy stables (#22900) chore: fix kv-store browser tests hangs (#22721) feat: kv-store sqlite backend with page level encryption (#22759) fix: install node 22 for aztec-cli acceptance test (#22917) feat: backport kv-store sqlite encryption (#22759) to v4-next (#22927) fix(docs): correct llms.txt links for versioned developer docs (#22819) feat(docs): improve discoverability of Aztec.nr API reference docs (#22861) feat(docs): backport improve discoverability of Aztec.nr API reference docs (#22861) to v4-next (#22931) feat(aztec-nr): add call_self stubs for utility functions (#22885) docs: add map and state variable docs (backport #22824) (#22880) refactor: `getPackageVersion` fn cleanup (#22941) fix(ci): skip acceptance test for canary -commit. tags (#22951) fix: closing db, correct stub side effects (#22939) END_COMMIT_OVERRIDE

Had been accidentally introduced in #22759

mverzilli and others added 14 commits April 24, 2026 10:25

chore(sqlite3mc-wasm): create workspace package skeleton

6115efd

fix(sqlite3mc-wasm): register in workspaces, point root export to des…

f49cf9a

…t, add tsBuildInfoFile Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore(sqlite3mc-wasm): vendor sqlite3mc 2.2.4 WASM release artifacts

cce34cf

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(kv-store): add optional encryptionKey to worker init message

e2fe0eb

feat(kv-store): add optional encryptionKey parameter to AztecSQLiteOP…

78c6965

…FSStore.open Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(kv-store): wire encryption key to sqlite3mc via PRAGMA cipher/key

1f9e0fc

fix(kv-store): open encrypted DBs via sqlite3mc-wrapped VFS; reject e…

fb2e837

…phemeral+key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(kv-store): reject encryptionKey with ephemeral at the boundary; …

605b434

…terminate worker on init failure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(kv-store): page-level encryption tests (persistent round-trip, w…

711dcdb

…rong-key, no-key, ephemeral-rejected) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(kv-store): encrypted-store smoke tests across map/set/singleton/…

bfe5143

…array/multi_map Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(kv-store): export openEncryptedStore helper from sqlite-opfs barrel

4304dac

mverzilli requested a review from nventuro as a code owner April 24, 2026 11:57

mverzilli changed the base branch from next to merge-train/fairies April 24, 2026 11:58

mverzilli removed the request for review from nventuro April 24, 2026 11:58

mverzilli mentioned this pull request Apr 24, 2026

feat: sqlite with page level encryption Thunkar/gregojuice#22

Open

mverzilli force-pushed the martin/sqlite-page-level-encryption branch from 662f5e6 to 4304dac Compare April 24, 2026 14:23

mverzilli added 4 commits April 24, 2026 15:44

mverzilli requested a review from Thunkar April 24, 2026 16:50

mverzilli mentioned this pull request Apr 28, 2026

feat(kv-store): encryption-at-rest for SQLite-OPFS + PXE opaque-keys (DO NOT MERGE, PROOF OF CONCEPT) #22683

Closed

6 tasks

Thunkar approved these changes May 4, 2026

View reviewed changes

vendor on build instead of on repo

dae51ea

mverzilli added the backport-to-v4-next label May 4, 2026

mverzilli merged commit 97c6e48 into merge-train/fairies May 4, 2026
21 of 25 checks passed

mverzilli deleted the martin/sqlite-page-level-encryption branch May 4, 2026 12:14

AztecBot mentioned this pull request May 4, 2026

feat: merge-train/fairies #22923

Merged

AztecBot mentioned this pull request May 4, 2026

feat: backport kv-store sqlite encryption (#22759) to v4-next #22927

Merged

mverzilli mentioned this pull request May 4, 2026

fix(kv-store): exclude @aztec/sqlite3mc-wasm from vitest optimizeDeps #22929

Merged

AztecBot mentioned this pull request May 4, 2026

chore: Accumulated backports to v4-next #22924

Merged

spalladino mentioned this pull request May 6, 2026

chore: remove top-level yarn.lock #22987

Merged

PhilWindle pushed a commit that referenced this pull request May 6, 2026

chore: remove top-level yarn.lock (#22987)

5f8441e

Had been accidentally introduced in #22759

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: kv-store sqlite backend with page level encryption#22759

feat: kv-store sqlite backend with page level encryption#22759
mverzilli merged 20 commits intomerge-train/fairiesfrom
martin/sqlite-page-level-encryption

mverzilli commented Apr 24, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

Thunkar left a comment

Uh oh!

mverzilli commented May 4, 2026

Uh oh!

Uh oh!

AztecBot commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mverzilli commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thunkar left a comment

Choose a reason for hiding this comment

Uh oh!

mverzilli commented May 4, 2026

Uh oh!

Uh oh!

AztecBot commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mverzilli commented Apr 24, 2026 •

edited

Loading

socket-security Bot commented Apr 24, 2026 •

edited

Loading