feat: SQLite refactor by lukejmann · Pull Request #197 · worldcoin/walletkit

lukejmann · 2026-02-09T18:31:54Z

rusqlite doesn't compile for wasm32-unknown-unknown, and SQLCipher requires OpenSSL which added build complexity. sqlite3mc solves both: it compiles to WASM via sqlite-wasm-rs, and its crypto is fully built-in.

Replace rusqlite and SQLCipher with a custom safe SQLite wrapper backed by sqlite3mc on all platforms
Both libsqlite3-sys and sqlite-wasm-rs expose the same ~20 SQLite C API functions, so one thin safe wrapper (storage/db/) handles both via a cfg-switched FFI module
All unsafe and C types are confined to ffi.rs
Native compiles the sqlite3mc amalgamation via cc in build.rs; WASM uses sqlite-wasm-rs
Both platforms use ChaCha20-Poly1305 with the same PRAGMA key syntax and database format
File locking uses flock/LockFileEx on native, no-op on WASM
UniFFI attributes are gated with cfg_attr so the crate compiles for WASM

sqlite3mc
sqlite3mc is a modified SQLite amalgamation that adds transparent page-level encryption. It ships as a single C file (sqlite3mc_amalgamation.c) that replaces the standard sqlite3.c -- no external libraries needed.
Encryption is handled at the pager layer. When a database is keyed with PRAGMA key, sqlite3mc derives a page encryption key from the provided material using PBKDF2-SHA256, then encrypts every database page on write and decrypts on read using ChaCha20-Poly1305 (the default cipher). Each page gets its own one-time key derived from the master key, the page number, and a 16-byte nonce, with a Poly1305 authentication tag appended for tamper detection. This is fully transparent to the SQL layer -- queries, transactions, and WAL journaling work identically to an unencrypted database.
sqlite3mc supports multiple cipher schemes (AES-128/256-CBC, SQLCipher-compatible AES-256, RC4, Ascon-128) selectable at runtime via PRAGMA cipher. We default to ChaCha20-Poly1305 via the CODEC_TYPE=CODEC_TYPE_CHACHA20 compile flag. All crypto primitives are compiled directly into the amalgamation.

Closes #184

Note

High Risk
Large refactor of encrypted on-disk storage and transaction/query code, plus new build-time SQLite FFI/crypto plumbing; regressions could impact data integrity, keying correctness, or runtime portability across targets.

Overview
Migrates credential storage (vault + cache) off rusqlite/SQLCipher onto a new internal walletkit-db crate that provides a minimal safe SQLite wrapper backed by sqlite3mc, enabling encrypted storage without OpenSSL and making the storage stack compile on wasm32.

Storage open/key/config/integrity-check logic is centralized in walletkit_db::cipher (WAL + synchronous=FULL + secure_delete=ON), and all vault/cache SQL execution paths are updated to the new statement/transaction APIs (including explicit transactional helpers for replay-guard idempotency).

Adds WASM-specific behavior: UniFFI exports/derives are cfg_attr-gated, and the cross-process storage lock becomes a native file lock with a WASM no-op implementation; intermediate keys are now passed as Zeroizing<[u8;32]> references to reduce key material lifetime/copies.

^{Written by Cursor Bugbot for commit 71bba79. This will update automatically on new commits. Configure here.}

Dzejkop · 2026-02-10T08:34:44Z

+            )
+        })?;
+
+    k_intermediate.zeroize();


This won't be called if we error out earlier, right? Consider putting k_intermediate into a Zeroizing wrapper so it zeroes on drop

Also under the hood of execute_batch the pragma statement below which contains the key hex repr will be cloned into a CString https://github.com/worldcoin/walletkit/blob/sqlite-refactor/walletkit-core/src/storage/db/ffi.rs#L108-L108 which is not zeroed on drop

Maybe https://github.com/worldcoin/walletkit/blob/sqlite-refactor/walletkit-core/src/storage/keys.rs#L61-L65 should return Zeroizing<[u8; 32]>?

updated – see bb4b868

Dzejkop · 2026-02-10T09:04:20Z

+
+    /// Opens an in-memory database (useful for tests).
+    #[allow(dead_code)]
+    pub fn open_in_memory() -> DbResult<Self> {


Test only? Can we move it into its own impl in #[cfg(test)]?

updated – de1608a

Dzejkop · 2026-02-10T09:12:35Z

@@ -8,7 +8,7 @@ unknown-registry = "deny"
 [bans]
 deny = [
    # openssl-sys is allowed to bundle it in Android apps which require it for sqlcipher


this comment seems to be outdated

Dzejkop · 2026-02-10T09:16:46Z

+    use std::path::PathBuf;
+    let path = PathBuf::from(format!(
+        "{}/walletkit-db-cipher-test-{}.sqlite",
+        std::env::temp_dir().display(),
+        uuid::Uuid::new_v4()
+    ));


Consider using https://crates.io/crates/tempfile

Dzejkop · 2026-02-10T09:34:30Z

+    k_intermediate: [u8; 32],
+    read_only: bool,
+) -> DbResult<Connection> {
+    let conn = Connection::open(path, read_only)?;


Consider adding PRAGMA SECURE_DELETE: https://www.sqlite.org/pragma.html#pragma_secure_delete

Doesn't seem like we'd make use of it now but could be relevant in the future if we ever start deleting data

updated bf5cfe2

Dzejkop · 2026-02-10T09:37:02Z

+pub(super) struct RawStmt {
+    ptr: *mut c_void,
+    /// Kept to extract error messages via `sqlite3_errmsg`.
+    db: *mut c_void,


Consider changing this to &'db RawDb to ensure statements don't outlive the db

updated – cfa102a

Dzejkop · 2026-02-10T10:39:36Z

-        SqlcipherError::CipherUnavailable => StorageError::CacheDb(err.to_string()),
-    }
+/// Maps an owned database error into a cache storage error.
+pub(super) fn map_db_err_owned(err: &DbError) -> StorageError {


Same as map_db_err? Consider removing this one?

updated – 7b5f36b

Dzejkop · 2026-02-10T10:50:21Z

@@ -0,0 +1,33 @@
+//! Minimal safe `SQLite` wrapper backed by `sqlite3mc`.


This should be an internal crate and not a module IMO

agreed – updated 328ef59

Dzejkop · 2026-02-10T10:59:19Z

+
+/// Convenience macro for building parameter lists.
+///
+/// Usage: `params![1_i64, blob.as_slice(), "text"]`


I noticed that params! is often used with explicit variants, e.g.:

https://github.com/worldcoin/walletkit/blob/sqlite-refactor/walletkit-core/src/storage/vault/mod.rs#L77-L81

updated: b557cc3

Dzejkop · 2026-02-10T11:01:19Z

+
+impl From<&str> for Value {
+    fn from(v: &str) -> Self {
+        Self::Text(v.to_string())


We allocate and copy string here only for it to be immediately taken as a reference and sent to sqlite copying it again

https://github.com/worldcoin/walletkit/blob/sqlite-refactor/walletkit-core/src/storage/db/ffi.rs#L253-L266

I think we should consider adding a Cow and track lifetimes to avoid copying/allocation

Actually on second check this From is never utilizied. In fact it looks like Value::Text is never constructed.

I guess let's leave it as is

Dzejkop · 2026-02-10T11:30:53Z

+    }
+
+    pub fn column_blob(&self, col: i32) -> Vec<u8> {
+        // Safety: blob pointer is valid until the next step/reset/finalize.


So this isn't actually enforced by the interface - so if we're being pedantic this means that column_blob should be unsafe.

I'm wondering if it's worth it to modify Statement::step to return some sort of exclusive guard. This would allow us to:

Ensure that column_xxx methods are only called after step returns SQLITE_ROW

Enforce this safety constraint

agreed added in cf12d41

Dzejkop · 2026-02-10T11:33:23Z

+}
+
+fn errmsg_from_ptr(db: *mut c_void) -> String {
+    // Safety: db is a valid sqlite3 handle (or null, which we check).


This is actually not true, because RawStmt can outlive RawDb at which point the db pointer will be invalid, related to #197 (comment)

…I layer

…onnection

Co-authored-by: Jakub Trąd <jakubtrad@gmail.com>

Resolve conflicts keeping walletkit-db in place of rusqlite: - Cargo.toml: keep walletkit-db workspace member - walletkit-core/Cargo.toml: walletkit-db dep instead of rusqlite, restore rand as optional with dev-dep - vault/mod.rs: walletkit-db imports, rewrite fetch_credential_and_blinding_factor to use prepare/bind/step API Co-authored-by: Cursor <cursoragent@cursor.com>

socket-security · 2026-02-18T01:11:21Z

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

lukejmann · 2026-02-18T01:13:19Z

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert (click "▶" to expand/collapse)
Warn
License policy violation: cargo unicode-ident under Unicode-3.0
License: Unicode-3.0 - the applicable license policy does not allow this license (4) (unicode-ident-1.0.22/Cargo.toml)

License: Unicode-3.0 - the applicable license policy does not allow this license (4) (unicode-ident-1.0.22/Cargo.toml)

License: Unicode-3.0 - the applicable license policy does not allow this license (4) (unicode-ident-1.0.22/LICENSE-UNICODE)

From: ? → cargo/reqwest@0.12.28 → cargo/tempfile@3.24.0 → cargo/ruint@1.17.2 → cargo/tokio@1.49.0 → cargo/rustls@0.23.36 → cargo/serde_json@1.0.149 → cargo/uniffi@0.31.0 → cargo/chrono@0.4.43 → cargo/sqlite-wasm-rs@0.5.2 → cargo/thiserror@2.0.18 → cargo/cc@1.2.54 → cargo/uuid@1.20.0 → cargo/alloy-core@1.5.3 → cargo/taceo-oprf@0.5.0 → cargo/alloy-primitives@1.5.6 → cargo/alloy@1.7.1 → cargo/sha2@0.10.9 → cargo/rand@0.8.5 → cargo/ciborium@0.2.2 → cargo/hkdf@0.12.4 → cargo/ctor@0.2.9 → cargo/strum@0.27.2 → cargo/chacha20poly1305@0.10.1 → cargo/semaphore-rs@0.5.0 → cargo/serde@1.0.228 → cargo/zeroize@1.8.2 → cargo/tracing-subscriber@0.3.22 → cargo/mockito@1.7.1 → cargo/unicode-ident@1.0.22

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore cargo/unicode-ident@1.0.22`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/worldcoin/diff-scan/210d9691-6338-42f6-b855-487610fca762?tab=alerts&alert_item_key=QYdTwzkq7Lt37zGw7KmAvY_vf58EVpHhZfF_TaJPxqT0).
Warn
License policy violation: cargo webpki-roots under CDLA-Permissive-2.0
License: CDLA-Permissive-2.0 - the applicable license policy does not allow this license (4) (webpki-roots-1.0.5/LICENSE)

License: CDLA-Permissive-2.0 - the applicable license policy does not allow this license (4) (webpki-roots-1.0.5/Cargo.toml)

From: ? → cargo/reqwest@0.12.28 → cargo/taceo-oprf@0.5.0 → cargo/alloy@1.7.1 → cargo/webpki-roots@1.0.5

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore cargo/webpki-roots@1.0.5`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/worldcoin/diff-scan/210d9691-6338-42f6-b855-487610fca762?tab=alerts&alert_item_key=QZd47fqnADRLFKxdEjbGgx8r5r47R3dNHmAddPqAj_0I).
View full report

@SocketSecurity ignore cargo/unicode-ident@1.0.22
@SocketSecurity ignore cargo/webpki-roots

Dzejkop · 2026-02-19T12:02:45Z

        )?;
        let vault =
-            VaultDb::new(&self.paths.vault_db_path(), keys.intermediate_key(), &guard)?;
+            VaultDb::new(&self.paths.vault_db_path(), *keys.intermediate_key(), &guard)?;


keys.intermediate_key() returns Zeroizing<[u8; 32]>`

VaultDb::new(...) takes in [u8; 32] and then passes that to cipher::open_encrypted which https://github.com/worldcoin/walletkit/blob/sqlite-refactor/walletkit-db/src/cipher.rs#L53-L53

immediately puts the [u8; 32] into a Zeroizing<[u8; 32]>

could we just pass a &Zeroizing<[u8; 32]> reference to VaultDb::new and then transitively to open_encrypted? Same for CacheDb::new.

good catch – updated in b96745f

* Harden sqlite unsafe boundaries and statement state API * Drop defensive sqlite open null-pointer success check

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

cursor · 2026-02-25T18:59:48Z

+    /// See [`Connection::prepare`].
+    pub fn prepare(&self, sql: &str) -> DbResult<Statement<'_>> {
+        self.conn.prepare(sql)
+    }


Transaction missing query_row_optional forces code duplication

Low Severity

Transaction delegates execute_batch, execute, query_row, and prepare to Connection but omits query_row_optional. This forces get_cache_entry_tx in cache/util.rs to manually reimplement the prepare → bind → step → match pattern that Connection::query_row_optional already encapsulates. Adding a one-line delegation (self.conn.query_row_optional(sql, params, mapper)) to Transaction would eliminate ~30 lines of duplicated logic and reduce divergence risk.

Additional Locations (1)

walletkit-core/src/storage/cache/util.rs#L175-L206

lukejmann added 2 commits February 9, 2026 10:14

remove rusqlite

dc81c11

refactor RawDb, add uniffi and wasm flags

1f07c0a

lukejmann changed the title ~~Sqlite-refactor~~ SQLite refactor Feb 9, 2026

lukejmann added 3 commits February 9, 2026 10:42

source sqlite3mc in build.rs

1bac70e

remove extra file

28521a4

clippy

0fc6929

lukejmann changed the title ~~SQLite refactor~~ feat: SQLite refactor Feb 9, 2026

lukejmann added 3 commits February 9, 2026 11:23

fmt

7131888

revert deny.toml

3c2f028

fix clippy

4276eae

Dzejkop reviewed Feb 10, 2026

View reviewed changes

Comment thread walletkit-db/src/ffi.rs

Dzejkop reviewed Feb 10, 2026

View reviewed changes

Comment thread walletkit-db/src/ffi.rs

Dzejkop reviewed Feb 10, 2026

View reviewed changes

lukejmann added 6 commits February 17, 2026 10:45

zeroize key material on all exit paths and scrub CString copies in FF…

bb4b868

…I layer

move open_in_memory into #[cfg(test)] impl block

de1608a

remove old deny.toml comment

88e8ca6

use tempfile

1b6eb54

use PRAGMA secure_delete

bf5cfe2

tie RawStmt lifetime to RawDb to prevent statements outliving their c…

cfa102a

…onnection

lukejmann and others added 8 commits February 17, 2026 11:21

remove duplicate map_db_err_owned in favor of map_db_err

7b5f36b

extract walletkit-db

328ef59

simplify params! callsites to rely on Value::from conversions

b557cc3

Update walletkit-db/src/ffi.rs

857a1d5

Co-authored-by: Jakub Trąd <jakubtrad@gmail.com>

add StepResult row guard

cf12d41

Update walletkit-db/src/ffi.rs

99f590b

Co-authored-by: Jakub Trąd <jakubtrad@gmail.com>

add errmsg_from_ptr safety comment

a123418

Dzejkop reviewed Feb 19, 2026

View reviewed changes

paolodamico mentioned this pull request Feb 19, 2026

Remove vendored openssl #184

Closed

Dzejkop and others added 4 commits February 23, 2026 12:36

Codex/ffi unsafe hardening (#220)

faa4bb1

* Harden sqlite unsafe boundaries and statement state API * Drop defensive sqlite open null-pointer success check

pass K_intermediate as &Zeroizing<[u8; 32]> end-to-end

b96745f

Merge remote-tracking branch 'origin/main' into sqlite-refactor

b1b4032

fmt

2fbf39e

Dzejkop mentioned this pull request Feb 25, 2026

fix(ci): mark flaky staging integration test as #[ignore] #247

Closed

Merge main into sqlite-refactor

71bba79

Dzejkop approved these changes Feb 25, 2026

View reviewed changes

cursor Bot reviewed Feb 25, 2026

View reviewed changes

murph approved these changes Feb 25, 2026

View reviewed changes

Dzejkop merged commit 229816a into main Feb 25, 2026
16 checks passed

Dzejkop deleted the sqlite-refactor branch February 25, 2026 19:55

This was referenced Mar 2, 2026

WalletKit Release 0.7.0 #259

Closed

WalletKit Release 0.6.6 #262

Closed

WalletKit Release 0.7.0 #263

Merged

WalletKit Release 0.7.1 #266

Merged

WalletKit Release 0.7.2 #269

Merged

This was referenced Mar 11, 2026

chore: release v0.8.0 #287

Closed

chore: release v0.9.0 #289

Merged

		@@ -0,0 +1,33 @@
		//! Minimal safe `SQLite` wrapper backed by `sqlite3mc`.

Conversation

lukejmann commented Feb 9, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dzejkop Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dzejkop Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

socket-security Bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lukejmann commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Feb 25, 2026

Choose a reason for hiding this comment

Transaction missing query_row_optional forces code duplication

Uh oh!

Uh oh!

Reviewers

lukejmann commented Feb 9, 2026 •

edited by cursor Bot

Loading

Dzejkop Feb 10, 2026 •

edited

Loading

Dzejkop Feb 10, 2026 •

edited

Loading

socket-security Bot commented Feb 18, 2026 •

edited

Loading

Transaction missing `query_row_optional` forces code duplication