feat(db): the db module — SQLite behind a five-op spec, sim host, pocket-db reference core

[siwei-yuan]

What

The fourth module-shaped vertical slice (after ui, strike and audio), built spec-first per the RUNTIMES.md §5 discipline: SQLite mounted as globalThis.db behind a five-op, append-only spec.

• contracts/spec/db.ts pins five synchronous ops (open/close/exec/query/lastError), the JSON value encoding (blobs as {"$b": base64}, integers past 2^53−1 fail loudly instead of losing precision), logical database names the host maps under the app's own data root, and the resource ceilings (4 databases, 4096 result rows per query).
• Statement caching is host-side, keyed by the sql string — the guest holds no statement handles, so there is nothing to finalize and nothing to leak.
• No events, no clock: every op completes inside the guest's single per-tick turn (law 3 holds unchanged). SQL time and randomness resolve host-side; golden-tested apps must not depend on random() or 'now'-relative SQL — the Date.now rule applied to the dialect.
• ATTACH is refused (the one SQL statement that names a file) so the app's data root stays the sandbox boundary; load_extension stays off.
• data.sqlite capability registered ahead of any stock TARGET advertising it — the audio.pcm precedent: the sim host and the reference core implement and test the whole contract.
• @pocketjs/framework/db SDK: the bun:sqlite shape (Database, cached Statement .get/.all/.values/.run, transaction with savepoint nesting), throwing where the namespace is unmounted — data code that silently drops writes is a corruption bug, not a missing enhancement.
• hosts/sim/db.ts: bun:sqlite behind the op namespace, injected via bootWorld extraGlobals.
• engine/crates/pocket-db: the reference core over rusqlite (bundled), a real SQLite authorizer for the ATTACH refusal, Storage::Memory/Dir, mountable on any pocket-mod guest — the adoption path a device host copies, with SQLite's own VFS as the port point.
• docs/DB.md maps the boundary and the three-move adoption path.

Verification

• bun run test: 11/11 stages green (tests/db.test.ts 17 pass — op contract, SDK, and an oracle comparison running identical statements through the op namespace and bun:sqlite directly; full suite 313+ pass, all existing goldens byte-identical).
• cargo test -p pocket-db: 9/9, including a live QuickJS guest round-trip through pocket-mod.
• cargo check --workspace clean.
• Real-hardware spike (ESP32-P4 @ 360 MHz, LittleFS on flash, rusqlite bundled cross-compiled with the esp toolchain): SQLite 3.53.2 opens in 15 ms, inserts 288 rows in one transaction in ~350 ms, aggregates + a 24-row chart window in 5–9 ms, and the data survives close/reopen and a full power cycle (boot #0 wrote 288 rows; boot feat: polish PSP UI demos and rendering #1 found them and grew the table to 576). ~70 KB heap, 20 KB database file. The embedded recipe (unix-none VFS, SQLITE_OMIT_WAL, TEMP_STORE=3, lstat=stat, six newlib shims) is exactly the port-point work docs/DB.md assigns to a device host — the spec, SDK and core needed no changes.

Scope

No stock target advertises data.sqlite yet, matching how audio.pcm landed before its first console adoption. Device hosts adopt by the three moves in docs/DB.md; the ESP32-P4 spike above is the first external consumer of that path.

🤖 Generated with Claude Code

[siwei-yuan]

Pushed a second commit: ESP-IDF support in-crate, and databases as ordinary files in the app root.

• cfg(target_os = "espidf"): the newlib shims SQLite's syscall table references, plus the unix-none VFS on open (LittleFS has no fcntl locks) with flash-friendly pragmas — validated on a Waveshare ESP32-P4 over a LittleFS workspace (open 15 ms, 288-row tx ~0.4 s, ~70–80 KB heap, data intact across power cycling). The build-environment half a firmware supplies (LIBSQLITE3_FLAGS, the sys/ioctl.h shim) is documented in docs/DB.md; desktop builds see none of this.
• mount is now a default feature: default-features = false drops the pocket-mod/rquickjs dependency, so an MCU firmware with its own QuickJS wiring compiles only the module core plus SQLite — verified cargo check clean for riscv32imafc-esp-espidf, bundled libsqlite3.a included.
• Storage::Dir maps a name to <dir>/<name>.sqlite — an ordinary, visible file in the app's own data root (created on first open), the same root the fs module (feat(fs): the fs module — a per-app file tree behind a nine-op spec, sim host, pocket-fs reference core #238) binds. The database is the app's own asset: backup is a file copy, and overwriting it corrupts the app's own data — the same trust class as deleting its own files; SQLite fails loudly on a corrupt image.

On-device conformance for both data modules lives in #238's hosts/esp32p4/examples/data-smoke.

🤖 Generated with Claude Code

[doodlewind]

Verdict: approved. The module is well-built — the spec boundary is clean, the SDK is a faithful bun:sqlite shape, and the reference core's tests are real. I probed both hosts with the same binding/ATTACH matrix rather than reading the diff, found four issues, and pushed the fixes as 6dbb0f4 on this branch.

What the probe found (and the commit fixes)

1. ATTACH <expr> AS x bypassed both refusals — a real sandbox escape. rusqlite maps a NULL filename (any non-literal ATTACH argument) to AuthAction::Unknown, so an authorizer matching only AuthAction::Attach allows it, and the sim regex only matched the DATABASE-keyword and string-literal spellings. ATTACH hex('2f746d702f78') AS o sailed through both hosts and left a real file on disk in each. The core now also sets SQLITE_LIMIT_ATTACHED=0 (rusqlite limits feature) so every spelling is refused at the engine level — the authorizer stays for the clearer not authorized on the literal form — and the sim matches \battach\b anywhere (the documented false-positive trade widens accordingly). Both sides gained tests.
2. The sim was more lenient than the core on named parameters. bun binds unprefixed keys ({a: 1}) and silently ignores unknown ones ({$zzz: 2}); the core fails both with unknown parameter. An app developed on the sim would break on device. The sim now refuses unprefixed keys with the core's message; the residual leniency (a prefixed key the statement never names is ignored by bun — no parameter-name introspection to close it) is documented in the header.
3. close() lost data the device host keeps. The sim header claims named databases persist "the way a device keeps its files", but close() then reopen produced a fresh empty database where Storage::Dir keeps the file. Closed named databases are now stashed via serialize() and restored on reopen (Database.deserialize(image, {safeIntegers: true})), and the close→reopen path is pinned in the SDK test.
4. DB_NAME_PATTERN tightened 64 → 57 chars. A 64-char name + .sqlite = 71 bytes, past the fs module's 64-byte segment ceiling (#238) — such a database file is invisible to a co-mounted fs module (pocket-fs deliberately skips unaddressable names in list()), contradicting the "visible like any of its files" contract both PRs document. 57 + 7 = exactly 64.

Consistent-and-fine, verified explicitly: missing named params bind NULL identically on both hosts (rusqlite clears cached-statement bindings, so no stale-binding leak); positional count mismatches fail loudly on both; big-integer and blob round-trips agree.

Verified

• cargo test -p pocket-db 10/10 (incl. the new expression-ATTACH test), clippy clean, cargo check --workspace clean
• tests/db.test.ts 18/18; full gate green except one pre-existing flake: symbian-runtime.test.ts "E7 scan matrix" times out intermittently on macOS (Gatekeeper first-launch stall on a freshly linked temp executable — same code passes/fails across runs, file identical to main, flakes on main too). Not this PR's fault; follow-up below.
• bunx tsc --noEmit clean; contract byte-compare green (the name pattern isn't part of the generated surface).
• The ESP-IDF claims (unix-none VFS, newlib shims, default-features = false for riscv32imafc-esp-espidf) are taken as hardware-verified per the ppa-smoke precedent — reproducing them locally means pulling the full ESP-IDF toolchain; the on-device evidence in #238's data-smoke is convincing.

Left alone, deliberately

• SQL random()/'now' are unpinned in the sim (bun exposes no VFS hook) — already documented, spec already forbids golden-tested apps from relying on them.
• The nanosleep shim's 64-bit time_t assumes espidf_time64 (documented in the recipe; matches the validated IDF v5.5.x config).
• Follow-up (not this PR): bump the symbian-runtime test's 5s timeout or pre-warm the executable — it flakes on managed macOS hosts.

🤖 Generated with Claude Code