Skip to content

v0.2.9 — the connect cell, cut by three quarters

Choose a tag to compare

@punnerud punnerud released this 03 Aug 14:59
· 98 commits to main since this release

Verified on Linux x86-64, macOS/Apple Silicon and Windows x86-64 — tests, clippy and the crash harnesses green on all three (CI runs the full suite on each). Prebuilt binaries attached below; sha256sums included.

macOS binaries follow shortly. The Apple-silicon build host was offline when this was cut, so the macOS CLI and shim are uploaded to this same release as soon as it is back. macOS is tested every push through CI — only the prebuilt artifact is late. pip install mpedb and cargo install are unaffected on every platform.

Covers 0.2.8 as well. 0.2.8's wheels went to PyPI but its GitHub release never got cut (the macOS build host was offline that night), so its notes are folded in here. If you are on 0.2.7, this release is both rounds.

The connect cell, cut by three quarters (0.2.9)

The Django 4.2 project driving this series re-profiled after 0.2.8 and reported the suite gap at noise level — with connect cost the last item on sqlite's win list at ~30×. That cell is now measured, attributed, and mostly gone:

  • A fresh memfd is already zero. format() wiped the whole control region — metas, lock area, reader table, intent ring — which on a fresh in-memory backing is not a memset but 92 page faults: 235 of a :memory: connect's ~330 µs. That wipe exists for reused shared files, whose control bytes carry a previous incarnation. A private backing (a memfd this very open created — by definition nothing else can attach) is kernel-zeroed by construction and now skips it. Every control invariant reads identically; the full wipe stays on every shared path, so the cross-process apparatus is exactly as SIGKILL-safe as before.
  • Two /proc reads per open, gone. The Linux boot id was read three times per open and our own start time once — both per-process constants. boot_id is now cached for the process lifetime (except on Windows, whose pair carries a live uptime clock that must be read fresh); the own-start-time cache is keyed by pid, because Python consumers fork and a child must not inherit its parent's reader identity.
  • MPEDB_OPEN_TRACE=1 makes the open path self-attributing (per-stage µs through the facade, the engine and the shm init), with a connect_cost.rs loop that splits open from close.

Measured on a quiet box, :memory: open+close: 391 µs → ~100 µs (75 open, 15 close). What remains is the mapping's own lifecycle; the cross-process machinery now costs about 30 µs of it, and the reader table and intent ring cost nothing at all — their pages are never faulted in.

What 0.2.8 brought (folded in)

  • ROLLBACK TO SAVEPOINT stops paying for DDL that never happened. Every rollback rebuilt the session's schema view and cleared the entire shared plan cache — machinery that exists solely for rolled-back DDL, now gated on whether the session ever ran any. Savepoint statements went from ~2.4 µs to ~0.77 µs (level with the bench's null control), the disk savepoint-cycle cell from 4.1× to 1.93× stdlib. The suite-level effect is larger than the cell: Django's savepoint-per-test isolation no longer evicts every cached plan once per test.
  • executemany got its engine half (query_many): plan and per-statement facts resolved once, rows looped inside an engine savepoint — any row error rolls the batch back whole, so the per-row road reruns it with double-apply structurally impossible. 1.8 → 1.5 µs/row.
  • Two report items settled by measurement, not code: the "legacy BEGIN" divergence is pinned stock-identical on the native route (stdlib refuses it too — the report expected otherwise), and the memory-vs-disk asymmetry in the microbench attributes to the control arm's storage sensitivity; mpedb's own per-op cost is backing-independent.

Also since 0.2.7

  • The playground opens again (punnerud.github.io/mpedb). It had been trapping at open since late July: the notify epoch stamp and the :memory: namer read std::time/std::process::id directly, both of which panic on wasm32, while every native suite stayed green and the Pages workflow's "success" only ever meant deployed. Both cures landed — and Pages gained the gate that was missing: the freshly built module must open, not merely compile.
  • benchmarks/concurrency.md — "how many concurrent locks before it melts", answered with a curve: 1→1 024 writer processes against same-shape sqlite (WAL + synchronous=FULL + fullfsync) and PostgreSQL 16 controls on one host. It climbs to a 128-process peak (47 k durable commits/s, 5× PostgreSQL's best) and sags; it does not melt. Plus the latency half, because throughput can hide starvation: mpedb's ordered ring holds p99 ≈ 2× p50 through 128 writers, while the sqlite control's worst writer starves 15+ seconds from 8 processes up.

Attached binaries

file contents
mpedb-0.2.9-linux-x86_64.tar.gz mpedb CLI, Linux x86-64
mpedb-0.2.9-windows-x86_64.zip mpedb.exe, Windows x86-64
libmpedb_sqlite3-0.2.9-linux-x86_64.tar.gz sqlite3 drop-in shim, Linux
SHA256SUMS checksums for the above
mpedb-0.2.9-macos-arm64.tar.gz mpedb CLI, macOS Apple Silicon — uploaded shortly
libmpedb_sqlite3-0.2.9-macos-arm64.tar.gz sqlite3 drop-in shim, macOS — uploaded shortly