Benchmarks

Benchmarks

Curated performance comparison of granary vs. reference C SQLite (3.45.1), both driven from the same OCaml harness (test/bench_compare.ml) with prepared statements, WAL mode, matched page cache (1024 × 4 KiB pages), fsync-per-commit durability (SQLite synchronous=FULL), and an identical deterministic dataset. Each workload is the best of N timed repeats after a discarded warmup. cpu/wall (process CPU time ÷ wall clock) tells CPU-bound (≈ 1) apart from fsync/I/O-wait-bound (≪ 1) workloads.

Full raw data, methodology, and history live in the repo under docs/benchmarks/ and bench/results/ — this page is a snapshot summary, not the source of truth.

These are the curated reference numbers, run on real dedicated hardware (see Host table below). The Benchmark (#345) GitHub Actions workflow (workflow_dispatch) also runs on shared, variable-performance GitHub-hosted cloud runners for CI visibility — those results land on separate CI-Benchmark-Results-* pages and are not comparable to the numbers on this page.

Host (NVMe reference)

host	cores	storage
reference host	12	NVMe (Samsung MZVL2512, md-RAID)

Parameters: 5000 rows, 500 point-lookup / autocommit-insert ops, 20 scan repeats, 50 single-row commits, 2 timed repeats, 1024-page cache.

Head-to-head, plaintext, NVMe (ops/sec; higher is better)

workload	SQLite	granary	SQLite advantage	cpu/wall (sqlite → granary)
point lookup WHERE pk=?	689,626	187,967	~3.7×	1.02 → 1.00
range scan / aggregate	7,233	868	~8.3×	1.00 → 1.07
insert (autocommit)	113.4	56.9	~2.0×	0.03 → 0.50
insert (batch, 1 txn)	345,017	7,752	~44.5×	0.46 → 0.83
commit throughput	117.4	40.8	~2.9×	0.04 → 0.58

• Reads are CPU-bound (cpu/wall ≈ 1.0): point lookup ~3.7×, scan/aggregate ~8.3× slower than SQLite. The point-lookup path is a real B-tree seek, not a full scan.
• Single-row writes are fsync-bound (cpu/wall 0.50–0.58): ~2–3× SQLite on NVMe, the residual being granary's heavier per-insert CPU on top of the same fsync.
• Batch insert (~44×) is the largest remaining gap — O(n) now (down from O(n²)), but with a high constant from copy-on-write write-amplification per row. Tracked as the next perf target.

Status vs. the "within 10× of SQLite" goal: 4 of 5 workloads are within 10× (point lookup 3.7×, scan 8.3×, autocommit insert 2.0×, commit 2.9×). Batch insert (~44×) is the outlier.

Encryption tax (AES-256-GCM at rest)

workload	plaintext → encrypted	overhead
point lookup WHERE pk=?	187,967 → 343,177	within noise (sub-ms totals)
range scan / aggregate	868 → 718	~1.2× (≈ +20%)
insert (batch, 1 txn)	7,752 → 7,004	~1.1×
commit throughput	40.8 → 35.0	~1.2×

The frame-cache caches decrypted pages, so per-page AES-256-GCM cost is amortized across cache-resident reads. Writes are barely affected.

Durability knob (PRAGMA synchronous) on a fsync-bound HDD host

The NVMe numbers above run at synchronous=full (one fsync per commit — matches SQLite's default). On a small, fsync-bound HDD host (md-RAID over spinning disks — the profile that caps how many apps a box can co-host), the durability knob (full / batched / off) is the dominant lever:

workload	mode	wall (s)	ops/s	speedup vs full
insert_one	full	89.45	11.2	1.0×
insert_one	batched	9.83	101.7	9.1×
insert_one	off	8.18	122.2	10.9×
commit_n	full	42.78	11.7	1.0×
commit_n	batched	6.20	80.6	6.9×
commit_n	off	5.02	99.6	8.5×

(SQLite reference at synchronous=FULL: insert_one 14.1 ops/s, commit_n 36.3 ops/s.)

Relaxing durability lifts per-connection commit throughput ~7× (batched) to ~9–11× (off). On a fsync-bound host that headroom translates directly into more co-hosted apps before the disk's fsync ceiling binds. batched bounds the crash-loss window to a configurable commit count; off treats the store as reconstructible/ephemeral.

Reproduce

GRANARY_BENCH_ROWS=5000 GRANARY_BENCH_OPS=500 GRANARY_BENCH_SCANS=20 \
  GRANARY_BENCH_COMMITS=50 GRANARY_BENCH_REPEATS=2 scripts/bench222.sh <host-label>
# builds the bench image, runs locally, writes bench/results/<host-label>.csv

granary is a young engine (pre-release, 0.0.x) — these numbers are expected to keep moving as the write path is optimized further. See Home.