Badger<>Pebble Showdown takeaways

[drew-512]

Aloha badger friends,

We just completed a several-day, best-tuned-vs-best-tuned evaluation of Badger
v4.9.2 against Pebble (v1.1.5, then v2.1.6) as the storage engine for a
federated content platform — an append-heavy durable journal plus an
LSM-projection store, behind a seam that runs both engines against identical
workloads. Sharing the Badger-relevant findings in case they're useful
upstream.

Method: seed-deterministic synthetic data (a small-value-heavy
"representative" profile and a stress profile with 12% 1 MiB values), 6-run
interleaved medians, Go 1.26.3, on three hosts: a quiet 6-core arm64 Linux VM
(ext4), a 1-CPU x86_64 Linux box under live service load (ext4), and an M1 Pro
(macOS 14.8.7).

Where Badger clearly won

metric (6-run medians)	badger v4.9.2	pebble v2.1.6 (value separation on)
bulk rebuild, stress corpus (blind write-batch + sweep)	5.8K txs/s	1.3K txs/s
durable-append p99.9 under load, 1-CPU box	1.3–2.9 ms	4.7–8.1 ms
stress cabinet merge, 1-CPU box	~1.0 ms/tx	~0.95–1.3 ms/tx (parity)

The value-log architecture is the story: with Pebble v1 (no value
separation), Badger won the 1 MiB-value axis by 2.7–6× everywhere. Pebble v2's
blob-file value separation closed most of that, but Badger still owns bulk
ingest of large values and the extreme tail on constrained hardware.

Things that bit us (v4.9.2 specifics)

1. The atomic txn ceiling is reachable from surprisingly small inputs.
   With default options, MaxBatchCount ≈ 104K entries — a single 3.5 MB
   application message expanding to ~95K entries hits it, and ErrTxnTooBig
   from a mid-batch commit poisons the batch. We now pre-check every unit
   against 0.9× DB.MaxBatchSize()/MaxBatchCount() and classify before
   committing. Suggestion: the docs could call out how low the entry-count
   ceiling sits relative to real-world message sizes.
2. Reverse iteration ergonomics. "Last key ≤ k under a prefix" requires
   seeking to a manually 0xFF-padded successor; we shipped a silent bug from
   getting this wrong, and per-step reverse iteration (skiplist descent +
   tombstone wading) benchmarked ~2.7× slower than an equivalent forward scan
   at 1M keys, so we restructured to forward walks. A native bounded/reverse
   helper would remove a real footgun.
3. Tuning observation: raising NumMemtables 5→8 and
   NumLevelZeroTablesStall to 30 (compression off) improved medians
   slightly but worsened durable-append p99.9 at 100K-tx scale (6.3 ms vs
   2.9 ms stock) — more buffered state, burstier flush/compaction debt.
   Stock settings were the better tail choice for us.
4. Forgiving iterator buffer lifetimes can mask caller bugs. Our code held
   Item.Value bytes past Next() into an async writer — harmless on Badger,
   wire corruption on engines that recycle buffers. Not a Badger defect, but
   its leniency hid the bug for months.

Questions

On macOS, is SyncWrites=true intended to guarantee power-loss durability?
Reading v4.9.2 (with ristretto's z/mmap_darwin.go), the WAL/vlog/table sync
paths go through msync(MS_SYNC) / File.Sync on mmap'd files, and we don't
see an F_FULLFSYNC (fcntl) anywhere — on Darwin, plain fsync/msync does not
flush the drive cache, so acknowledged writes can be lost on power failure.
(This is also why Badger benches ~50–80× faster than F_FULLFSYNC-honest engines
on Macs — an unfair comparison in both directions.) If this is by design,
a doc note would help; if not, would a platform-gated full-fsync option be
welcome as a PR?

I've been a fan of badger for years now, but unfortunately
we ultimately chose Pebble v2 for value separation and unbounded batch
semantics fitting our shape — but Badger's large-value ingest and
constrained-hardware tails remain genuinely best-in-class in our data, and
am happy to share fuller numbers or the corpus generator if y'all are interested.

[matthewmcneely]

@drew-512 thanks for the thorough writeup — the durability callout is a good one, and you're right about the part that matters.

One correction on the detail: os.File.Sync() on Darwin does issue fcntl(F_FULLFSYNC). Go added that in 1.12 (golang/go#26650) and it's still there in current toolchains, so the manifest and directory syncs are covered.

The gap you're pointing at is real, though. The WAL and value log are memory-mapped and synced via msync(MS_SYNC), which the runtime's F_FULLFSYNC fix doesn't reach. So with SyncWrites=true on macOS, an acknowledged write can still sit in the drive's write cache and be lost on power failure. This is effectively a re-opening of #563, which was closed back when the WAL predated the mmap rearchitecture. It's most likely the cause for some reported data loss issues on Darwin following power loss events.

The fix would be a platform-gated F_FULLFSYNC after msync on Darwin, behind an opt so users don't eat the (significant) perf cost by default. We'd welcome a PR if you're interested.

[drew-512]

Hello from Austin Texas.

Right on, appreciate your time and consideration mate. Will let it simmer for a bit and will post back here with a PR sketch for feedback.

Appreciate all y'alls work on badger.