Profiling and Benchmarking

Profiling and Benchmarking

The zero‑allocation work was driven by two reproducible harnesses and one firm methodology. Both harnesses spin a throwaway, seeded PostgreSQL 18 container, build the server, drive load, and clean up after themselves — safe to run unattended. (They live in a sibling vanilla-perf/scripts/ tree, not in the server repo; the recipes below are self‑contained.)

Methodology — two rules that mattered

1. Measure excess over the GC floor, not raw RSS

Under -gc none nothing is freed, so RSS grows — but a Boehm‑GC build of the same server also grows somewhat (glibc arena, connection buffers, thread stacks). A rising RSS line is not by itself proof of a collectable leak.

Build both ways, drive identical load, and report gc_none_growth − boehm_growth. That difference is the genuinely‑collectable per‑request allocation — the only part -gc none turns into a permanent leak.

This reframing changed the conclusion more than once: e.g. after one stage, raw RSS still grew ~21 B/req, but Boehm grew ~10 B/req too, so the real residual was ~11 B/req — and it traced to connection setup, not the request path.

2. Load shape decides what you can even see

A single sequential client never exercises the pipelining queue, and persistent keep‑alive connections never exercise per‑connection setup/teardown. Different bugs hide behind different load shapes:

To surface…	drive…
per‑request render/parse allocations	sequential or low‑concurrency
pipelining‑queue allocations	concurrent, with clients > pool connections so queries multiplex
per‑connection (setup/teardown) allocations	connection churn — many short‑lived connections (one request per connection)

The arena's load generator reconnects ~17–19 k times per run, which is why a per‑connection leak that's invisible under persistent‑connection load dominated the real benchmark.

Harness 1 — callgrind: allocations per request, by call site

This is the scalpel. It answers exactly which function allocates, and how many times per request.

Key ideas baked into the recipe:

• Build with -cc gcc, not the default tcc. callgrind can't resolve V app symbols from tcc's debug info — every function shows as a hex address. gcc emits proper DWARF, so main__* / pg_async__* are named.
• --instr-atstart=no, then turn instrumentation on after a hard warmup. Warmup (pool bring‑up, SCRAM handshake, buffers reaching high‑water) runs uninstrumented, so only steady‑state per‑request work is counted.
• Don't dump with a live callgrind_control -d — it hangs when every worker thread is parked in epoll_wait (no thread reaches an instrumentation point to service the request). Instead SIGTERM the valgrind process: the signal interrupts epoll_wait, valgrind runs callgrind's fini, and the final dump is flushed.
• Parse the raw dump for allocator call counts. callgrind compresses fn/cfn names to ids, so build an id→name map, then sum calls= to each V allocator entry (vcalloc, malloc_uninit, memdup, …) attributed to its immediate caller. calls / measured‑requests = allocations per request. (The self‑cost table doesn't show call counts, and allocators are cheap‑but‑frequent, so they never surface there — parse the dump directly.)
• Drive the load shape that exposes the bug (see rule 2): concurrent for the pipeline queue, churn for per‑connection cost.

Outcome examples from this method: it pinned the residual async‑db allocations to begin_msg's array literal (4/req) and the reactor queue churn; later it confirmed the DB request path at ~0.003 allocs/req and the ConnState pool at 3.0 → 0.001 allocs/reconnect.

Harness 2 — RSS slope: the leak, in bytes/request

A coarser but end‑to‑end check: build -gc none and Boehm, run each under wrk load for a fixed window while sampling /proc/<pid>/status VmRSS, and report bytes/request plus the RSS trajectory (so you can see linear growth vs plateau). A hard RSS cap kills the server if a runaway leak would exhaust the box — so it's safe to leave running.

Report both builds side by side and subtract (rule 1). The trajectory shape is as informative as the slope: a one‑time jump that then flattens is connection setup; a sustained linear climb is a true per‑request leak.

heaptrack — the caveat that started it all

heaptrack does see allocations under -gc none (they're plain libc malloc, unlike Boehm's GC_malloc). But it attributes from process start, so one‑time bring‑up (SCRAM/PBKDF2, lazy pool init) can blur the per‑request signal and lead you to conclude "the residual is just startup" when it isn't. callgrind with post‑warmup instrumentation is the disambiguator.

Running the arena benchmark

The full cross‑framework benchmark is the MDA2AV/HttpArena harness (load generator: gcannon, io_uring). When iterating on vanilla, trigger it without saving results so you don't overwrite the leaderboard. The local harnesses above are the fast inner loop; the arena run is the at‑scale confirmation (128 cores, thousands of connections) — which is where, e.g., connection‑churn and pool‑saturation effects show up that a 16‑core box can't fully reproduce.

When a local run can't reproduce something the arena shows, that's a signal about scale or load shape, not necessarily a dead end — reach for connection churn or forced pool saturation before assuming the harness is at fault.