Gotchas and Lessons Learned

Gotchas and Lessons Learned

The non‑obvious things — the ones that cost real debugging time, changed a conclusion, or became upstream issues. Grouped by theme.

Measurement

A rising RSS line is not a leak

Under -gc none nothing is freed, so RSS grows — but a Boehm build grows too (glibc arena, connection buffers, thread stacks). Measure the excess over the Boehm GC floor, not raw RSS. This flipped more than one conclusion. See Profiling and Benchmarking.

Load shape decides what you can see

A bug invisible under one load shape dominates under another. Per‑request allocations need any load; pipeline‑queue allocations need concurrent load with clients > pool conns; per‑connection allocations need connection churn. The arena reconnects ~17–19 k times/run, so a per‑connection leak that hid under persistent‑connection local tests was the real‑world dominant cost.

heaptrack blurs one‑time bring‑up into the per‑request signal

It attributes from process start, so SCRAM/PBKDF2 and lazy pool init look like per‑request cost. We once concluded "the residual is just startup" and were wrong. callgrind with post‑warmup instrumentation (turn it on after warmup) is the disambiguator.

callgrind needs gcc, not tcc

V's default -g build uses tcc, whose debug info callgrind can't resolve — V app functions show as bare hex addresses. Build the profiling binary with -cc gcc for real DWARF.

Don't dump callgrind with a live control command on an idle reactor

callgrind_control -d hangs when all worker threads are parked in epoll_wait (nothing reaches an instrumentation point). SIGTERM the valgrind process instead — the signal interrupts the syscall and callgrind flushes its dump on fini.

V language quirks (filed upstream)

Allocation does not scale across cores

On an 8‑core/16‑thread box, a tight per‑thread allocation microbenchmark showed the default (Boehm) GC's aggregate throughput stay flat at ~18 M allocs/s from 1→16 threads (16 cores ≈ 1 core; per‑thread throughput collapsed ~16×), while -gc none (libc malloc) scaled ~2.5× before plateauing past 4 cores. This is the reason vanilla is thread‑per‑core and -gc none. Filed: vlang/v#27488.

Empty/zero‑length array literals allocate

[]T{} (and a default‑initialised []T struct field) allocate a backing buffer even though len == 0 and cap == 0 — __new_array(0,0,…) still calls alloc_array_data(0) → vcalloc(header). Under -gc none each such construction is a permanent leak. The fix in our code was to append elements / use module consts instead of literals; filed upstream as vlang/v#27487 (a 0‑cap array should have data == nil).

Option/Result error‑boxing allocates on the hot path

error() boxes a MessageError — even when the caller discards it with or {}. So find_byte (which returns !int and error()s on not‑found) allocated per lookup; switching to a -1‑returning find_byte_idx removed it. Prefer non‑erroring "not found" signals on hot paths.

runtime.nr_cpus() ignores CPU affinity

It returns the system's online CPU count, not the sched_getaffinity set — taskset -c 0 still reports all cores. Programs that size thread pools by it over‑spawn under cgroup/container/taskset limits. Keep in mind when pinning for profiling.

A few other sharp edges

• &Struct{} as an if‑expression branch has miscompiled to invalid C under some build modes (kin to vlang/v#27485) — use the statement form (mut x := &T(unsafe{nil}); if … { x = … } else { x = &T{…} }).
• v fmt reformats single‑line unsafe { … } // comment blocks to multi‑line; the codebase keeps the compact form on hot paths intentionally, so fmt -verify may flag it.
• Date.now() / Math.random() etc. are not the concern here, but: integer→string (n.str()) allocates — use a digit‑writer into a reused buffer (emit_int) on hot response paths.

The crud "4xx" that wasn't a bug

A benchmark run reported ~1.4 % 4xx on the crud profile — and only crud. The investigation is a good template for "is this us or the harness?":

1. It wasn't the load generator. The committed leaderboard showed all 21 frameworks (including a prior vanilla build) at 0 crud 4xx with the same gcannon. And gcannon's template substitution (write_padded) keeps request length fixed, so its Content-Length is always correct — confirmed at the source.
2. It wasn't request framing. A faithful replay of the real crud fixtures — correct substitution, both fresh‑connection and keep‑alive‑with‑reconnect, up to 96 threads — returned 100 % 2xx. The framer is also robust to arbitrary TCP segment splits.
3. It was our fallback choice. Under pool saturation, park() sheds to a caller‑chosen fallback. Reads shed to a benign 200; crud write/get shed to 400/404 — misreporting backpressure as a client error. Reproduced locally by forcing saturation (DATABASE_MAX_CONN=1): a create burst returned 0.21 % 400.
4. Fix: crud write/get shed → 503 (honest backpressure), genuine 400/404 preserved. The deeper backpressure policy is tracked in vanilla#51.

Lesson: before filing a bug against a dependency/harness, check whether the symptom is yours — the leaderboard (everyone else at 0) was the decisive datapoint.

Backpressure semantics are a real design decision

Shedding under saturation keeps closed‑loop throughput from collapsing, but what status a shed returns is a policy, not an afterthought. 503 is honest; an empty 200 is "optimistic"; 400/404 are wrong. And reducing shedding by raising max_inflight trades directly against memory (max_inflight × 16 KiB reply buffer per connection) — exactly the budget the rest of this work was protecting. There's rarely a free win; there's a tradeoff to measure.

Process lessons

• Verify before claiming. Two "obvious" V bugs (a codegen miscompile, a fmt non‑idempotency) didn't reproduce minimally / turned out to be defined behavior — so they weren't filed. Two that did reproduce on master with a minimal repro were. A minimal, upstream‑reproducible repro is the bar.
• Adversarial review of buffer reuse pays off. Reusing a buffer is only safe under invariants (single‑threaded worker; len‑bounded reads; push gated to len==0). Each reuse change was reviewed specifically for stale‑state bleed, double‑free/aliasing, and dangling references before landing.