Skip to content

Rewrite the README on fresh benchmarks, and fix the benchmarks first - #80

Merged
martinus merged 1 commit into
mainfrom
readme-fresh-benchmarks
Jul 28, 2026
Merged

Rewrite the README on fresh benchmarks, and fix the benchmarks first#80
martinus merged 1 commit into
mainfrom
readme-fresh-benchmarks

Conversation

@martinus

Copy link
Copy Markdown
Owner

The README's numbers were from 2022 — an i7-8700, clang 13, and whatever boost and abseil were current then. Re-measured against boost 1.90.0 and abseil 20260107.1 with gcc 16.1.1 -O3 -DNDEBUG.

Three things had to be fixed before the numbers meant anything.

bench_push_back counted a batch it never performed

The loop pushes num_iters elements. The batch was the sum of num_iters draws of rng.bounded(16) — left over from a version that pushed a random number of elements per iteration. Every ns/push_back this ever reported, including the one in the README, was about 7.5x too small.

bench_sort_shuffle's string half never ran

It asserted vec.front() == "10000017441998304507", which is not the lexicographic minimum of the numbers it generates and never has been, so the test case aborted. The correct value is 10028905880159877647, confirmed identical under gcc and clang; the uint64_t half next to it has the right constant.

Four containers in one process do not measure the same thing

This is the one that mattered. glibc adapts its mmap threshold the first time a large block is freed, so whichever container runs first pays for mmap/munmap on every iteration and trains the allocator for everyone behind it. nanobench excludes kernel cycles, which is what makes it visible: the affected row reports ~37 cycles at 27 ns, an implied 1.4 GHz on a machine where every other row implies 4.5 GHz.

It follows position in the list, not the container:

order 1st 2nd 3rd 4th
std::vector first 23.5 3.7 4.3 3.8
svector first 22.9 25.2 22.6 23.4

Two published results reverse once each container gets its own process, both of which had been flattering svector:

workload in one process one process each
emplace_back std::string absl 7.1, svector 8.4, std::vector 27.5 svector 22.5, std::vector 23.5, absl 23.7, boost 30.2
insert at random index, std::string svector slowest at 722 svector fastest at 675, absl 1021
insert at front, std::string svector fastest at 1324 boost/std::vector 1327/1322, svector 1506

test/bench/solo.cpp runs one workload against one container and exits — which is also what a program that uses one of them looks like. scripts/bench/render_charts.py drives it and renders the charts from the median of 9 runs. The doctest bench suite stays as it is; it is still the right tool for A/B testing a change to svector against itself, which is what it gets used for.

Also here

  • accumulate, emplace_back and fill_random used 100 epochs of 10 ms where everything else uses the default epoch count at 100 ms. Same wall time, and it stops nanobench reporting absl's emplace_back as unstable.
  • show_comparison now prints the README's compactness table for whichever boost and abseil are installed, instead of three hardcoded sizeofs.
  • name_of_type() now says out loud that find() cannot fail. Without it, meson setup --buildtype=release — the command the README tells you to run — does not compile on gcc 16, because -O3 -Werror decides remove_prefix(npos) is reachable.

What the numbers say

svector is 2.2x behind std::vector on push_back of a trivial type (19.3 instructions vs 8.5) and 18% behind on a subscript. That is the tagged pointer being paid for on every single operation, and the README now says so plainly instead of claiming svector is "quite fast" there. Everywhere the elements do real work it is level with the alternatives or ahead — fastest of the four at emplace_backing strings — at a third the size of the next smallest object.

The compactness table is unchanged (8/7/1 against absl's 24/16/8 and boost's 32/8/24), but the old claim that boost rounds inline capacity down to a multiple of 8 bytes is no longer true of boost 1.90: small_vector<uint8_t, 15> really does give you 15.

Verification

108 cases / 629,413 assertions green on gcc, clang, C++20 and under ASAN+UBSAN, and the strict release build (werror=true, -O3) now compiles with absl and boost enabled.

The numbers in the README were from 2022, taken on an i7-8700 with clang 13
against whatever boost and abseil were current then. Re-measured against
boost 1.90.0 and abseil 20260107.1 with gcc 16.1.1 -O3 -DNDEBUG. Three of the
benchmarks had to be fixed before their numbers meant anything.

bench_push_back counted a batch it never performed. The loop pushes
num_iters elements, the batch was the sum of num_iters draws of
rng.bounded(16), left over from a version that pushed a random number of
elements per iteration. Every ns/push_back this ever reported, including the
one in the README, was about 7.5x too small.

bench_sort_shuffle's string half asserted a value that is not the
lexicographic minimum of the numbers it generates, and never has been, so
that half of the benchmark aborted instead of running. The uint64_t half
next to it has the right constant.

The worse problem is structural: four containers benchmarked one after the
other in a single process do not measure the same thing. glibc adapts its
mmap threshold the first time a large block is freed, so whoever runs first
pays for mmap and munmap on every iteration and trains the allocator for
everyone behind them. On emplace_back of 10000 std::string that is worth a
factor of three, and it follows position in the list rather than the
container: put svector first and svector becomes the slow one. Two results
reverse completely once each container gets its own process, and both of the
reversals were in svector's favour in the old arrangement.

So test/bench/solo.cpp runs one workload against one container and exits,
scripts/bench/render_charts.py drives it and renders the charts from the
median of several runs, and the README says which numbers came from where.
The doctest bench suite stays as it is; it is still the right tool for A/B
testing a change to svector against itself, which is what it gets used for.

Also here:
 * accumulate, emplace_back and fill_random used 100 epochs of 10ms where
   everything else uses the default epoch count at 100ms. Same wall time,
   and it stops nanobench reporting absl's emplace_back as unstable
 * show_comparison prints the README's compactness table for whatever boost
   and abseil are installed, instead of three hardcoded sizeofs
 * name_of_type() now says out loud that find() cannot fail, which is what
   -O3 -Werror on gcc 16 wanted to hear. Without it the release build the
   README tells you to run does not compile

What the numbers say: svector is 2.2x behind std::vector on push_back of a
trivial type and 18% behind on a subscript, which is the tagged pointer being
paid for on every operation. Everywhere the elements do real work it is level
with the alternatives or ahead, at a third the size of the next smallest
object. The old README's claim that boost rounds inline capacity down to a
multiple of 8 is no longer true of boost 1.90.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@martinus
martinus merged commit 2b721ed into main Jul 28, 2026
6 checks passed
@martinus
martinus deleted the readme-fresh-benchmarks branch July 28, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant