Rewrite the README on fresh benchmarks, and fix the benchmarks first - #80
Merged
Conversation
The numbers in the README were from 2022, taken on an i7-8700 with clang 13 against whatever boost and abseil were current then. Re-measured against boost 1.90.0 and abseil 20260107.1 with gcc 16.1.1 -O3 -DNDEBUG. Three of the benchmarks had to be fixed before their numbers meant anything. bench_push_back counted a batch it never performed. The loop pushes num_iters elements, the batch was the sum of num_iters draws of rng.bounded(16), left over from a version that pushed a random number of elements per iteration. Every ns/push_back this ever reported, including the one in the README, was about 7.5x too small. bench_sort_shuffle's string half asserted a value that is not the lexicographic minimum of the numbers it generates, and never has been, so that half of the benchmark aborted instead of running. The uint64_t half next to it has the right constant. The worse problem is structural: four containers benchmarked one after the other in a single process do not measure the same thing. glibc adapts its mmap threshold the first time a large block is freed, so whoever runs first pays for mmap and munmap on every iteration and trains the allocator for everyone behind them. On emplace_back of 10000 std::string that is worth a factor of three, and it follows position in the list rather than the container: put svector first and svector becomes the slow one. Two results reverse completely once each container gets its own process, and both of the reversals were in svector's favour in the old arrangement. So test/bench/solo.cpp runs one workload against one container and exits, scripts/bench/render_charts.py drives it and renders the charts from the median of several runs, and the README says which numbers came from where. The doctest bench suite stays as it is; it is still the right tool for A/B testing a change to svector against itself, which is what it gets used for. Also here: * accumulate, emplace_back and fill_random used 100 epochs of 10ms where everything else uses the default epoch count at 100ms. Same wall time, and it stops nanobench reporting absl's emplace_back as unstable * show_comparison prints the README's compactness table for whatever boost and abseil are installed, instead of three hardcoded sizeofs * name_of_type() now says out loud that find() cannot fail, which is what -O3 -Werror on gcc 16 wanted to hear. Without it the release build the README tells you to run does not compile What the numbers say: svector is 2.2x behind std::vector on push_back of a trivial type and 18% behind on a subscript, which is the tagged pointer being paid for on every operation. Everywhere the elements do real work it is level with the alternatives or ahead, at a third the size of the next smallest object. The old README's claim that boost rounds inline capacity down to a multiple of 8 is no longer true of boost 1.90. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README's numbers were from 2022 — an i7-8700, clang 13, and whatever boost and abseil were current then. Re-measured against boost 1.90.0 and abseil 20260107.1 with gcc 16.1.1
-O3 -DNDEBUG.Three things had to be fixed before the numbers meant anything.
bench_push_backcounted a batch it never performedThe loop pushes
num_iterselements. The batch was the sum ofnum_itersdraws ofrng.bounded(16)— left over from a version that pushed a random number of elements per iteration. Everyns/push_backthis ever reported, including the one in the README, was about 7.5x too small.bench_sort_shuffle's string half never ranIt asserted
vec.front() == "10000017441998304507", which is not the lexicographic minimum of the numbers it generates and never has been, so the test case aborted. The correct value is10028905880159877647, confirmed identical under gcc and clang; theuint64_thalf next to it has the right constant.Four containers in one process do not measure the same thing
This is the one that mattered. glibc adapts its mmap threshold the first time a large block is freed, so whichever container runs first pays for
mmap/munmapon every iteration and trains the allocator for everyone behind it. nanobench excludes kernel cycles, which is what makes it visible: the affected row reports ~37 cycles at 27 ns, an implied 1.4 GHz on a machine where every other row implies 4.5 GHz.It follows position in the list, not the container:
std::vectorfirstsvectorfirstTwo published results reverse once each container gets its own process, both of which had been flattering svector:
emplace_backstd::stringstd::vector27.5std::vector23.5, absl 23.7, boost 30.2std::stringstd::stringstd::vector1327/1322, svector 1506test/bench/solo.cppruns one workload against one container and exits — which is also what a program that uses one of them looks like.scripts/bench/render_charts.pydrives it and renders the charts from the median of 9 runs. The doctest bench suite stays as it is; it is still the right tool for A/B testing a change to svector against itself, which is what it gets used for.Also here
accumulate,emplace_backandfill_randomused 100 epochs of 10 ms where everything else uses the default epoch count at 100 ms. Same wall time, and it stops nanobench reporting absl'semplace_backas unstable.show_comparisonnow prints the README's compactness table for whichever boost and abseil are installed, instead of three hardcodedsizeofs.name_of_type()now says out loud thatfind()cannot fail. Without it,meson setup --buildtype=release— the command the README tells you to run — does not compile on gcc 16, because-O3 -Werrordecidesremove_prefix(npos)is reachable.What the numbers say
svector is 2.2x behind
std::vectoronpush_backof a trivial type (19.3 instructions vs 8.5) and 18% behind on a subscript. That is the tagged pointer being paid for on every single operation, and the README now says so plainly instead of claiming svector is "quite fast" there. Everywhere the elements do real work it is level with the alternatives or ahead — fastest of the four atemplace_backing strings — at a third the size of the next smallest object.The compactness table is unchanged (8/7/1 against absl's 24/16/8 and boost's 32/8/24), but the old claim that boost rounds inline capacity down to a multiple of 8 bytes is no longer true of boost 1.90:
small_vector<uint8_t, 15>really does give you 15.Verification
108 cases / 629,413 assertions green on gcc, clang, C++20 and under ASAN+UBSAN, and the strict release build (
werror=true,-O3) now compiles with absl and boost enabled.