Modern C++ benchmarking
Assembly Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


This is a set of benchmarks in C++ that tries to compare “raw/C-ish code” or old C++ style implementations vs “library-based, modern C++” implementations of some algorithms and compares their execution time.

For every benchmark, two implementations are introduced:

  • raw implementation.
  • modern C++ implementation.

The goal is to put them front to front to see how they perform against each other, on a per-compiler basis.

Plots are generated, grouping, per-compiler, the two versions put front to front.

I am particularly interested in measuring the abstraction penalty incurred by the use of a C++ vs C-ish plain approaches when compiling programs with optimization, since one of the goals of C++ is the zero-overhead principle.

My first experiment makes use of Eric Niebler’s ranges library. There is a standard C++ proposal for inclusion based on this work.

Benchmark style and guidelines

The scope of this benchmark set is very targeted: I want to show how typical, older-style or C-ish code or old-style C++ code performs against idiomatic modern C++ code.

I want to limit the benchmarks to code that is easy to write for the average programmer, things that would be likely to be written in the wild as a normal attemp to perform an action, that look simple.

For example, if there is an I/O benchmark of C I/O vs C++ I/O, I consider cheating to start to mess up with buffers and locales and advanced tricks to improve performance, because this is not what a normal programmer would write as a first attemp.

Contributions will be accepted. Suggestions and ideas for new benchmarks are welcome as well. I will reserve for myself the right to accept or reject a benchmark to the set of benchmarks, with the hope of keeping it focused. :).

Compile and run the benchmarks

So you want to run the benchmark yourself in your computer…


  • python 3 (though currently cmake is looking for python 2!).
  • gnuplot.
  • compilers to run the benchmarks against (only tested g++ and clang++ in OS X at this time).

NOTE: Not tested in Visual Studio compiler yet, but it should work.

git clone
cd the-cpp-abstraction-penalty
git submodule init
git submodule update

#Example ./ clang++ g++
./ [compiler-exes]

This will do the following:

  1. Build the binaries for your compilers.
  2. Run the binaries for the benchmark.
  3. Put, for each benchmark, a png file in build-all directory that you can merrily open when done to see the chart.

WARNING: Only tested on my Mac OSX computer.

How to contribute a new benchmark

Your benchmark must return the number of milliseconds that took for it to execute as an integer number. The provided benchmarks return the median of 5 executions, so for now I recomend to just do this. There is a utility file in benchmarks/util that you can use for timing.

NOTE: Maybe I will take a more scientific look at how to achieve more accurate results in the future, but this will suffice for now.

Getting your benchmark to work

  1. Create a directory for your benchmark in benchmarks/your-benchmark-directory.
  2. Use a template like the one below for your CMakeLists.txt in the newly created directory:
if (UNIX)

set(TCPPAP_BENCHMARK_TITLE "Title for benchmark")
set(TCPPAP_BENCHMARK_NAME <dir-name-for-benchmark>)
set(TCPPAP_BENCHMARK_NAME my-benchmark-name)
set(TCPPAP_OLD_STYLE_SOURCES my_file1.cpp ...)
set(TCPPAP_MODERNCPP_STYLE_SOURCES my_modern_file1.cpp ...)

#Example benchmark parameters
set(TCPPAP_PROGRAM_ARGS 5 10000000)

  1. Take a look at the provided main function examples for inspiration if needed, such as Sieve.
  2. Fork the repo and create a branch for working on your benchmark.
  3. When done, just send a pull request. I will try to take a look, mainly at weekends due to my workload. :)

Variables reference

Read-only variableMeaning
TCPPAP_BENCHMARKS_DIRDirectory where benchmarks hang from.
TCPPAP_MODULES_DIRSubmodules directory for any dependencies for benchmarks.
User variableMeaning
TCPPAP_BENCHMARK_TITLEFree form title such as “std::memcpy vs std::copy”
TCPPAP_BENCHMARK_NAMEThis is the benchmark name, which must match the directory name
TCPPAP_MODERNCPP_STYLE_SOURCESSource files for modern benchmark executable.
TCPPAP_MODERNCPP_STYLE_INCLUDESUse for include directories for only modern style benchmark exe.
TCPPAP_OLD_STYLE_SOURCESSource files for old benchmark executable.
TCPPAP_OLD_STYLE_INCLUDESUse for include directories for only old style benchmark exe.
TCPPAP_INCLUDESUse for include directories for both modern and old style benchmark exe.
TCPPAP_PROGRAM_ARGSArguments passed to the benchmark programs when running

Benchmarks results

Hardware information

I am using a 2,4 GHz Intel Core i5 4 GB 1600 MHz DDR3 with graphics card Intel Iris 1536 MB. My OS is OSX Yosemite 10.10.1.

The versions of the compilers used for the benchmarks in my computer are:

  • g++-5 (Homebrew gcc5 5.2.0) 5.2.0.
  • Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn) Target: x86_64-apple-darwin14.0.0 Thread model: posix.