Modern C++ benchmarking
Assembly Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
benchmarks
cmake
plots
scripts
submodules
.gitignore
.gitmodules
CMakeLists.txt
LICENSE
README.org
README.org.in
ROADMAP.org
build-all-benchmarks.py

README.org

Purpose

This is a set of benchmarks in C++ that tries to compare “raw/C-ish code” or old C++ style implementations vs “library-based, modern C++” implementations of some algorithms and compares their execution time.

For every benchmark, two implementations are introduced:

  • raw implementation.
  • modern C++ implementation.

The goal is to put them front to front to see how they perform against each other, on a per-compiler basis.

Plots are generated, grouping, per-compiler, the two versions put front to front.

I am particularly interested in measuring the abstraction penalty incurred by the use of a C++ vs C-ish plain approaches when compiling programs with optimization, since one of the goals of C++ is the zero-overhead principle.

My first experiment makes use of Eric Niebler’s ranges library. There is a standard C++ proposal for inclusion based on this work.

Benchmark style and guidelines

The scope of this benchmark set is very targeted: I want to show how typical, older-style or C-ish code or old-style C++ code performs against idiomatic modern C++ code.

I want to limit the benchmarks to code that is easy to write for the average programmer, things that would be likely to be written in the wild as a normal attemp to perform an action, that look simple.

For example, if there is an I/O benchmark of C I/O vs C++ I/O, I consider cheating to start to mess up with buffers and locales and advanced tricks to improve performance, because this is not what a normal programmer would write as a first attemp.

Contributions will be accepted. Suggestions and ideas for new benchmarks are welcome as well. I will reserve for myself the right to accept or reject a benchmark to the set of benchmarks, with the hope of keeping it focused. :).

Compile and run the benchmarks

So you want to run the benchmark yourself in your computer…

Prerequisites:

  • python 3 (though currently cmake is looking for python 2!).
  • gnuplot.
  • compilers to run the benchmarks against (only tested g++ and clang++ in OS X at this time).

NOTE: Not tested in Visual Studio compiler yet, but it should work.

git clone https://github.com/germandiagogomez/the-cpp-abstraction-penalty.git
cd the-cpp-abstraction-penalty
git submodule init
git submodule update

#Example ./build-all-benchmarks.py clang++ g++
./build-all-benchmarks.py [compiler-exes]

This will do the following:

  1. Build the binaries for your compilers.
  2. Run the binaries for the benchmark.
  3. Put, for each benchmark, a png file in build-all directory that you can merrily open when done to see the chart.

WARNING: Only tested on my Mac OSX computer.

How to contribute a new benchmark

Your benchmark must return the number of milliseconds that took for it to execute as an integer number. The provided benchmarks return the median of 5 executions, so for now I recomend to just do this. There is a utility file in benchmarks/util that you can use for timing.

NOTE: Maybe I will take a more scientific look at how to achieve more accurate results in the future, but this will suffice for now.

Getting your benchmark to work

  1. Create a directory for your benchmark in benchmarks/your-benchmark-directory.
  2. Use a template like the one below for your CMakeLists.txt in the newly created directory:
if (UNIX)
  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_COMPILER_FLAGS} -std=c++14")
endif()

set(TCPPAP_BENCHMARK_TITLE "Title for benchmark")
set(TCPPAP_BENCHMARK_NAME <dir-name-for-benchmark>)
set(TCPPAP_BENCHMARK_NAME my-benchmark-name)
set(TCPPAP_OLD_STYLE_SOURCES my_file1.cpp ...)
set(TCPPAP_MODERNCPP_STYLE_SOURCES my_modern_file1.cpp ...)

#Example benchmark parameters
set(TCPPAP_PROGRAM_ARGS 5 10000000)

include(MakeBenchmark)
  1. Take a look at the provided main function examples for inspiration if needed, such as Sieve.
  2. Fork the repo and create a branch for working on your benchmark.
  3. When done, just send a pull request. I will try to take a look, mainly at weekends due to my workload. :)

Variables reference

Read-only variableMeaning
TCPPAP_BENCHMARKS_DIRDirectory where benchmarks hang from.
TCPPAP_MODULES_DIRSubmodules directory for any dependencies for benchmarks.
User variableMeaning
TCPPAP_BENCHMARK_TITLEFree form title such as “std::memcpy vs std::copy”
TCPPAP_BENCHMARK_NAMEThis is the benchmark name, which must match the directory name
TCPPAP_MODERNCPP_STYLE_SOURCESSource files for modern benchmark executable.
TCPPAP_MODERNCPP_STYLE_INCLUDESUse for include directories for only modern style benchmark exe.
TCPPAP_OLD_STYLE_SOURCESSource files for old benchmark executable.
TCPPAP_OLD_STYLE_INCLUDESUse for include directories for only old style benchmark exe.
TCPPAP_INCLUDESUse for include directories for both modern and old style benchmark exe.
TCPPAP_PROGRAM_ARGSArguments passed to the benchmark programs when running

Benchmarks results

Hardware information

I am using a 2,4 GHz Intel Core i5 4 GB 1600 MHz DDR3 with graphics card Intel Iris 1536 MB. My OS is OSX Yosemite 10.10.1.

The versions of the compilers used for the benchmarks in my computer are:

  • g++-5 (Homebrew gcc5 5.2.0) 5.2.0.
  • Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn) Target: x86_64-apple-darwin14.0.0 Thread model: posix.

Results

plots/01-sieve.png


plots/02-formatted_read.png


plots/03-trivial-type-copy.png


plots/04-non-trivial-type-copy.png


plots/05-qsort-vs-stl-sort.png