Table of Content

Overview
- Measurements
Test Cases
- Brainfuck
  - bench.b
  - mandel.b
- Base64
- Json
- Matmul
- Primes
Tests Execution
Contribution
- Makefile guide

Overview

The benchmarks follow the criteria:

They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

time spent for the benchmark execution (loading required data and code self-testing are not measured);
memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2021-07-06

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

Verbose (default). Prints the output immediately.
Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
C++/g++	0.921_±0.033	1.48_±00.00 + 0.00_±00.00	16.51_±00.81
Racket (Syntax Objects)	1.433_±0.053	106.75_±00.60 + 0.00_±00.00	30.49_±02.37
C/gcc	1.848_±0.076	0.54_±00.01 + 0.00_±00.00	39.40_±02.59
Rust	1.864_±0.062	2.12_±00.06 + 0.00_±00.00	35.78_±02.93
D/ldc2	1.864_±0.041	2.97_±00.03 + 0.00_±00.00	34.74_±01.31
Kotlin	1.868_±0.048	40.89_±00.11 + 0.48_±00.10	35.11_±02.66
D/gdc	1.943_±0.066	6.30_±00.05 + 0.00_±00.00	41.12_±02.40
OCaml	2.140_±0.055	2.59_±00.04 + 2.51_±00.06	48.78_±02.79
Go/gccgo	2.175_±0.057	20.99_±00.34 + 0.00_±00.00	39.59_±03.11
Go	2.201_±0.075	3.45_±00.01 + 0.00_±00.00	48.46_±01.43
Java	2.216_±0.047	38.08_±00.10 + 0.88_±00.14	43.65_±02.77
Vala/gcc	2.251_±0.068	3.72_±00.06 + 0.00_±00.00	48.29_±01.13
C/clang	2.288_±0.040	0.52_±00.03 + 0.00_±00.00	43.78_±03.77
Racket	2.300_±0.156	115.73_±00.07 + 2.32_±00.52	52.15_±04.19
Nim/gcc	2.364_±0.029	1.78_±00.04 + 0.00_±00.00	47.10_±04.60
V/gcc	2.366_±0.027	0.55_±00.01 + 0.00_±00.00	54.95_±01.10
Nim/clang	2.367_±0.070	2.32_±00.03 + 0.00_±00.00	48.55_±04.79
C#/.NET Core	2.374_±0.107	34.24_±00.03 + 0.01_±00.00	49.14_±04.65
F#/.NET Core	2.403_±0.046	36.74_±00.05 + 0.30_±00.00	42.82_±02.23
Crystal	2.491_±0.116	3.30_±00.05 + 0.00_±00.00	51.84_±03.87
Vala/clang	2.500_±0.048	3.64_±00.03 + 0.00_±00.00	48.54_±02.57
MLton	2.822_±0.058	1.42_±00.04 + 0.25_±00.00	53.42_±03.42
Chez Scheme	2.996_±0.129	24.88_±00.00 + 4.21_±00.03	62.23_±04.69
Julia	3.264_±0.050	197.94_±00.14 + 0.75_±00.03	54.51_±01.22
D/dmd	3.535_±0.134	3.67_±00.04 + 0.00_±00.00	77.38_±03.85
Scala	3.599_±0.080	55.41_±00.10 + 261.15_±04.05	77.90_±03.26
Node.js	4.208_±0.175	31.23_±00.07 + 1.33_±00.00	82.18_±10.81
C#/Mono	4.421_±0.031	20.01_±00.04 + 0.00_±00.00	77.47_±03.78
V/clang	4.676_±0.107	0.50_±00.00 + 0.00_±00.00	102.59_±03.08
Haskell (MArray)	4.885_±0.166	3.72_±00.09 + 1.11_±00.00	95.27_±06.35
Lua/luajit	6.784_±0.057	2.88_±00.02 + 0.00_±00.00	116.90_±02.65
Ruby/truffleruby (--jvm)	7.802_±0.288	580.68_±15.18 + 679.16_±31.19	281.34_±11.68
Ruby/truffleruby	8.594_±0.336	414.21_±00.64 + 541.02_±07.15	209.20_±18.29
Python/pypy	15.296_±0.235	65.41_±00.09 + 45.49_±00.04	301.08_±09.78
Haskell	15.741_±0.410	3.79_±00.06 + 0.82_±00.00	372.84_±21.06
Lua	57.706_±2.164	2.93_±00.03 + 0.00_±00.00	1175.05_±76.72
Ruby (--jit)	61.809_±1.173	14.08_±00.02 + 0.23_±00.00	1150.99_±65.44
Ruby	96.539_±2.652	13.99_±00.05 + 0.00_±00.00	1747.94_±66.65
Ruby/jruby	107.399_±2.175	186.72_±01.62 + 130.70_±19.19	2264.93_±46.81
Elixir	118.727_±4.015	56.04_±01.20 + 0.00_±00.00	2381.04_±129.88
Python	227.596_±5.242	10.34_±00.04 + 0.00_±00.00	5115.22_±223.46
Tcl (FP)	285.989_±6.642	4.25_±00.08 + 0.00_±00.00	5522.61_±280.47
Perl	354.862_±6.481	6.51_±00.05 + 0.00_±00.00	7796.46_±338.55
Tcl (OOP)	544.022_±11.873	4.25_±00.08 + 0.00_±00.00	11120.82_±814.03

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C/gcc	12.423_±0.162	0.50_±00.00 + 1.11_±00.02	271.35_±04.89
Rust	12.586_±0.178	2.03_±00.06 + 0.25_±00.00	284.06_±11.12
C++/g++	12.910_±0.304	3.18_±00.05 + 0.47_±00.00	289.47_±07.83
D/gdc	13.800_±0.242	6.72_±00.05 + 0.52_±00.00	281.84_±12.34
D/ldc2	13.902_±0.141	3.02_±00.05 + 0.77_±00.00	239.36_±05.93
Go	16.117_±0.330	3.40_±00.08 + 1.27_±00.01	339.69_±16.91
Kotlin	17.034_±0.522	40.80_±00.12 + 0.88_±00.32	329.00_±31.56
V/gcc	17.119_±0.737	1.67_±00.04 + 1.03_±00.00	359.04_±34.09
Racket (Syntax Objects)	17.444_±0.120	106.37_±00.15 + 72.70_±00.26	332.19_±21.78
C/clang	18.167_±0.686	0.54_±00.01 + 1.09_±00.03	381.14_±34.92
C#/.NET Core	19.055_±0.224	34.28_±00.02 + 0.96_±00.00	341.42_±04.72
Vala/gcc	19.880_±0.559	3.52_±00.04 + 2.06_±00.02	375.90_±18.11
Vala/clang	22.178_±0.405	3.54_±00.03 + 2.06_±00.03	424.87_±38.99
Nim/clang	22.230_±0.709	2.31_±00.05 + 0.51_±00.00	449.93_±38.24
Nim/gcc	23.202_±0.447	1.81_±00.06 + 0.51_±00.00	414.37_±29.55
Java	23.358_±0.630	38.01_±00.15 + 1.42_±00.13	438.99_±41.36
Go/gccgo	23.445_±0.761	21.30_±00.07 + 1.28_±00.00	443.45_±33.96
Crystal	24.551_±1.035	3.32_±00.01 + 0.46_±00.01	518.51_±24.86
Scala	25.120_±0.293	55.66_±00.09 + 141.27_±00.33	530.51_±11.43
V/clang	26.637_±1.047	1.66_±00.03 + 1.03_±00.00	538.04_±45.94
F#/.NET Core	35.454_±0.288	36.79_±00.07 + 2.03_±00.03	634.62_±03.81
OCaml	37.562_±1.228	3.84_±00.01 + 6.69_±00.39	775.01_±52.89
Racket	38.757_±0.747	115.69_±00.28 + 2.32_±00.26	732.68_±17.03
Node.js	39.350_±0.187	31.24_±00.05 + 5.94_±00.01	846.41_±04.74
Chez Scheme	40.316_±1.217	25.48_±00.04 + 3.64_±00.00	863.75_±64.05
D/dmd	42.491_±0.121	3.70_±00.05 + 0.77_±00.00	963.22_±03.28
C#/Mono	44.502_±0.272	20.11_±00.02 + 0.88_±00.00	980.24_±07.72
MLton	51.981_±1.646	1.41_±00.04 + 4.11_±00.00	1104.45_±82.03
Haskell (MArray)	60.083_±0.844	3.78_±00.10 + 2.50_±00.13	1379.17_±42.02
Python/pypy	64.354_±0.552	65.38_±00.11 + 46.41_±00.02	1229.14_±09.32
Julia	78.738_±1.108	198.50_±00.04 + 0.78_±00.05	1455.76_±70.63
Ruby/truffleruby (--jvm)	134.368_±3.421	584.53_±06.57 + 745.92_±114.67	2794.06_±50.97
Ruby/truffleruby	150.668_±4.223	414.64_±00.37 + 585.68_±11.10	2907.83_±122.54
Haskell	226.291_±4.319	4.04_±00.10 + 25.91_±00.00	4356.11_±160.91
Lua/luajit	247.906_±5.559	2.87_±00.03 + 0.86_±00.00	4766.50_±239.31

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/gcc (aklomp)	0.155_±0.004	1.90_±00.03 + 0.00_±00.00	3.70_±00.39
Rust	1.186_±0.015	2.49_±00.10 + 0.01_±00.00	21.80_±01.84
C/gcc	1.218_±0.036	1.85_±00.00 + 0.00_±00.00	23.94_±01.91
Nim/gcc	1.254_±0.014	2.17_±00.05 + 4.44_±00.00	28.92_±01.10
Crystal	1.612_±0.015	3.72_±00.03 + 1.83_±00.04	39.30_±00.13
Nim/clang	1.634_±0.061	2.67_±00.07 + 4.38_±00.00	36.75_±01.05
D/ldc2	1.967_±0.032	3.41_±00.03 + 3.66_±00.00	35.17_±00.62
V/gcc	1.977_±0.035	2.12_±00.05 + 0.00_±00.00	34.56_±02.67
Ruby	2.019_±0.072	14.45_±00.03 + 56.75_±00.53	46.19_±02.13
Ruby (--jit)	2.132_±0.068	14.43_±00.04 + 57.92_±00.42	41.36_±02.27
Vala/clang	2.142_±0.078	0.00_±00.00 + 0.00_±00.00	46.28_±02.00
D/gdc	2.175_±0.121	7.06_±00.04 + 3.46_±00.00	44.27_±06.07
Java	2.202_±0.066	38.90_±00.21 + 280.75_±09.17	49.87_±01.20
Vala/gcc	2.224_±0.030	0.00_±00.00 + 0.00_±00.00	42.00_±03.33
Scala	2.403_±0.086	53.57_±00.13 + 312.97_±21.36	52.53_±02.64
Kotlin	2.406_±0.019	40.63_±00.16 + 327.63_±29.57	48.30_±01.93
C++/g++ (libcrypto)	2.408_±0.048	5.39_±00.03 + 0.07_±00.00	56.94_±01.66
Go	2.558_±0.017	4.56_±00.04 + 5.61_±00.26	49.88_±00.83
V/clang	2.782_±0.142	1.82_±00.26 + 0.13_±00.13	55.12_±04.19
Node.js	2.792_±0.046	31.72_±00.05 + 36.18_±00.07	58.48_±03.21
Perl (MIME::Base64)	2.901_±0.089	14.12_±00.06 + 0.08_±00.06	62.70_±02.33
PHP	3.062_±0.142	15.76_±00.11 + 0.00_±00.00	64.64_±06.94
Go/gccgo	3.514_±0.009	22.45_±00.34 + 7.19_±00.29	74.67_±01.32
D/dmd	3.825_±0.098	3.80_±00.02 + 3.69_±00.03	72.83_±04.90
Python	4.112_±0.145	10.09_±00.02 + 0.18_±00.00	81.15_±07.44
Tcl	4.896_±0.097	4.83_±00.04 + 0.19_±00.01	87.33_±03.34
Python/pypy	5.049_±0.052	65.29_±00.02 + 45.68_±00.04	97.29_±08.84
F#/.NET Core	5.668_±0.090	37.03_±00.03 + 37.94_±05.46	107.68_±02.72
C#/.NET Core	5.707_±0.069	34.48_±00.09 + 39.49_±06.99	112.02_±02.38
Julia	6.078_±0.079	216.92_±00.09 + 52.82_±10.27	120.10_±06.44
Ruby/truffleruby (--jvm)	6.928_±0.193	568.10_±10.90 + 364.17_±12.04	138.24_±11.06
C#/Mono	7.404_±0.387	20.56_±00.04 + 18.48_±00.02	174.66_±04.52
Ruby/jruby	12.104_±0.464	188.35_±04.81 + 134.19_±05.02	247.75_±11.06
Perl (MIME::Base64::Perl)	16.679_±0.661	15.40_±00.10 + 0.22_±00.06	363.19_±30.65
Ruby/truffleruby	23.147_±0.630	405.72_±01.19 + 305.43_±02.85	441.44_±36.07

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

gason mutates input strings;
simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:

Json

Language	Time, s	Memory, MiB	Energy, J
C++/g++ (DAW JSON Link NoCheck)	0.088_±0.001	113.03_±00.02 + 0.00_±00.00	1.92_±00.02
C++/g++ (simdjson On-Demand)	0.096_±0.001	113.21_±00.05 + 59.81_±00.00	2.07_±00.08
C++/g++ (DAW JSON Link)	0.124_±0.002	112.80_±00.20 + 0.00_±00.00	2.56_±00.05
Rust (Serde Custom)	0.137_±0.003	111.86_±00.08 + 0.00_±00.00	3.15_±00.18
Rust (Serde Typed)	0.151_±0.004	111.87_±00.07 + 11.70_±00.19	2.73_±00.18
C++/g++ (gason)	0.158_±0.002	113.00_±00.04 + 96.74_±00.06	4.07_±00.20
C++/g++ (simdjson DOM)	0.172_±0.003	113.28_±00.03 + 176.60_±00.00	3.68_±00.16
C++/g++ (RapidJSON)	0.227_±0.006	113.03_±00.03 + 128.77_±00.06	5.02_±00.42
D/ldc2 (Mir Amazon's Ion DOM)	0.229_±0.006	112.70_±00.04 + 16.26_±00.00	4.97_±00.44
D/ldc2 (Mir Asdf DOM)	0.243_±0.004	112.67_±00.07 + 61.34_±00.01	5.26_±00.16
C++/g++ (RapidJSON Precise)	0.290_±0.009	113.07_±00.02 + 128.71_±00.05	6.29_±00.56
C++/g++ (Boost.JSON)	0.529_±0.020	113.13_±00.01 + 435.76_±00.06	11.13_±00.67
C++/g++ (RapidJSON SAX)	0.589_±0.009	112.79_±00.02 + 0.00_±00.00	10.30_±00.40
C++/g++ (RapidJSON SAX Precise)	0.638_±0.025	112.82_±00.03 + 0.00_±00.00	13.41_±01.12
Go (jsoniter)	0.695_±0.023	231.35_±00.02 + 13.59_±00.28	14.49_±01.06
Crystal (Schema)	0.733_±0.011	113.73_±00.03 + 42.72_±00.03	15.30_±00.17
Crystal (Pull)	0.764_±0.019	113.71_±00.04 + 18.18_±00.03	15.22_±01.41
Java (DSL-JSON)	0.803_±0.025	264.04_±00.14 + 311.38_±02.85	20.76_±00.82
V/clang	0.847_±0.020	111.15_±00.01 + 496.21_±00.06	19.21_±01.32
Rust (Serde Untyped)	0.854_±0.029	111.89_±00.08 + 839.98_±00.00	17.32_±01.54
Python/pypy	0.862_±0.012	285.48_±00.09 + 121.42_±00.01	16.56_±00.28
Node.js	0.877_±0.025	251.02_±00.03 + 78.29_±00.74	23.02_±00.55
Julia (JSON3)	0.960_±0.024	368.78_±00.22 + 373.25_±00.20	18.56_±00.83
V/gcc	0.969_±0.043	111.14_±00.01 + 496.21_±00.00	20.08_±02.07
Perl (Cpanel::JSON::XS)	0.986_±0.010	124.67_±00.04 + 402.78_±00.03	23.49_±00.48
Crystal	1.030_±0.023	113.72_±00.02 + 396.45_±00.01	22.79_±00.49
C#/.NET Core (System.Text.Json)	1.060_±0.043	479.22_±00.09 + 138.78_±00.06	22.38_±02.34
Go	1.097_±0.036	117.39_±00.05 + 83.29_±00.11	24.76_±01.61
Nim/clang (Packedjson)	1.103_±0.041	112.47_±00.10 + 294.04_±00.13	25.56_±01.59
PHP	1.277_±0.046	124.99_±00.14 + 682.01_±00.00	25.92_±02.27
Nim/gcc (Packedjson)	1.380_±0.057	112.05_±00.11 + 293.91_±00.00	28.11_±02.64
Go/gccgo	1.450_±0.030	142.23_±06.08 + 96.03_±00.17	33.10_±01.45
C++/g++ (json-c)	1.532_±0.030	112.93_±00.19 + 1216.23_±00.11	36.83_±01.14
Clojure	1.609_±0.044	473.39_±03.17 + 576.83_±05.74	46.18_±01.78
Nim/clang	1.610_±0.055	112.41_±00.08 + 925.03_±00.06	37.65_±01.70
C#/.NET Core	1.730_±0.058	487.98_±00.04 + 294.07_±00.00	37.47_±04.26
CPython (UltraJSON)	1.780_±0.023	121.82_±00.03 + 549.14_±01.04	41.90_±00.80
C++/g++ (Nlohmann)	1.809_±0.011	113.15_±00.02 + 447.88_±00.00	34.77_±01.10
Nim/gcc	1.844_±0.070	112.11_±00.04 + 923.29_±00.06	37.81_±03.36
Python	1.848_±0.014	120.09_±00.03 + 377.21_±00.00	38.12_±01.17
C#/Mono	2.236_±0.089	476.19_±00.08 + 0.00_±00.00	47.40_±05.36
D/gdc	2.283_±0.028	116.61_±00.04 + 600.39_±00.00	53.37_±00.83
Haskell	2.291_±0.051	4.44_±00.05 + 4.68_±00.03	43.10_±02.05
Ruby	2.426_±0.058	124.00_±00.02 + 410.69_±00.01	57.63_±01.08
Ruby (YAJL)	2.443_±0.047	123.96_±00.02 + 283.34_±00.00	44.94_±01.47
Scala (uPickle)	2.516_±0.053	284.73_±00.09 + 762.67_±23.36	59.39_±04.12
F#/.NET Core (System.Text.Json)	2.577_±0.083	485.37_±00.05 + 453.88_±02.45	55.03_±05.61
D/ldc2	2.585_±0.057	112.92_±00.06 + 680.09_±00.04	53.34_±01.70
Ruby (--jit)	2.589_±0.022	124.01_±00.06 + 410.80_±00.04	50.43_±00.51
Rust (jq)	3.584_±0.035	113.83_±00.02 + 777.86_±00.90	81.39_±00.50
Ruby/jruby	3.876_±0.084	453.42_±10.48 + 1412.22_±28.09	108.53_±06.77
C++/g++ (Boost.PropertyTree)	4.483_±0.118	112.95_±00.04 + 1440.06_±00.00	101.16_±06.49
D/dmd	5.098_±0.075	113.65_±00.06 + 680.25_±00.01	102.46_±04.22
Vala/gcc	5.473_±0.217	0.00_±00.00 + 0.00_±00.00	99.96_±06.82
Vala/clang	5.602_±0.056	0.00_±00.00 + 0.00_±00.00	101.12_±00.60
Perl (JSON::Tiny)	12.605_±0.279	125.30_±00.07 + 528.63_±00.10	231.12_±05.85
Ruby/truffleruby (--jvm)	16.962_±0.464	721.72_±12.93 + 1648.53_±20.81	479.86_±14.66
Ruby/truffleruby	40.369_±0.806	738.03_±01.52 + 2269.36_±70.04	834.73_±65.49

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.081_±0.002	6.70_±00.06 + 55.90_±00.12	4.58_±00.10
Python (NumPy)	0.107_±0.003	27.62_±00.10 + 57.64_±00.06	6.15_±00.19
Nim/gcc (Arraymancer)	0.158_±0.029	5.58_±00.10 + 57.28_±00.12	8.16_±00.82
Nim/clang (Arraymancer)	0.189_±0.019	6.44_±00.14 + 57.38_±00.07	9.07_±00.70
C++/g++ (Eigen)	0.203_±0.007	3.60_±00.08 + 85.24_±00.01	4.59_±00.22
Julia (threads: 8)	0.207_±0.004	251.36_±00.42 + 53.11_±00.12	11.03_±00.42
Java (ND4J)	0.232_±0.023	145.25_±01.16 + 87.50_±00.13	10.77_±00.94
Julia (threads: 1)	0.552_±0.013	251.36_±00.14 + 53.60_±00.09	12.89_±00.44
Julia (no BLAS)	1.220_±0.018	216.72_±00.33 + 51.63_±00.00	30.49_±00.74
D/ldc2	1.991_±0.007	3.52_±00.12 + 70.11_±00.13	44.85_±00.56
D/dmd	2.135_±0.024	3.59_±00.10 + 70.12_±00.01	48.46_±00.73
D/gdc	2.149_±0.057	6.63_±00.06 + 70.71_±00.02	48.48_±03.20
Java	3.244_±0.025	38.55_±00.15 + 80.66_±00.55	76.29_±01.27
Vala/gcc	3.271_±0.021	0.00_±00.00 + 0.00_±00.00	79.07_±01.08
C/gcc	3.357_±0.055	2.01_±00.04 + 68.06_±00.00	77.55_±01.22
Nim/gcc	3.372_±0.017	2.56_±00.05 + 65.48_±00.00	81.10_±03.55
Vala/clang	3.426_±0.024	0.00_±00.00 + 0.00_±00.00	74.62_±00.92
Rust	3.445_±0.016	2.71_±00.12 + 68.32_±00.00	74.65_±00.66
Scala	3.449_±0.027	55.68_±00.06 + 153.92_±00.12	82.04_±01.31
Nim/clang	3.469_±0.050	3.07_±00.02 + 66.00_±00.00	74.45_±00.98
Go	3.543_±0.023	3.80_±00.06 + 73.36_±00.09	80.52_±03.47
Go/gccgo	3.564_±0.086	21.59_±00.18 + 72.83_±00.16	78.12_±02.04
Swift	3.646_±0.088	145.20_±00.05 + 60.64_±00.03	86.24_±03.09
Crystal	3.728_±0.104	4.15_±00.12 + 59.64_±00.11	79.90_±06.27
V/gcc	3.757_±0.127	1.87_±00.05 + 69.35_±00.00	80.73_±07.42
Kotlin	3.800_±0.048	38.80_±00.22 + 80.71_±00.38	88.40_±02.97
Node.js	4.925_±0.088	35.33_±00.09 + 71.08_±00.44	89.30_±04.41
Python/pypy	6.541_±0.073	65.85_±00.13 + 69.18_±00.04	114.00_±03.02
C#/.NET Core	7.373_±0.141	33.97_±00.11 + 69.11_±00.00	133.57_±05.53
C#/Mono	10.775_±0.202	20.06_±00.06 + 69.06_±00.02	240.98_±06.98
V/clang	14.013_±0.187	2.20_±00.04 + 69.35_±00.00	237.08_±15.02
Ruby/truffleruby	46.906_±0.541	573.59_±00.56 + 684.43_±03.60	1118.10_±14.38
Ruby/truffleruby (--jvm)	68.521_±0.596	650.54_±12.48 + 693.22_±91.72	1633.85_±53.37
Ruby	220.767_±2.987	15.22_±00.05 + 68.61_±00.03	4856.07_±55.59
Ruby (--jit)	225.230_±3.868	15.20_±00.02 + 68.91_±00.03	4870.24_±53.87
Python	241.153_±8.661	10.53_±00.02 + 68.58_±00.00	4955.07_±173.31
Tcl	358.681_±3.162	7.26_±00.03 + 400.44_±00.00	6919.11_±392.92
Perl	405.584_±6.809	9.00_±00.06 + 599.62_±00.04	9364.17_±679.44
Ruby/jruby	500.132_±7.243	263.98_±04.55 + 1004.45_±37.01	10848.95_±221.31

Primes

Testing:

generating primes using the optimized sieve of Atkin;
prefix search for their decimal numbers using Trie data structure.

Primes

Language	Time, s	Memory, MiB	Energy, J
Crystal	0.161_±0.002	3.36_±00.04 + 88.63_±01.41	3.94_±00.16
Java	0.178_±0.002	36.02_±00.30 + 106.19_±01.54	5.60_±00.32
Rust	0.219_±0.004	2.40_±00.06 + 72.88_±00.00	4.34_±00.03
Node.js	0.286_±0.003	31.00_±00.09 + 175.57_±00.20	8.05_±00.04
C++/g++	0.288_±0.012	3.19_±00.03 + 125.23_±00.00	5.43_±00.35
Scala	0.423_±0.016	55.71_±00.12 + 246.59_±02.58	14.71_±00.50
Python/pypy	0.963_±0.039	64.83_±00.13 + 248.84_±00.14	22.58_±00.69
Ruby/truffleruby	1.254_±0.028	397.95_±01.44 + 446.35_±01.65	33.48_±02.20
Ruby/truffleruby (--jvm)	1.717_±0.045	577.56_±19.73 + 370.37_±41.51	63.60_±03.08
Ruby (--jit)	2.396_±0.029	14.12_±00.07 + 150.16_±00.00	64.55_±02.34
Ruby	2.619_±0.066	14.01_±00.08 + 149.86_±00.03	46.43_±04.26
Ruby/jruby	2.876_±0.080	190.72_±06.08 + 362.10_±07.62	78.15_±02.13
Python	5.477_±0.123	10.16_±00.05 + 236.08_±00.60	114.04_±06.36

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bullseye/sid

Language	Version
.NET Core	5.0.204
C#/.NET Core	3.9.0-6.21160.10 (59eedc33)
C#/Mono	6.12.0.122
C/clang	11.0.1
C/gcc	10.2.1
Chez Scheme	9.5.4
Clojure	"1.10.3"
Crystal	1.0.0
D/dmd	v2.097.0
D/gdc	10.2.1
D/ldc2	1.26.0
Elixir	1.10.3
F#/.NET Core	11.3.2.0 for F# 5.0
Go	go1.16.5
Go/gccgo	10.2.1
Haskell	9.0.1
Java	16.0.1
Julia	v"1.6.1"
Kotlin	1.5.20
Lua	Lua 5.4
Lua/luajit	LuaJIT 2.1.0-beta3
MLton	20210117
Nim	1.4.8
Node.js	v16.4.0
OCaml	4.11.1
PHP	7.4.15
Perl	v5.32.1
Python	3.9.2
Python/pypy	7.3.5-final0 for Python 3.7.10
Racket	"8.1"
Ruby	3.0.1p64
Ruby/jruby	9.2.19.0
Ruby/truffleruby	21.1.0
Rust	1.53.0
Scala	3.0.0
Swift	swift-5.4.2-RELEASE
Tcl	8.6
V	0.2.2 0ac0ab6
Vala	0.48.17

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

versions (print installed language versions);
shell (start the shell);
brainfuck bench (build and run Brainfuck bench.b benchmarks);
brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
base64 (build and run Base64 benchmarks);
json (build and run Json benchmarks);
matmul (build and run Matmul benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simlify Docker usage:

make build (build the image);
make versions (run the image with the versions command);
make shell (run the image with the `shell' command);
make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

NumPy for matmul tests (Debian package python3-numpy).
UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

Boost for JSON tests (Debian package libboost-dev).
JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

Coursier for downloading Maven artifacts.

For Lua:

LuaRocks for installing dependencies (Debian package luarocks).

For Haskell:

network for TCP connectivity between the tests and the test runner.
raw-strings-qq for raw string literals used in tests.

For Perl:

cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

The test runner listens on localhost:9001;
All messages are sent using TCP sockets closed immediately after the message has been sent;
There are two messages sent from a test (it establishes the measurement boundary):
1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

append a new target (the final binary location) into executables variable;
append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

append a new target (the final artifact location) into artifacts variable;
append the proper target rule to compile the test;
append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

append run[<script_file>] into all_runners variable;
append run[<script_file>] rule to run the test.

License

fulcanelly/benchmarks

ErrorLooks like something went wrong!

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 12

Packages 0

Uh oh!

Languages

Error
Looks like something went wrong!

Packages