Table of Content

Overview
- Measurements
Test Cases
- Brainfuck
  - bench.b
  - mandel.b
- Base64
- Json
- Matmul
- Primes
Tests Execution
Contribution

Overview

The benchmarks follow the criteria:

They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

time spent for the benchmark execution (loading required data and code self-testing are not measured);
memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2022-12-10

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

Verbose (default). Prints the output immediately.
Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
Racket (Syntax Objects)	1.297_±0.000	110.95_±00.38 + 0.00_±00.00	47.09_±00.05
C++/g++	1.300_±0.001	1.84_±00.02 + 0.00_±00.00	51.96_±00.05
Rust	1.545_±0.001	0.94_±00.02 + 0.00_±00.00	61.73_±00.09
Java	1.651_±0.000	37.60_±00.21 + 1.55_±00.33	64.40_±00.14
V/gcc	1.654_±0.000	1.84_±00.01 + 0.00_±00.00	64.02_±00.14
D/ldc2	1.657_±0.000	3.02_±00.04 + 0.00_±00.00	66.26_±00.09
D/gdc	1.661_±0.000	6.59_±00.02 + 0.00_±00.00	68.09_±00.48
C/gcc	1.667_±0.000	0.89_±00.01 + 0.00_±00.00	64.70_±00.13
C/clang	1.670_±0.000	0.89_±00.01 + 0.00_±00.00	65.33_±00.38
Kotlin/JVM	1.683_±0.009	41.55_±00.07 + 0.43_±00.09	65.59_±00.30
C++/clang++	1.742_±0.000	1.61_±00.01 + 0.00_±00.00	67.71_±00.22
Zig	1.835_±0.000	0.91_±00.02 + 0.00_±00.00	71.17_±00.13
Go	1.905_±0.000	3.11_±00.04 + 0.00_±00.00	73.82_±00.15
OCaml	1.912_±0.001	2.87_±00.02 + 1.98_±00.00	85.40_±00.82
Chez Scheme	1.933_±0.000	24.84_±00.03 + 4.34_±00.06	79.82_±00.12
F#/.NET Core	1.952_±0.000	108.79_±00.11 + 0.60_±00.00	80.12_±00.23
Nim/gcc	1.962_±0.001	2.00_±00.03 + 0.00_±00.00	77.19_±00.80
C#/.NET Core	1.979_±0.000	104.35_±00.10 + 0.21_±00.00	81.29_±00.08
Racket	2.016_±0.013	92.74_±00.18 + 0.00_±00.00	81.31_±00.66
Vala/gcc	2.090_±0.001	4.34_±00.03 + 0.00_±00.00	78.35_±00.12
Nim/clang	2.111_±0.000	2.29_±00.01 + 0.00_±00.00	81.86_±00.13
Go/gccgo	2.176_±0.000	24.22_±00.05 + 0.00_±00.00	85.32_±00.77
Vala/clang	2.266_±0.000	4.31_±00.04 + 0.00_±00.00	85.08_±00.08
V/clang	2.356_±0.025	1.88_±00.02 + 0.00_±00.00	101.86_±01.18
Crystal	2.363_±0.000	2.90_±00.07 + 0.00_±00.00	94.12_±00.63
Julia	2.581_±0.006	207.32_±00.06 + 0.65_±00.02	99.89_±00.47
MLton	2.608_±0.000	0.98_±00.02 + 0.92_±00.08	104.55_±01.65
C#/Mono	2.685_±0.000	24.98_±00.11 + 0.00_±00.00	107.55_±00.29
D/dmd	3.218_±0.001	3.52_±00.03 + 0.00_±00.00	123.14_±01.32
Scala	3.228_±0.010	66.81_±00.18 + 166.50_±25.97	135.27_±01.02
Haskell (MArray)	3.851_±0.000	4.90_±01.05 + 5.80_±01.02	154.33_±00.11
Haskell (FP)	3.889_±0.003	5.92_±00.05 + 4.93_±00.00	165.24_±00.42
Node.js	4.150_±0.002	42.88_±00.04 + 0.31_±00.00	165.97_±00.46
Swift	5.450_±0.000	15.28_±00.83 + 0.00_±00.00	204.49_±00.28
Ruby/truffleruby (--jvm)	6.413_±0.133	409.74_±02.52 + 674.12_±70.76	309.38_±09.75
Lua/luajit	6.491_±0.016	2.43_±00.01 + 0.00_±00.00	259.21_±00.50
Ruby/truffleruby	7.043_±0.033	303.07_±00.24 + 359.89_±06.85	332.99_±01.45
Python/pypy	12.191_±0.015	60.78_±00.15 + 30.13_±00.02	510.36_±02.22
Idris	15.762_±0.045	20.63_±00.07 + 8.82_±00.06	680.89_±05.22
Ruby (--jit)	40.816_±0.150	270.58_±00.01 + 0.25_±00.01	1635.12_±05.64
Elixir	43.256_±0.087	71.45_±01.13 + 0.00_±00.00	1928.27_±14.75
Lua	50.611_±0.046	2.27_±00.04 + 0.00_±00.00	1955.56_±02.08
Ruby	87.260_±1.288	13.96_±00.04 + 0.00_±00.00	3609.87_±41.40
Ruby/jruby	87.322_±1.059	189.47_±01.28 + 116.82_±02.04	3849.08_±46.49
Python	195.534_±1.465	10.41_±00.04 + 0.00_±00.00	7834.28_±87.23
Tcl (FP)	271.758_±1.871	3.81_±00.09 + 0.00_±00.00	11395.20_±79.30
Perl	315.489_±2.478	7.14_±00.03 + 0.00_±00.00	12703.58_±98.49
Tcl (OOP)	524.988_±2.867	3.96_±00.02 + 0.00_±00.00	22223.01_±139.78

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C++/g++	10.145_±0.016	1.83_±00.02 + 2.39_±00.04	410.27_±02.84
C/gcc	14.201_±0.004	0.89_±00.02 + 0.81_±00.03	557.98_±02.04
Racket (Syntax Objects)	14.285_±0.030	112.31_±00.49 + 71.15_±00.08	566.11_±01.26
Kotlin/JVM	14.315_±0.027	41.63_±00.10 + 1.19_±00.17	586.70_±01.57
D/ldc2	14.920_±0.020	3.04_±00.02 + 0.84_±00.04	598.76_±02.58
Zig	15.007_±0.008	0.91_±00.01 + 1.41_±00.00	592.52_±02.01
C/clang	15.128_±0.013	0.89_±00.02 + 0.87_±00.09	647.86_±05.03
Rust	15.262_±0.021	0.91_±00.01 + 1.10_±00.03	598.93_±02.53
D/gdc	15.371_±0.073	6.57_±00.02 + 1.36_±00.02	644.91_±02.18
C++/clang++	15.573_±0.006	1.62_±00.01 + 1.98_±00.05	636.74_±04.30
V/gcc	15.903_±0.340	1.85_±00.04 + 1.19_±00.01	632.63_±15.82
Crystal	15.924_±0.168	2.91_±00.03 + 0.60_±00.02	663.38_±11.25
C#/.NET Core	16.346_±0.019	104.49_±00.05 + 0.88_±00.00	682.67_±04.50
Swift	18.069_±0.067	16.12_±00.02 + 0.00_±00.00	727.04_±03.39
Vala/gcc	18.476_±0.004	4.22_±00.04 + 1.45_±00.02	691.94_±00.47
Java	18.659_±0.023	37.73_±00.23 + 2.05_±00.31	744.47_±01.69
Go	19.488_±0.018	3.07_±00.04 + 1.28_±00.00	730.68_±00.89
Go/gccgo	19.922_±0.005	24.25_±00.12 + 1.27_±00.01	819.93_±00.81
Vala/clang	20.250_±0.007	4.26_±00.03 + 1.46_±00.12	783.48_±00.44
Scala	20.975_±0.012	67.25_±00.16 + 141.43_±00.12	881.11_±01.30
Nim/gcc	22.399_±0.278	2.00_±00.02 + 0.52_±00.00	901.24_±14.68
V/clang	23.011_±0.086	1.87_±00.01 + 1.18_±00.00	977.86_±04.52
Nim/clang	24.833_±0.482	2.28_±00.01 + 0.51_±00.00	987.29_±23.00
OCaml	26.219_±0.039	3.45_±00.02 + 3.74_±00.12	1247.55_±10.90
Chez Scheme	27.790_±0.046	25.31_±00.03 + 3.92_±00.00	1215.67_±02.96
Node.js	28.194_±0.527	42.91_±00.05 + 6.17_±00.19	1178.23_±21.84
Julia	29.584_±0.036	207.95_±00.09 + 0.49_±00.01	1126.40_±06.02
C#/Mono	31.521_±0.047	25.01_±00.04 + 0.82_±00.00	1337.07_±04.01
F#/.NET Core	34.950_±0.035	108.86_±00.10 + 2.22_±00.06	1442.68_±02.23
Lua/luajit	35.086_±0.039	2.44_±00.03 + 0.44_±00.00	1401.10_±01.44
Racket	36.847_±0.435	93.33_±00.71 + 0.25_±00.08	1621.75_±16.77
Haskell (MArray)	37.327_±0.011	5.72_±00.02 + 6.11_±00.00	1520.44_±07.50
D/dmd	37.414_±0.004	3.50_±00.05 + 0.97_±00.03	1365.92_±05.55
MLton	44.858_±0.031	1.61_±00.09 + 4.11_±00.00	1924.75_±13.80
Python/pypy	48.295_±0.176	60.86_±00.03 + 30.54_±00.28	2043.82_±21.99
Ruby/truffleruby (--jvm)	66.508_±0.063	408.76_±03.32 + 611.12_±29.67	2784.36_±07.21
Idris	67.460_±0.094	21.96_±00.06 + 9.56_±00.02	2915.79_±10.83
Haskell (FP)	79.443_±0.133	5.82_±00.04 + 75.83_±00.00	3315.86_±27.08
Ruby/truffleruby	81.328_±0.292	302.54_±01.09 + 263.13_±20.03	3816.09_±42.42

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/gcc (aklomp)	0.099_±0.000	2.07_±00.02 + 0.00_±00.00	4.64_±00.06
C/clang (aklomp)	0.099_±0.001	2.07_±00.04 + 0.00_±00.00	4.59_±00.07
PHP	0.106_±0.000	17.31_±00.09 + 0.00_±00.00	4.78_±00.07
Rust	0.970_±0.000	2.08_±00.02 + 0.21_±00.03	39.37_±00.21
C/clang	0.997_±0.000	2.00_±00.07 + 0.00_±00.00	36.74_±00.21
C/gcc	1.012_±0.000	2.07_±00.03 + 0.00_±00.00	37.18_±00.48
D/ldc2	1.037_±0.001	3.40_±00.27 + 3.61_±00.19	43.16_±00.52
Nim/gcc	1.082_±0.001	1.68_±00.05 + 5.23_±00.09	42.43_±00.42
Nim/clang	1.095_±0.001	1.90_±00.04 + 5.27_±00.07	43.55_±00.35
Crystal	1.102_±0.001	3.49_±00.04 + 1.18_±00.04	44.99_±00.15
V/gcc	1.349_±0.000	2.39_±00.03 + 2376.18_±00.26	50.68_±00.01
V/clang	1.380_±0.001	2.38_±00.01 + 2376.79_±00.86	52.53_±00.10
Ruby (--jit)	1.386_±0.010	271.00_±00.04 + 90.38_±10.11	54.11_±00.59
Ruby	1.388_±0.001	14.65_±00.01 + 58.72_±00.08	53.68_±00.33
Java	1.511_±0.013	39.32_±00.04 + 254.12_±22.32	62.27_±00.93
Scala	1.606_±0.055	64.90_±00.28 + 330.26_±14.92	68.84_±01.17
Kotlin/JVM	1.636_±0.028	42.55_±00.12 + 249.05_±04.14	69.13_±01.45
Vala/gcc	1.643_±0.001	5.54_±00.05 + 0.13_±00.02	62.89_±00.31
Vala/clang	1.644_±0.000	5.50_±00.05 + 0.15_±00.02	63.06_±00.69
C++/clang++ (libcrypto)	1.711_±0.002	5.09_±00.05 + 0.71_±00.03	68.44_±00.89
C++/g++ (libcrypto)	1.712_±0.003	5.70_±00.10 + 0.72_±00.04	68.35_±00.55
Node.js	1.724_±0.003	42.61_±00.06 + 43.70_±00.10	69.32_±00.88
Go	1.877_±0.001	4.25_±00.03 + 4.01_±00.11	80.54_±00.28
Perl (MIME::Base64)	1.951_±0.017	14.90_±00.05 + 0.13_±00.05	78.31_±00.82
F#/.NET Core	2.342_±0.043	109.31_±00.06 + 46.62_±03.14	87.36_±00.83
D/gdc	2.436_±0.004	7.55_±00.04 + 3.35_±00.00	106.92_±00.60
C#/.NET Core	2.647_±0.058	104.54_±00.13 + 47.83_±06.26	96.55_±02.21
Python	2.711_±0.002	10.28_±00.05 + 0.60_±00.01	105.56_±00.92
Zig	2.951_±0.000	1.58_±00.02 + 0.00_±00.00	130.14_±01.64
D/dmd	3.026_±0.001	4.15_±00.04 + 3.35_±00.00	129.20_±01.47
Python/pypy	3.582_±0.002	60.86_±00.01 + 31.50_±00.15	157.11_±01.03
Tcl	3.632_±0.004	5.10_±00.02 + 0.00_±00.00	146.59_±01.31
Go/gccgo	3.714_±0.003	25.25_±00.16 + 7.84_±00.38	166.93_±00.68
C#/Mono	4.712_±0.019	25.65_±00.17 + 18.70_±00.02	194.51_±00.78
Julia	5.051_±0.005	226.05_±00.04 + 42.61_±00.43	189.58_±03.14
Ruby/truffleruby (--jvm)	5.136_±0.010	407.71_±02.03 + 211.28_±15.72	240.17_±02.78
Ruby/jruby	10.708_±0.121	186.69_±00.85 + 79.64_±00.52	421.28_±03.40
Perl (MIME::Base64::Perl)	13.619_±0.141	16.21_±00.08 + 0.23_±00.05	572.56_±03.68
Ruby/truffleruby	15.536_±0.010	302.86_±00.68 + 263.68_±04.10	649.08_±08.68

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

gason mutates input strings;
simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:

Json

Language	Time, s	Memory, MiB	Energy, J
C++/clang++ (simdjson On-Demand)	0.066_±0.000	112.52_±00.02 + 60.11_±00.03	2.71_±00.01
C++/g++ (simdjson On-Demand)	0.066_±0.000	113.29_±00.32 + 60.11_±00.29	2.72_±00.02
C++/clang++ (DAW JSON Link NoCheck)	0.070_±0.001	112.38_±00.06 + 0.00_±00.00	2.95_±00.03
C++/g++ (DAW JSON Link NoCheck)	0.085_±0.000	113.23_±00.05 + 0.00_±00.00	3.36_±00.01
C++/clang++ (DAW JSON Link)	0.086_±0.001	112.41_±00.06 + 0.00_±00.00	3.61_±00.05
C++/g++ (DAW JSON Link)	0.101_±0.000	113.23_±00.03 + 0.00_±00.00	4.14_±00.02
Rust (Serde Typed)	0.102_±0.000	111.62_±00.07 + 12.02_±00.13	4.29_±00.03
Rust (Serde Custom)	0.108_±0.000	111.35_±00.03 + 0.00_±00.00	4.45_±00.02
C++/g++ (gason)	0.133_±0.000	113.15_±00.09 + 96.96_±00.04	5.26_±00.01
C++/g++ (simdjson DOM)	0.133_±0.000	113.27_±00.32 + 176.90_±00.30	5.70_±00.02
C++/clang++ (gason)	0.144_±0.000	112.35_±00.06 + 96.97_±00.00	5.82_±00.01
C++/clang++ (simdjson DOM)	0.144_±0.000	112.47_±00.04 + 177.15_±00.00	6.15_±00.02
D/ldc2 (Mir Asdf DOM)	0.145_±0.000	112.78_±00.05 + 61.27_±00.03	5.98_±00.03
C++/g++ (RapidJSON)	0.168_±0.000	113.23_±00.05 + 122.23_±00.59	7.01_±00.07
Scala (jsoniter-scala)	0.175_±0.001	288.19_±00.12 + 25.08_±00.53	9.59_±00.05
C++/clang++ (RapidJSON)	0.230_±0.000	112.41_±00.07 + 128.97_±00.03	9.57_±00.04
D/ldc2 (Mir Amazon's Ion DOM)	0.236_±0.000	112.89_±00.02 + 80.70_±00.00	9.85_±00.07
Go (Sonic)	0.237_±0.002	119.77_±00.09 + 111.84_±00.06	10.21_±00.10
C++/g++ (RapidJSON Precise)	0.243_±0.001	113.23_±00.07 + 128.88_±00.06	10.27_±00.09
Go (goccy/go-json)	0.271_±0.000	115.83_±00.09 + 112.15_±00.19	10.63_±00.02
C++/clang++ (RapidJSON Precise)	0.313_±0.000	112.41_±00.01 + 129.00_±00.00	13.43_±00.08
C/clang (yajl)	0.368_±0.000	110.86_±00.01 + 0.00_±00.00	16.11_±00.17
C/gcc (yajl)	0.371_±0.001	110.87_±00.05 + 0.00_±00.00	15.98_±00.04
Zig	0.387_±0.001	110.92_±00.02 + 12.18_±00.00	19.13_±00.06
Nim/clang (jsony)	0.401_±0.000	111.40_±00.01 + 42.74_±00.01	17.16_±00.05
C++/g++ (Boost.JSON)	0.401_±0.000	113.30_±00.05 + 435.95_±00.07	16.68_±00.06
C++/g++ (RapidJSON SAX)	0.404_±0.001	113.02_±00.02 + 0.00_±00.00	16.62_±00.05
Nim/gcc (jsony)	0.408_±0.001	111.08_±00.03 + 42.77_±00.04	17.46_±00.08
C++/clang++ (Boost.JSON)	0.421_±0.001	112.42_±00.04 + 436.31_±00.00	17.73_±00.11
C++/g++ (RapidJSON SAX Precise)	0.451_±0.001	113.02_±00.01 + 0.00_±00.00	19.37_±00.14
Go (jsoniter)	0.517_±0.003	230.90_±00.03 + 1.35_±00.18	21.68_±00.15
Node.js	0.552_±0.002	152.51_±00.05 + 187.99_±00.47	25.65_±00.06
C#/.NET Core (System.Text.Json)	0.569_±0.001	561.19_±00.19 + 139.73_±00.01	25.20_±00.08
Rust (Serde Untyped)	0.578_±0.001	111.60_±00.03 + 839.97_±00.00	23.67_±00.11
C++/clang++ (RapidJSON SAX)	0.600_±0.000	194.64_±00.01 + 0.00_±00.00	23.95_±00.06
Java (DSL-JSON)	0.603_±0.024	259.68_±00.36 + 190.97_±02.88	30.71_±00.70
V/clang	0.631_±0.000	111.45_±00.03 + 496.21_±00.00	26.92_±00.13
V/gcc	0.637_±0.001	111.44_±00.04 + 496.21_±00.03	26.60_±00.16
Crystal (Pull)	0.655_±0.004	113.07_±00.01 + 18.41_±00.03	28.93_±00.44
Crystal (Schema)	0.676_±0.003	113.04_±00.04 + 48.84_±00.04	29.69_±00.38
Python/pypy	0.688_±0.004	280.92_±00.08 + 125.74_±00.04	29.52_±00.25
C++/clang++ (RapidJSON SAX Precise)	0.736_±0.002	194.60_±00.04 + 0.00_±00.00	30.11_±00.22
Julia (JSON3)	0.765_±0.003	443.49_±00.12 + 252.04_±02.56	31.12_±00.16
Perl (Cpanel::JSON::XS)	0.778_±0.005	125.51_±00.07 + 402.87_±00.01	32.36_±00.19
Go	0.856_±0.000	117.05_±00.14 + 79.91_±00.13	34.96_±00.15
PHP	0.958_±0.002	126.54_±00.05 + 682.21_±00.00	39.86_±00.14
Crystal	0.966_±0.002	113.04_±00.02 + 392.50_±00.02	41.41_±00.39
Nim/clang (Packedjson)	0.976_±0.001	112.15_±00.01 + 294.42_±00.00	40.29_±00.34
Nim/gcc (Packedjson)	0.995_±0.001	111.83_±00.02 + 294.42_±00.00	41.30_±00.28
C#/.NET Core	1.063_±0.007	569.35_±00.12 + 295.46_±00.14	47.16_±00.46
C++/g++ (json-c)	1.176_±0.005	113.39_±00.08 + 1216.08_±00.00	48.91_±00.17
C++/clang++ (json-c)	1.181_±0.007	112.68_±00.06 + 1216.08_±00.00	49.16_±00.56
Clojure	1.215_±0.015	460.72_±06.56 + 546.03_±53.52	64.32_±00.42
Go/gccgo	1.269_±0.002	138.83_±00.11 + 83.59_±00.17	53.56_±00.42
C++/clang++ (Nlohmann)	1.272_±0.007	112.52_±00.07 + 360.17_±00.03	54.07_±00.55
Nim/gcc	1.336_±0.002	111.83_±00.03 + 919.88_±00.00	55.94_±00.16
Nim/clang	1.344_±0.002	112.15_±00.03 + 925.03_±00.00	55.50_±00.20
CPython (UltraJSON)	1.444_±0.002	122.17_±00.08 + 544.51_±01.81	53.21_±00.20
Ruby	1.449_±0.008	123.97_±00.04 + 263.13_±00.02	60.95_±00.39
Ruby (--jit)	1.449_±0.006	380.63_±00.03 + 263.24_±00.01	61.11_±00.37
C++/g++ (Nlohmann)	1.507_±0.002	113.38_±00.05 + 448.03_±00.03	61.19_±00.43
Python	1.525_±0.003	120.25_±00.10 + 375.17_±00.02	59.36_±00.18
F#/.NET Core (System.Text.Json)	1.601_±0.004	570.59_±00.09 + 232.18_±06.44	69.79_±00.32
Ruby (YAJL)	1.965_±0.005	123.81_±00.04 + 282.68_±00.01	82.27_±00.88
D/ldc2	2.082_±0.001	112.86_±00.06 + 680.20_±00.05	86.06_±00.50
C#/Mono	2.312_±0.040	252.46_±00.07 + 31.53_±00.01	96.95_±01.22
Haskell	2.539_±0.005	116.59_±00.75 + 697.45_±00.73	107.60_±01.21
Rust (jq)	2.640_±0.002	113.50_±00.04 + 778.25_±01.42	110.56_±01.02
Ruby/jruby	2.930_±0.031	461.99_±01.62 + 841.93_±75.18	148.36_±04.17
C++/clang++ (Boost.PropertyTree)	3.166_±0.007	194.88_±00.04 + 1232.84_±00.00	130.92_±01.68
C++/g++ (Boost.PropertyTree)	3.336_±0.004	113.22_±00.05 + 1440.12_±00.03	139.30_±00.49
Vala/gcc	3.613_±0.014	114.79_±00.03 + 940.47_±00.05	154.92_±00.77
Vala/clang	3.633_±0.006	114.81_±00.05 + 940.45_±00.03	156.14_±00.64
D/gdc	3.660_±0.004	116.67_±00.04 + 681.24_±00.12	156.47_±00.70
D/dmd	4.514_±0.003	113.37_±00.04 + 680.34_±00.05	177.98_±00.69
Perl (JSON::Tiny)	9.382_±0.067	125.86_±00.04 + 528.91_±00.03	404.50_±03.82
Ruby/truffleruby (--jvm)	12.311_±0.034	505.03_±05.57 + 2467.82_±54.26	747.19_±05.43
Ruby/truffleruby	22.970_±0.503	451.72_±03.63 + 2314.87_±07.88	1119.70_±23.10

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.043_±0.000	6.06_±00.02 + 57.94_±00.12	4.41_±00.02
V/gcc (VSL + CBLAS)	0.047_±0.000	6.96_±00.02 + 58.03_±00.00	4.55_±00.04
V/clang (VSL + CBLAS)	0.047_±0.000	7.03_±00.05 + 57.93_±00.05	4.57_±00.02
Nim/gcc (Arraymancer)	0.066_±0.004	4.91_±00.04 + 57.94_±00.17	5.47_±00.28
Python (NumPy)	0.067_±0.000	30.07_±00.04 + 58.18_±00.05	6.27_±00.06
Nim/clang (Arraymancer)	0.069_±0.002	6.24_±00.10 + 57.58_±00.12	6.02_±00.48
C++/g++ (Eigen)	0.072_±0.002	31.90_±06.49 + 58.05_±06.57	5.39_±00.10
C++/clang++ (Eigen)	0.072_±0.003	36.08_±05.02 + 54.22_±05.03	5.90_±00.13
Java (ND4J)	0.079_±0.002	106.62_±01.24 + 91.92_±00.12	6.18_±00.16
Rust (ndarray)	0.089_±0.001	2.39_±00.05 + 68.47_±00.00	6.05_±00.11
Julia (threads: 2)	0.137_±0.000	237.36_±00.09 + 52.34_±00.39	7.19_±00.05
Julia (threads: 1)	0.187_±0.000	237.13_±00.04 + 52.82_±00.46	8.46_±00.04
V/clang (VSL)	0.242_±0.005	7.04_±00.09 + 51.83_±00.00	17.31_±00.37
V/gcc (VSL)	0.477_±0.001	7.06_±00.04 + 51.82_±00.00	32.25_±00.08
Julia (no BLAS)	1.052_±0.021	224.75_±00.14 + 51.52_±00.02	46.61_±00.88
D/ldc2	1.718_±0.003	3.27_±00.02 + 70.46_±00.03	63.13_±00.15
D/gdc	1.870_±0.002	6.83_±00.03 + 70.88_±00.07	73.06_±00.14
D/dmd	1.874_±0.001	3.58_±00.03 + 70.48_±00.02	71.02_±00.09
C/gcc	3.029_±0.000	1.47_±00.03 + 68.70_±00.03	109.60_±00.60
V/gcc	3.038_±0.001	2.13_±00.03 + 69.01_±00.00	112.35_±00.44
V/clang	3.060_±0.000	2.43_±00.02 + 68.95_±00.03	104.62_±00.04
Vala/clang	3.061_±0.000	3.93_±00.08 + 69.80_±00.11	104.74_±00.17
C/clang	3.062_±0.000	1.46_±00.02 + 68.73_±00.05	104.49_±00.11
Rust	3.065_±0.000	2.11_±00.04 + 68.57_±00.00	105.21_±00.14
Zig	3.074_±0.001	1.50_±00.07 + 68.89_±00.03	108.77_±00.55
Nim/gcc	3.090_±0.000	2.51_±00.02 + 66.26_±00.00	111.06_±00.60
Java	3.093_±0.004	39.19_±00.08 + 68.43_±00.16	109.77_±01.88
Swift	3.095_±0.000	7.30_±00.04 + 68.92_±00.02	109.29_±00.43
Nim/clang	3.122_±0.001	2.82_±00.03 + 66.00_±00.00	106.98_±00.17
Vala/gcc	3.126_±0.001	3.89_±00.02 + 69.86_±00.02	114.22_±00.23
Go	3.153_±0.001	3.85_±00.04 + 73.12_±00.11	113.47_±00.19
Go/gccgo	3.160_±0.001	24.62_±00.01 + 73.50_±00.10	112.00_±00.14
Crystal	3.161_±0.000	3.44_±00.04 + 59.97_±00.04	115.46_±00.14
Node.js	3.238_±0.003	47.55_±00.19 + 76.79_±00.11	130.43_±00.45
Python/pypy	3.274_±0.001	61.70_±00.04 + 68.69_±00.06	136.27_±00.85
Scala	3.325_±0.008	65.60_±00.29 + 140.85_±00.30	121.60_±00.97
Kotlin/JVM	3.664_±0.003	40.44_±00.17 + 68.06_±00.29	152.07_±01.68
C#/.NET Core	4.384_±0.001	106.09_±00.16 + 69.06_±00.00	175.92_±00.95
C#/Mono	7.414_±0.001	25.41_±00.05 + 69.60_±00.04	299.70_±01.56
Ruby/truffleruby	17.828_±2.987	380.74_±06.95 + 518.10_±27.76	660.11_±94.70
Ruby/truffleruby (--jvm)	28.869_±0.417	552.53_±21.56 + 367.72_±55.70	994.44_±18.01
Ruby	198.042_±0.817	14.98_±00.04 + 69.05_±00.01	8578.84_±83.38
Ruby (--jit)	198.618_±1.593	271.64_±00.04 + 72.66_±00.01	8637.61_±80.58
Perl	225.366_±0.521	9.57_±00.03 + 599.64_±00.07	9143.92_±52.78
Python	278.785_±0.562	10.71_±00.27 + 68.87_±00.27	11177.48_±38.78
Tcl	335.868_±4.545	7.32_±00.05 + 400.18_±00.00	14014.34_±226.31
Ruby/jruby	399.505_±23.779	268.81_±01.64 + 756.38_±38.36	16344.90_±758.83

Primes

Testing:

generating primes using the optimized sieve of Atkin;
prefix search for their decimal numbers using Trie data structure.

Notes:

All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
The results are always sorted (could be unstable or stable though).

Primes

Language	Time, s	Memory, MiB	Energy, J
C++/g++	0.071_±0.000	3.68_±00.04 + 77.91_±00.26	2.71_±00.02
Zig	0.072_±0.000	0.91_±00.02 + 57.81_±00.03	2.97_±00.03
C++/clang++	0.072_±0.000	1.65_±00.05 + 61.74_±00.35	2.70_±00.02
V/clang	0.106_±0.000	1.91_±00.04 + 212.02_±00.30	4.25_±00.01
V/gcc	0.107_±0.000	1.87_±00.05 + 208.57_±00.33	4.35_±00.03
Rust	0.115_±0.000	0.94_±00.01 + 74.00_±00.01	4.47_±00.06
Crystal	0.151_±0.000	2.88_±00.06 + 89.06_±00.05	6.01_±00.03
Java	0.164_±0.002	38.52_±00.14 + 153.60_±05.25	9.25_±00.20
Node.js	0.225_±0.002	42.34_±00.13 + 150.08_±00.35	11.26_±00.08
Nim/clang	0.298_±0.000	1.99_±00.03 + 598.51_±01.16	11.77_±00.05
Scala	0.299_±0.006	67.35_±00.22 + 148.34_±04.80	15.31_±00.13
Nim/gcc	0.324_±0.031	1.69_±00.02 + 599.54_±15.34	12.39_±01.06
Lua/luajit	0.338_±0.002	1.21_±00.03 + 157.08_±00.75	13.26_±00.21
Julia	0.593_±0.000	226.44_±00.06 + 376.93_±00.44	23.15_±00.20
Python/pypy	0.830_±0.002	59.75_±00.10 + 251.87_±00.06	32.55_±00.20
Ruby/truffleruby (--jvm)	1.221_±0.033	409.10_±02.87 + 542.80_±45.51	79.93_±02.22
Ruby (--jit)	1.442_±0.004	270.68_±00.04 + 146.96_±00.01	58.58_±00.76
Lua	1.493_±0.004	2.28_±00.05 + 284.23_±00.90	58.17_±00.36
Ruby/truffleruby	1.657_±0.050	299.73_±00.64 + 496.36_±29.59	92.44_±02.52
Ruby	2.063_±0.011	13.96_±00.03 + 146.97_±00.02	83.40_±00.73
Ruby/jruby	2.143_±0.020	187.79_±00.61 + 545.57_±11.80	111.13_±02.43
Python	4.984_±0.041	10.42_±00.03 + 234.84_±00.78	194.67_±02.44

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language	Version
.NET Core	7.0.100
C#/.NET Core	4.4.0-4.22520.11 (9e075f03)
C#/Mono	6.8.0.105
Chez Scheme	9.5.8
Clojure	"1.11.1"
Crystal	1.6.2
D/dmd	v2.101.0
D/gdc	12.2.0
D/ldc2	1.30.0
Elixir	1.14.0
F#/.NET Core	12.4.0.0 for F# 7.0
Go	go1.19.4
Go/gccgo	12.2.0
Haskell	9.4.3
Idris 2	0.6.0
Java	19.0.1
Julia	v"1.8.3"
Kotlin	1.7.21
Lua	5.4.4
Lua/luajit	2.1.0-beta3
MLton	20210117
Nim	1.6.10
Node.js	v19.2.0
OCaml	4.14.0
PHP	8.1.12
Perl	v5.36.0
Python	3.10.8
Python/pypy	7.3.10-final0 for Python 3.9.15
Racket	"8.7"
Ruby	3.1.3p185
Ruby/jruby	9.4.0.0
Ruby/truffleruby	22.3.0
Rust	1.65.0
Scala	3.2.1
Swift	5.7.1
Tcl	8.6
V	0.3.2 d62fc77
Vala	0.56.3
Zig	0.10.0
clang/clang++	14.0.6
gcc/g++	12.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

The pinned packages could be missing, in that case please update them with:

$ ./run.sh update_apt

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

versions (print installed language versions);
shell (start the shell);
brainfuck bench (build and run Brainfuck bench.b benchmarks);
brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
base64 (build and run Base64 benchmarks);
json (build and run Json benchmarks);
matmul (build and run Matmul benchmarks);
primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

./run.sh build (build the image);
./run.sh make versions (run the image with the versions command);
sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

NumPy for matmul tests (Debian package python3-numpy).
UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

Boost for JSON tests (Debian package libboost-dev).
JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

Coursier for downloading Maven artifacts.

For Haskell:

network for TCP connectivity between the tests and the test runner.
raw-strings-qq for raw string literals used in tests.

For Perl:

cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

The test runner listens on localhost:9001;
All messages are sent using TCP sockets closed immediately after the message has been sent;
There are two messages sent from a test (it establishes the measurement boundary):
1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

append a new target (the final binary location) into executables variable;
append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

append a new target (the final artifact location) into artifacts variable;
append the proper target rule to compile the test;
append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

append run[<script_file>] into all_runners variable;
append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc

Docker image update

Debian packages are pinned and updated with the script (first, please ensure that the image is fine with the linter):

./run.sh lint
./run.sh update_apt

Name		Name	Last commit message	Last commit date
Latest commit History 821 Commits
base64		base64
brainfuck		brainfuck
common		common
docker		docker
json		json
matmul		matmul
primes		primes
.gitignore		.gitignore
.rubocop.yml		.rubocop.yml
Directory.Build.props		Directory.Build.props
LICENSE		LICENSE
README.md		README.md
analyze.rb		analyze.rb
repeat.rb		repeat.rb
run.sh		run.sh
xtime.rb		xtime.rb

License

dadhi/benchmarks

Folders and files

Latest commit

History

Repository files navigation

Table of Content

Overview

Measurements

Test Cases

Brainfuck

bench.b

mandel.b

Base64

Json

Matmul

Primes

Tests Execution

Environment

Using Docker

Manual Execution

Prerequisites

Contribution

Makefile guide

Binary executables

Compiled artifacts

Scripting language

README update

Docker image update

About

Resources

License

Stars

Watchers

Forks

Languages