ElixirBoardsBenchmark

A benchmark of different possible implementations of a 2D game board in elixir. The current benchmark is specifically for the board size 9x9.

What is a board implementation? See the Board behaviour - creating a new board, get(x, y) and set(x, y).

What is benchmarked?

• get of (0, 0), (4, 4) and (8, 8)
• set of (0, 0), (4, 4) and (8, 8)
• getting and setting of (0, 0), (4, 4) and (8, 8)
• getting and setting of all coordinates in one rune

The benchmarking code itself is in benchmark.exs.

Table of Contents

• ElixirBoardsBenchmark
  • But Why?
  • Adding your own implemenation?
  • Results
    • System Information
    • Getting and Setting Full Board
    • Mixed Bag (3 sets, 3 gets)
    • get(0,0)
    • get(4, 4)
    • get (8, 8)
    • set(0, 0)
    • set(4, 4)
    • set(8, 8)

But Why?

Fun. No concrete use case at the moment other than looking at performance of different data structures in these circumstances. Additionally, this happened to have been one of the first questions I asked the elixir community.

Adding your own implemenation?

Have another implementation? Great! Add it to lib/board/ with a name describing the data structure, adopt the Board behaviour then add the module name to the module list in test/board_test.exs and benchmark.exs. Done! (if the tests pass and the benchmark runs of course)

Results

System Information

Operating System	Linux
CPU Information	Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Number of Available Cores	8
Available Memory	15.61 GB
Elixir Version	1.8.2
Erlang Version	22.0

Getting and Setting Full Board

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple1D	133.95 K	7.47 μs	±23.29%	6.93 μs	10.80 μs
Tuple2D	132.16 K	7.57 μs	±29.17%	7.21 μs	14.02 μs
MapTuple	126.54 K	7.90 μs	±25.69%	7.59 μs	16.30 μs
ProcessDictionary	64.68 K	15.46 μs	±14.61%	15.12 μs	21.71 μs
ETSSet	60.35 K	16.57 μs	±9.17%	16.04 μs	21.09 μs
Array2D	56.76 K	17.62 μs	±17.45%	17.15 μs	21.45 μs
MapTupleFull	55.44 K	18.04 μs	±11.00%	16.92 μs	22.01 μs
MapTupleHalfFull	53.70 K	18.62 μs	±8.36%	17.96 μs	22.32 μs
Array1D	50.74 K	19.71 μs	±10.60%	19.29 μs	22.71 μs
ETSOrderedSet	39.53 K	25.30 μs	±10.51%	24.82 μs	28.42 μs
Map2D	36.24 K	27.59 μs	±8.32%	27.71 μs	32.27 μs
List2D	29.65 K	33.73 μs	±4.12%	33.31 μs	36.99 μs
MapTupleQuarterFull	28.23 K	35.42 μs	±3.86%	34.96 μs	38.34 μs
List1D	15.41 K	64.90 μs	±2.84%	64.91 μs	70.59 μs

Comparison

Name	IPS	Slower
Tuple1D	133.95 K
Tuple2D	132.16 K	1.01x
MapTuple	126.54 K	1.06x
ProcessDictionary	64.68 K	2.07x
ETSSet	60.35 K	2.22x
Array2D	56.76 K	2.36x
MapTupleFull	55.44 K	2.42x
MapTupleHalfFull	53.70 K	2.49x
Array1D	50.74 K	2.64x
ETSOrderedSet	39.53 K	3.39x
Map2D	36.24 K	3.7x
List2D	29.65 K	4.52x
MapTupleQuarterFull	28.23 K	4.75x
List1D	15.41 K	8.69x

Memory Usage

Name	Memory	Factor
Tuple1D	54.91 KB
Tuple2D	18.51 KB	0.34x
MapTuple	12.74 KB	0.23x
ProcessDictionary	9.52 KB	0.17x
ETSSet	11.55 KB	0.21x
Array2D	34.33 KB	0.63x
MapTupleFull	25.79 KB	0.47x
MapTupleHalfFull	24.06 KB	0.44x
Array1D	38.10 KB	0.69x
ETSOrderedSet	11.55 KB	0.21x
Map2D	49.55 KB	0.9x
List2D	22.60 KB	0.41x
MapTupleQuarterFull	47.56 KB	0.87x
List1D	59.38 KB	1.08x

---

Mixed Bag (3 sets, 3 gets)

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple1D	6.02 M	166.04 ns	±721.06%	127 ns	424 ns
Tuple2D	5.89 M	169.76 ns	±1323.53%	153 ns	350 ns
MapTuple	5.01 M	199.45 ns	±907.01%	178 ns	296 ns
ProcessDictionary	3.09 M	323.35 ns	±379.04%	307 ns	547 ns
Array1D	2.16 M	462.20 ns	±835.43%	421 ns	698 ns
MapTupleFull	1.84 M	542.80 ns	±29.17%	506 ns	730 ns
ETSSet	1.83 M	547.30 ns	±32.92%	515 ns	1474.88 ns
MapTupleHalfFull	1.79 M	560.05 ns	±41.55%	528 ns	953.76 ns
Array2D	1.76 M	569.76 ns	±405.50%	528 ns	873 ns
Map2D	1.29 M	777.13 ns	±19.67%	754 ns	2011.91 ns
ETSOrderedSet	1.10 M	907.35 ns	±39.78%	853 ns	1961.88 ns
List2D	0.91 M	1093.69 ns	±30.31%	1068 ns	1514.97 ns
MapTupleQuarterFull	0.82 M	1215.76 ns	±36.83%	1183 ns	2882 ns
List1D	0.44 M	2277.54 ns	±171.70%	2160 ns	3321.13 ns

Comparison

Name	IPS	Slower
Tuple1D	6.02 M
Tuple2D	5.89 M	1.02x
MapTuple	5.01 M	1.2x
ProcessDictionary	3.09 M	1.95x
Array1D	2.16 M	2.78x
MapTupleFull	1.84 M	3.27x
ETSSet	1.83 M	3.3x
MapTupleHalfFull	1.79 M	3.37x
Array2D	1.76 M	3.43x
Map2D	1.29 M	4.68x
ETSOrderedSet	1.10 M	5.46x
List2D	0.91 M	6.59x
MapTupleQuarterFull	0.82 M	7.32x
List1D	0.44 M	13.72x

Memory Usage

Name	Memory	Factor
Tuple1D	1344 B
Tuple2D	512 B	0.38x
MapTuple	368 B	0.27x
ProcessDictionary	104 B	0.08x
Array1D	1088 B	0.81x
MapTupleFull	400 B	0.3x
ETSSet	248 B	0.18x
MapTupleHalfFull	384 B	0.29x
Array2D	1112 B	0.83x
Map2D	1712 B	1.27x
ETSOrderedSet	248 B	0.18x
List2D	656 B	0.49x
MapTupleQuarterFull	1624 B	1.21x
List1D	2048 B	1.52x

---

get(0,0)

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple1D	44.12 M	22.66 ns	±842.77%	20 ns	33 ns
Tuple2D	42.46 M	23.55 ns	±846.67%	20 ns	64 ns
Array1D	30.38 M	32.92 ns	±84.61%	32 ns	49 ns
MapTuple	29.09 M	34.38 ns	±111.15%	32 ns	61 ns
MapTupleQuarterFull	18.86 M	53.03 ns	±37.27%	50 ns	95 ns
Array2D	18.62 M	53.70 ns	±67.02%	50 ns	80 ns
List1D	18.26 M	54.75 ns	±56.06%	53 ns	117 ns
ProcessDictionary	17.19 M	58.18 ns	±1393.09%	52 ns	71 ns
Map2D	15.79 M	63.34 ns	±25.86%	60 ns	108 ns
MapTupleHalfFull	10.54 M	94.87 ns	±27.72%	91 ns	142 ns
MapTupleFull	10.29 M	97.16 ns	±18.01%	93 ns	132 ns
ETSSet	9.74 M	102.63 ns	±26.57%	100 ns	144 ns
List2D	9.04 M	110.57 ns	±69.64%	105 ns	200 ns
ETSOrderedSet	6.47 M	154.65 ns	±19.27%	152 ns	190 ns

Comparison

Name	IPS	Slower
Tuple1D	44.12 M
Tuple2D	42.46 M	1.04x
Array1D	30.38 M	1.45x
MapTuple	29.09 M	1.52x
MapTupleQuarterFull	18.86 M	2.34x
Array2D	18.62 M	2.37x
List1D	18.26 M	2.42x
ProcessDictionary	17.19 M	2.57x
Map2D	15.79 M	2.79x
MapTupleHalfFull	10.54 M	4.19x
MapTupleFull	10.29 M	4.29x
ETSSet	9.74 M	4.53x
List2D	9.04 M	4.88x
ETSOrderedSet	6.47 M	6.82x

---

get(4, 4)

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple2D	37.76 M	26.48 ns	±722.24%	24 ns	40 ns
Tuple1D	36.87 M	27.12 ns	±618.93%	24 ns	57 ns
Array1D	28.57 M	35.00 ns	±78.21%	34 ns	45 ns
MapTuple	27.00 M	37.04 ns	±131.99%	34 ns	73 ns
ProcessDictionary	18.89 M	52.94 ns	±843.86%	49 ns	67 ns
Array2D	16.70 M	59.88 ns	±34.66%	56 ns	85 ns
Map2D	14.28 M	70.03 ns	±49.51%	64 ns	115 ns
MapTupleHalfFull	10.38 M	96.36 ns	±30.87%	93 ns	146 ns
MapTupleFull	9.97 M	100.30 ns	±20.77%	97 ns	136 ns
ETSSet	9.17 M	109.02 ns	±14.73%	107 ns	145 ns
List2D	5.77 M	173.18 ns	±47.97%	164 ns	293 ns
ETSOrderedSet	5.64 M	177.34 ns	±34.69%	173 ns	211 ns
MapTupleQuarterFull	3.99 M	250.61 ns	±25.89%	247 ns	308 ns
List1D	2.46 M	407.10 ns	±14.32%	400 ns	632 ns

Comparison

Name	IPS	Slower
Tuple2D	37.76 M
Tuple1D	36.87 M	1.02x
Array1D	28.57 M	1.32x
MapTuple	27.00 M	1.4x
ProcessDictionary	18.89 M	2.0x
Array2D	16.70 M	2.26x
Map2D	14.28 M	2.64x
MapTupleHalfFull	10.38 M	3.64x
MapTupleFull	9.97 M	3.79x
ETSSet	9.17 M	4.12x
List2D	5.77 M	6.54x
ETSOrderedSet	5.64 M	6.7x
MapTupleQuarterFull	3.99 M	9.46x
List1D	2.46 M	15.37x

---

get (8, 8)

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple2D	42.47 M	23.55 ns	±788.60%	21 ns	40 ns
Tuple1D	40.98 M	24.40 ns	±725.07%	22 ns	41 ns
Array1D	29.67 M	33.70 ns	±161.51%	33 ns	45 ns
MapTuple	28.54 M	35.03 ns	±986.95%	32 ns	50 ns
ProcessDictionary	19.71 M	50.73 ns	±1279.45%	47 ns	65 ns
Array2D	17.88 M	55.92 ns	±85.10%	52 ns	80 ns
Map2D	13.28 M	75.31 ns	±32.34%	73 ns	126 ns
MapTupleHalfFull	12.12 M	82.53 ns	±31.49%	80 ns	126 ns
ETSSet	9.90 M	101.05 ns	±16.04%	99 ns	135 ns
MapTupleFull	9.85 M	101.53 ns	±19.29%	99 ns	136 ns
ETSOrderedSet	5.59 M	178.80 ns	±41.70%	169 ns	394.49 ns
MapTupleQuarterFull	4.09 M	244.65 ns	±16.85%	242 ns	299 ns
List2D	3.76 M	265.82 ns	±35.71%	251 ns	486 ns
List1D	1.38 M	724.35 ns	±10.88%	715 ns	951 ns

Comparison

Name	IPS	Slower
Tuple2D	42.47 M
Tuple1D	40.98 M	1.04x
Array1D	29.67 M	1.43x
MapTuple	28.54 M	1.49x
ProcessDictionary	19.71 M	2.15x
Array2D	17.88 M	2.37x
Map2D	13.28 M	3.2x
MapTupleHalfFull	12.12 M	3.5x
ETSSet	9.90 M	4.29x
MapTupleFull	9.85 M	4.31x
ETSOrderedSet	5.59 M	7.59x
MapTupleQuarterFull	4.09 M	10.39x
List2D	3.76 M	11.29x
List1D	1.38 M	30.76x

---

set(0, 0)

Run Time

Name	IPS	Average	Devitation	Median	99th %
List1D	31.04 M	32.21 ns	±59.43%	30 ns	91 ns
Tuple2D	16.91 M	59.15 ns	±2301.62%	48 ns	87 ns
ProcessDictionary	14.86 M	67.31 ns	±666.24%	62 ns	80 ns
MapTuple	14.65 M	68.24 ns	±24850.94%	33 ns	88 ns
Tuple1D	12.04 M	83.05 ns	±5738.15%	43 ns	116 ns
List2D	10.49 M	95.30 ns	±484.89%	81 ns	204 ns
MapTupleQuarterFull	9.01 M	111.00 ns	±247.51%	81 ns	1388 ns
ETSSet	8.75 M	114.30 ns	±19.08%	112 ns	142 ns
MapTupleFull	8.65 M	115.64 ns	±16.84%	111 ns	151 ns
MapTupleHalfFull	6.87 M	145.65 ns	±195.37%	121 ns	174.11 ns
Array2D	6.73 M	148.59 ns	±31.02%	145 ns	237 ns
Array1D	5.77 M	173.25 ns	±1013.39%	149 ns	189 ns
ETSOrderedSet	5.49 M	182.25 ns	±22.92%	180 ns	227.75 ns
Map2D	4.09 M	244.47 ns	±115.24%	219 ns	1051 ns

Comparison

Name	IPS	Slower
List1D	31.04 M
Tuple2D	16.91 M	1.84x
ProcessDictionary	14.86 M	2.09x
MapTuple	14.65 M	2.12x
Tuple1D	12.04 M	2.58x
List2D	10.49 M	2.96x
MapTupleQuarterFull	9.01 M	3.45x
ETSSet	8.75 M	3.55x
MapTupleFull	8.65 M	3.59x
MapTupleHalfFull	6.87 M	4.52x
Array2D	6.73 M	4.61x
Array1D	5.77 M	5.38x
ETSOrderedSet	5.49 M	5.66x
Map2D	4.09 M	7.59x

---

set(4, 4)

Run Time

Name	IPS	Average	Devitation	Median	99th %
MapTuple	21.17 M	47.25 ns	±18021.59%	24 ns	55 ns
Tuple2D	20.96 M	47.70 ns	±4316.38%	38 ns	57 ns
ProcessDictionary	17.43 M	57.36 ns	±6745.68%	46 ns	62 ns
Tuple1D	12.43 M	80.46 ns	±8459.34%	33 ns	115 ns
MapTupleFull	9.54 M	104.85 ns	±25.06%	100 ns	242.92 ns
ETSSet	9.34 M	107.08 ns	±20.13%	105 ns	137 ns
MapTupleHalfFull	8.92 M	112.14 ns	±51.21%	108 ns	157 ns
Array2D	7.06 M	141.62 ns	±45.86%	138 ns	211 ns
Array1D	6.01 M	166.46 ns	±949.89%	142 ns	186 ns
ETSOrderedSet	5.55 M	180.21 ns	±30.79%	174 ns	271.61 ns
List2D	4.81 M	207.77 ns	±93.27%	201 ns	431 ns
MapTupleQuarterFull	4.03 M	247.92 ns	±28.34%	241 ns	307 ns
Map2D	3.95 M	253.35 ns	±117.41%	225 ns	1053.34 ns
List1D	2.80 M	357.35 ns	±94.13%	342 ns	611 ns

Comparison

Name	IPS	Slower
MapTuple	21.17 M
Tuple2D	20.96 M	1.01x
ProcessDictionary	17.43 M	1.21x
Tuple1D	12.43 M	1.7x
MapTupleFull	9.54 M	2.22x
ETSSet	9.34 M	2.27x
MapTupleHalfFull	8.92 M	2.37x
Array2D	7.06 M	3.0x
Array1D	6.01 M	3.52x
ETSOrderedSet	5.55 M	3.81x
List2D	4.81 M	4.4x
MapTupleQuarterFull	4.03 M	5.25x
Map2D	3.95 M	5.36x
List1D	2.80 M	7.56x

---

set(8, 8)

Run Time

Name	IPS	Average	Devitation	Median	99th %
Tuple2D	16.87 M	59.28 ns	±2026.15%	51 ns	70 ns
ProcessDictionary	14.36 M	69.65 ns	±4696.55%	59 ns	75 ns
MapTuple	12.16 M	82.24 ns	±23284.12%	36 ns	67 ns
Tuple1D	10.67 M	93.74 ns	±7351.11%	46 ns	111 ns
MapTupleHalfFull	8.92 M	112.16 ns	±42.16%	110 ns	158 ns
MapTupleFull	8.77 M	114.03 ns	±22.42%	111 ns	148 ns
ETSSet	8.58 M	116.53 ns	±17.78%	114 ns	145 ns
Array2D	6.58 M	152.06 ns	±35.69%	148 ns	245 ns
Array1D	5.63 M	177.74 ns	±978.38%	154 ns	202 ns
ETSOrderedSet	5.34 M	187.42 ns	±38.95%	183 ns	228 ns
Map2D	3.50 M	285.55 ns	±100.73%	260 ns	1090 ns
List2D	2.89 M	345.55 ns	±24.43%	328 ns	587.01 ns
MapTupleQuarterFull	2.62 M	381.31 ns	±16.51%	375 ns	436 ns
List1D	1.35 M	739.31 ns	±147.87%	705 ns	1340.80 ns

Comparison

Name	IPS	Slower
Tuple2D	16.87 M
ProcessDictionary	14.36 M	1.17x
MapTuple	12.16 M	1.39x
Tuple1D	10.67 M	1.58x
MapTupleHalfFull	8.92 M	1.89x
MapTupleFull	8.77 M	1.92x
ETSSet	8.58 M	1.97x
Array2D	6.58 M	2.57x
Array1D	5.63 M	3.0x
ETSOrderedSet	5.34 M	3.16x
Map2D	3.50 M	4.82x
List2D	2.89 M	5.83x
MapTupleQuarterFull	2.62 M	6.43x
List1D	1.35 M	12.47x

---