Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
469 lines (425 sloc) 27.3 KB

Benchmarks Results

Test Methods

Running tests

lein with-profile bench,small-heap,parallel run

Benchmarks framework

Criterium is used to run quick benchmarks.

Profiles

Benchmarks are run on all combinations of the following:

  • heap size: big (9G), medium (5G), small (2G).
  • Garbage collection: parallel, G1

Tests details

assoc

Assoc and fast assoc performance are tested with maps and records.

assoc-in

Assoc-in is tested vs. an inlined implementation with vanilla maps, gets and assoc, all core functions.

get

get was tested on map, record and fast-map, fast-get was tested on fast-map.

Moreover, different get methods were tested:

  • map on keyword.
  • keyword on map.
  • keyword on record.
  • .get from record.
  • .field from record.

merge

Fast map merge

fast map merge was implemented by Metosin and uses kv-reduce to assoc one map into another. Was compared vs. regular merge.

Inline Merge

Two different implementations of inlining merge were tested, one based on the core implementation of merge, and one on Metosin's.

get-in

get-in was tested against an inlined implementation.

select-keys

select-keys was tested against an inlined implementation.

Results Summary

assoc

NOTE: the first result is surprising as it does not correlate with benchmarks run in the REPL. Requires further study.

  • assoc to record ~twice as fast as associng to map.
  • fast-assoc ~ 5.7% faster than assoc. (Metosin)

Assoc-in

  • the inlined implementation is always faster and exhibits compounding returns for deeper maps.

get

NOTE: Same case as the assoc experiment, with benchmarks results being the opposite of what's measured in a running REPL.

  • get from record ~50% slower than from map.
  • fast-get from fast-map ~ 8.4% faster than getting from regular map. (Metosin)
  • map on keyword > keyword on map > get from map (ordered by speed)
  • field from record > keyword on record > .get from record > get from record.

merge

Fast Map merge

Distinctly faster than regular merge, by about 30%, but differences might depend on map sizes, so more benchmarks are required.

Inline merge & fast merge

Sees diminishing returns on the benefit of merging more maps, but the speedup is measurable.

get-in

Inline implementation faster by a factor of 4-5.

select-keys

Inline implementation faster by a factor of 10 or more, depends on the number of selected keys.

heap gc test execution time mean
big g1 assoc to map 60.603761 ns
big g1 assoc to record 38.332317 ns
big g1 fast-assoc to map 58.735587 ns
big g1 fast-assoc to record 39.049585 ns
big g1 assoc-in 1 59.427094 ns
big g1 assoc-in 2 87.421121 ns
big g1 assoc-in 3 184.219410 ns
big g1 assoc-in 4 211.092112 ns
big g1 inline-assoc-in 1 39.376813 ns
big g1 inline-assoc-in 2 125.912361 ns
big g1 inline-assoc-in 3 139.318034 ns
big g1 inline-assoc-in 4 148.861730 ns
big g1 update-in 1 152.947998 ns
big g1 inline-update-in 1 147.596307 ns
big g1 update-in 2 212.675980 ns
big g1 inline-update-in 2 163.617075 ns
big g1 update-in 3 255.303195 ns
big g1 inline-update-in 3 191.001550 ns
big g1 update-in 4 294.511887 ns
big g1 inline-update-in 4 217.878713 ns
big g1 get from map 11.339306 ns
big g1 map on keyword 8.574417 ns
big g1 keyword on map 10.460608 ns
big g1 get from record 14.457372 ns
big g1 keyword on record 5.998421 ns
big g1 .get from record 12.510472 ns
big g1 get field from record 3.684584 ns
big g1 get from fast-map 38.197091 ns
big g1 fast-get from fast-map 7.799408 ns
big g1 merge maps 457.884247 ns
big g1 fast merge maps 341.036887 ns
big g1 merge 2 maps 506.583409 ns
big g1 inline merge 2 maps 420.007590 ns
big g1 inline fast merge 2 maps 344.121518 ns
big g1 merge 3 maps 1.664652 µs
big g1 inline merge 3 maps 1.490882 µs
big g1 inline fast merge 3 maps 1.443558 µs
big g1 merge 4 maps 1.954119 µs
big g1 inline merge 4 maps 1.797741 µs
big g1 inline fast merge 4 maps 1.774046 µs
big g1 get-in 1 39.786330 ns
big g1 fast get-in 1 10.344779 ns
big g1 get some in 1 5.947930 ns
big g1 get-in 2 57.110968 ns
big g1 fast get-in 2 18.057849 ns
big g1 get some in 2 10.442508 ns
big g1 get-in 3 73.667482 ns
big g1 fast get-in 3 25.383331 ns
big g1 get some in 3 13.199312 ns
big g1 get-in 4 86.140015 ns
big g1 fast get-in 4 33.072601 ns
big g1 get some in 4 17.824613 ns
big g1 select 1/4 keys 192.296473 ns
big g1 fast select 1/4 keys 24.456551 ns
big g1 select 2/4 keys 276.708812 ns
big g1 fast select 2/4 keys 32.690009 ns
big g1 select 3/4 keys 366.079642 ns
big g1 fast select 3/4 keys 44.340308 ns
big g1 select 4/4 keys 463.241990 ns
big g1 fast select 4/4 keys 56.175028 ns
big parallel assoc to map 32.371441 ns
big parallel assoc to record 19.345769 ns
big parallel fast-assoc to map 30.053994 ns
big parallel fast-assoc to record 16.126589 ns
big parallel assoc-in 1 49.492236 ns
big parallel assoc-in 2 69.415102 ns
big parallel assoc-in 3 167.128542 ns
big parallel assoc-in 4 189.282406 ns
big parallel inline-assoc-in 1 27.377661 ns
big parallel inline-assoc-in 2 106.761482 ns
big parallel inline-assoc-in 3 112.802275 ns
big parallel inline-assoc-in 4 121.005657 ns
big parallel update-in 1 134.903868 ns
big parallel inline-update-in 1 135.611914 ns
big parallel update-in 2 193.789099 ns
big parallel inline-update-in 2 154.667823 ns
big parallel update-in 3 235.722744 ns
big parallel inline-update-in 3 184.080886 ns
big parallel update-in 4 280.051438 ns
big parallel inline-update-in 4 214.153675 ns
big parallel get from map 10.338142 ns
big parallel map on keyword 8.876696 ns
big parallel keyword on map 9.502878 ns
big parallel get from record 14.311665 ns
big parallel keyword on record 6.190041 ns
big parallel .get from record 11.535257 ns
big parallel get field from record 3.636666 ns
big parallel get from fast-map 34.894720 ns
big parallel fast-get from fast-map 7.051884 ns
big parallel merge maps 333.429777 ns
big parallel fast merge maps 226.055813 ns
big parallel merge 2 maps 362.941200 ns
big parallel inline merge 2 maps 281.370760 ns
big parallel inline fast merge 2 maps 227.559517 ns
big parallel merge 3 maps 1.176878 µs
big parallel inline merge 3 maps 1.049544 µs
big parallel inline fast merge 3 maps 997.989050 ns
big parallel merge 4 maps 1.476614 µs
big parallel inline merge 4 maps 1.280925 µs
big parallel inline fast merge 4 maps 1.256592 µs
big parallel get-in 1 39.999427 ns
big parallel fast get-in 1 10.316292 ns
big parallel get some in 1 5.910026 ns
big parallel get-in 2 51.284549 ns
big parallel fast get-in 2 17.784598 ns
big parallel get some in 2 10.200495 ns
big parallel get-in 3 68.618918 ns
big parallel fast get-in 3 25.367662 ns
big parallel get some in 3 13.067672 ns
big parallel get-in 4 82.598429 ns
big parallel fast get-in 4 19.698458 ns
big parallel get some in 4 17.315701 ns
big parallel select 1/4 keys 172.930122 ns
big parallel fast select 1/4 keys 11.368870 ns
big parallel select 2/4 keys 213.377422 ns
big parallel fast select 2/4 keys 16.799935 ns
big parallel select 3/4 keys 274.118673 ns
big parallel fast select 3/4 keys 24.881389 ns
big parallel select 4/4 keys 312.094822 ns
big parallel fast select 4/4 keys 34.382155 ns
med g1 assoc to map 48.017200 ns
med g1 assoc to record 20.643270 ns
med g1 fast-assoc to map 46.291795 ns
med g1 fast-assoc to record 25.650662 ns
med g1 assoc-in 1 43.117304 ns
med g1 assoc-in 2 67.682634 ns
med g1 assoc-in 3 172.921140 ns
med g1 assoc-in 4 198.950834 ns
med g1 inline-assoc-in 1 22.610115 ns
med g1 inline-assoc-in 2 31.903842 ns
med g1 inline-assoc-in 3 42.472088 ns
med g1 inline-assoc-in 4 50.987722 ns
med g1 update-in 1 140.507125 ns
med g1 inline-update-in 1 139.058500 ns
med g1 update-in 2 203.525480 ns
med g1 inline-update-in 2 153.105318 ns
med g1 update-in 3 240.717953 ns
med g1 inline-update-in 3 174.664928 ns
med g1 update-in 4 276.727790 ns
med g1 inline-update-in 4 199.965659 ns
med g1 get from map 8.834362 ns
med g1 map on keyword 5.912481 ns
med g1 keyword on map 7.808933 ns
med g1 get from record 11.003575 ns
med g1 keyword on record 3.749699 ns
med g1 .get from record 8.644511 ns
med g1 get field from record 2.704952 ns
med g1 get from fast-map 35.390073 ns
med g1 fast-get from fast-map 6.566297 ns
med g1 merge maps 447.927732 ns
med g1 fast merge maps 324.039570 ns
med g1 merge 2 maps 487.202800 ns
med g1 inline merge 2 maps 408.914161 ns
med g1 inline fast merge 2 maps 322.729569 ns
med g1 merge 3 maps 1.667746 µs
med g1 inline merge 3 maps 1.512118 µs
med g1 inline fast merge 3 maps 1.436334 µs
med g1 merge 4 maps 1.993162 µs
med g1 inline merge 4 maps 1.815098 µs
med g1 inline fast merge 4 maps 1.804004 µs
med g1 get-in 1 34.169670 ns
med g1 fast get-in 1 5.289748 ns
med g1 get some in 1 4.243272 ns
med g1 get-in 2 48.283999 ns
med g1 fast get-in 2 8.627374 ns
med g1 get some in 2 7.429364 ns
med g1 get-in 3 66.763657 ns
med g1 fast get-in 3 11.984588 ns
med g1 get some in 3 9.202201 ns
med g1 get-in 4 79.162261 ns
med g1 fast get-in 4 14.872275 ns
med g1 get some in 4 11.299371 ns
med g1 select 1/4 keys 176.967376 ns
med g1 fast select 1/4 keys 11.581138 ns
med g1 select 2/4 keys 255.269173 ns
med g1 fast select 2/4 keys 17.749808 ns
med g1 select 3/4 keys 332.351710 ns
med g1 fast select 3/4 keys 27.370380 ns
med g1 select 4/4 keys 409.945355 ns
med g1 fast select 4/4 keys 35.948932 ns
med parallel assoc to map 32.652118 ns
med parallel assoc to record 20.071973 ns
med parallel fast-assoc to map 30.042532 ns
med parallel fast-assoc to record 15.483539 ns
med parallel assoc-in 1 49.995857 ns
med parallel assoc-in 2 71.163329 ns
med parallel assoc-in 3 163.669857 ns
med parallel assoc-in 4 182.472549 ns
med parallel inline-assoc-in 1 28.230009 ns
med parallel inline-assoc-in 2 105.898229 ns
med parallel inline-assoc-in 3 114.227061 ns
med parallel inline-assoc-in 4 121.748115 ns
med parallel update-in 1 146.082382 ns
med parallel inline-update-in 1 139.446354 ns
med parallel update-in 2 206.589125 ns
med parallel inline-update-in 2 155.998110 ns
med parallel update-in 3 246.369180 ns
med parallel inline-update-in 3 186.458825 ns
med parallel update-in 4 290.608633 ns
med parallel inline-update-in 4 221.029629 ns
med parallel get from map 10.214164 ns
med parallel map on keyword 7.584589 ns
med parallel keyword on map 10.197128 ns
med parallel get from record 13.653376 ns
med parallel keyword on record 5.386355 ns
med parallel .get from record 11.545223 ns
med parallel get field from record 2.887781 ns
med parallel get from fast-map 34.650105 ns
med parallel fast-get from fast-map 8.487544 ns
med parallel merge maps 347.013400 ns
med parallel fast merge maps 226.685897 ns
med parallel merge 2 maps 362.919701 ns
med parallel inline merge 2 maps 293.498301 ns
med parallel inline fast merge 2 maps 223.999555 ns
med parallel merge 3 maps 1.191610 µs
med parallel inline merge 3 maps 1.060675 µs
med parallel inline fast merge 3 maps 1.028475 µs
med parallel merge 4 maps 1.481805 µs
med parallel inline merge 4 maps 1.267463 µs
med parallel inline fast merge 4 maps 1.299272 µs
med parallel get-in 1 43.327491 ns
med parallel fast get-in 1 9.256784 ns
med parallel get some in 1 5.114690 ns
med parallel get-in 2 51.754609 ns
med parallel fast get-in 2 17.045797 ns
med parallel get some in 2 9.199258 ns
med parallel get-in 3 67.543169 ns
med parallel fast get-in 3 23.886607 ns
med parallel get some in 3 11.760107 ns
med parallel get-in 4 81.026015 ns
med parallel fast get-in 4 30.731871 ns
med parallel get some in 4 16.774926 ns
med parallel select 1/4 keys 165.262113 ns
med parallel fast select 1/4 keys 10.629346 ns
med parallel select 2/4 keys 215.158509 ns
med parallel fast select 2/4 keys 16.485982 ns
med parallel select 3/4 keys 261.335112 ns
med parallel fast select 3/4 keys 25.094703 ns
med parallel select 4/4 keys 319.062781 ns
med parallel fast select 4/4 keys 33.688716 ns
small g1 assoc to map 47.666145 ns
small g1 assoc to record 24.798200 ns
small g1 fast-assoc to map 45.433991 ns
small g1 fast-assoc to record 26.473440 ns
small g1 assoc-in 1 57.366398 ns
small g1 assoc-in 2 161.849693 ns
small g1 assoc-in 3 193.376447 ns
small g1 assoc-in 4 216.236068 ns
small g1 inline-assoc-in 1 34.896216 ns
small g1 inline-assoc-in 2 44.813208 ns
small g1 inline-assoc-in 3 53.825374 ns
small g1 inline-assoc-in 4 63.011021 ns
small g1 update-in 1 152.824818 ns
small g1 inline-update-in 1 146.139102 ns
small g1 update-in 2 220.670602 ns
small g1 inline-update-in 2 178.283010 ns
small g1 update-in 3 272.357914 ns
small g1 inline-update-in 3 210.222583 ns
small g1 update-in 4 323.837439 ns
small g1 inline-update-in 4 241.690539 ns
small g1 get from map 6.890287 ns
small g1 map on keyword 5.983850 ns
small g1 keyword on map 7.379764 ns
small g1 get from record 11.147459 ns
small g1 keyword on record 3.900881 ns
small g1 .get from record 8.578816 ns
small g1 get field from record 3.021558 ns
small g1 get from fast-map 34.433630 ns
small g1 fast-get from fast-map 6.437504 ns
small g1 merge maps 443.831947 ns
small g1 fast merge maps 322.338372 ns
small g1 merge 2 maps 480.732274 ns
small g1 inline merge 2 maps 397.315515 ns
small g1 inline fast merge 2 maps 331.951032 ns
small g1 merge 3 maps 1.625117 µs
small g1 inline merge 3 maps 1.475430 µs
small g1 inline fast merge 3 maps 1.434441 µs
small g1 merge 4 maps 1.944576 µs
small g1 inline merge 4 maps 1.729100 µs
small g1 inline fast merge 4 maps 1.758550 µs
small g1 get-in 1 34.282855 ns
small g1 fast get-in 1 5.547193 ns
small g1 get some in 1 4.647768 ns
small g1 get-in 2 42.763183 ns
small g1 fast get-in 2 8.872948 ns
small g1 get some in 2 6.904743 ns
small g1 get-in 3 61.994779 ns
small g1 fast get-in 3 11.837029 ns
small g1 get some in 3 8.724223 ns
small g1 get-in 4 73.568642 ns
small g1 fast get-in 4 15.524171 ns
small g1 get some in 4 10.565953 ns
small g1 select 1/4 keys 175.533348 ns
small g1 fast select 1/4 keys 11.462009 ns
small g1 select 2/4 keys 254.691261 ns
small g1 fast select 2/4 keys 17.650708 ns
small g1 select 3/4 keys 340.676613 ns
small g1 fast select 3/4 keys 27.556893 ns
small g1 select 4/4 keys 413.283392 ns
small g1 fast select 4/4 keys 37.457283 ns
small parallel assoc to map 35.119336 ns
small parallel assoc to record 16.812822 ns
small parallel fast-assoc to map 31.005039 ns
small parallel fast-assoc to record 16.537999 ns
small parallel assoc-in 1 48.362913 ns
small parallel assoc-in 2 67.516749 ns
small parallel assoc-in 3 164.549447 ns
small parallel assoc-in 4 182.822911 ns
small parallel inline-assoc-in 1 27.653108 ns
small parallel inline-assoc-in 2 34.917387 ns
small parallel inline-assoc-in 3 42.037604 ns
small parallel inline-assoc-in 4 50.553205 ns
small parallel update-in 1 135.826912 ns
small parallel inline-update-in 1 129.139195 ns
small parallel update-in 2 195.473361 ns
small parallel inline-update-in 2 156.309724 ns
small parallel update-in 3 234.387162 ns
small parallel inline-update-in 3 179.661130 ns
small parallel update-in 4 276.173607 ns
small parallel inline-update-in 4 206.638998 ns
small parallel get from map 8.142737 ns
small parallel map on keyword 6.804318 ns
small parallel keyword on map 8.481796 ns
small parallel get from record 14.686627 ns
small parallel keyword on record 4.603083 ns
small parallel .get from record 11.364444 ns
small parallel get field from record 3.116383 ns
small parallel get from fast-map 33.578521 ns
small parallel fast-get from fast-map 7.810883 ns
small parallel merge maps 332.567936 ns
small parallel fast merge maps 215.901561 ns
small parallel merge 2 maps 358.261884 ns
small parallel inline merge 2 maps 282.749681 ns
small parallel inline fast merge 2 maps 217.665055 ns
small parallel merge 3 maps 1.173280 µs
small parallel inline merge 3 maps 1.036088 µs
small parallel inline fast merge 3 maps 993.909129 ns
small parallel merge 4 maps 1.488247 µs
small parallel inline merge 4 maps 1.266443 µs
small parallel inline fast merge 4 maps 1.277996 µs
small parallel get-in 1 35.961353 ns
small parallel fast get-in 1 6.784345 ns
small parallel get some in 1 5.655861 ns
small parallel get-in 2 46.913183 ns
small parallel fast get-in 2 11.176587 ns
small parallel get some in 2 9.516190 ns
small parallel get-in 3 61.262515 ns
small parallel fast get-in 3 16.723093 ns
small parallel get some in 3 13.619676 ns
small parallel get-in 4 72.770708 ns
small parallel fast get-in 4 20.918450 ns
small parallel get some in 4 16.417517 ns
small parallel select 1/4 keys 168.064962 ns
small parallel fast select 1/4 keys 11.479721 ns
small parallel select 2/4 keys 215.831260 ns
small parallel fast select 2/4 keys 17.746165 ns
small parallel select 3/4 keys 265.468694 ns
small parallel fast select 3/4 keys 24.542730 ns
small parallel select 4/4 keys 312.513226 ns
small parallel fast select 4/4 keys 30.892997 ns
You can’t perform that action at this time.