Calibrate benchmark parameters and skips.

[vext01]

Calibrate benchmark parameters and skip CPython and the slow JRubyTruffle/Fasta.

I did a small run with 1 execution, 10 iterations, then chopped 5 iterations and took a mean. This gives us an idea of how long an execution sized 2000 could take.

Wrote a quick script to compute this for us:

$ python2.7 rough.py warmup_results.json.bz2 
binarytrees:C:default-c                 : 0.888
binarytrees:CPython:default-python      : skipped
binarytrees:Graal:default-java          : 0.176
binarytrees:HHVM:default-php            : 1.952
binarytrees:Hotspot:default-java        : 0.174
binarytrees:JRubyTruffle:default-ruby   : 2.815
binarytrees:LuaJIT:default-lua          : 1.526
binarytrees:PyPy:default-python         : 0.476
binarytrees:V8:default-javascript       : 0.457
fannkuch_redux:C:default-c              : 0.377
fannkuch_redux:CPython:default-python   : skipped
fannkuch_redux:Graal:default-java       : 0.351
fannkuch_redux:HHVM:default-php         : 1.402
fannkuch_redux:Hotspot:default-java     : 0.340
fannkuch_redux:JRubyTruffle:default-ruby: 2.322
fannkuch_redux:LuaJIT:default-lua       : 0.510
fannkuch_redux:PyPy:default-python      : 1.516
fannkuch_redux:V8:default-javascript    : 0.281
fasta:C:default-c                       : 0.067
fasta:CPython:default-python            : skipped
fasta:Graal:default-java                : 0.151
fasta:HHVM:default-php                  : 0.735
fasta:Hotspot:default-java              : 0.100
fasta:JRubyTruffle:default-ruby         : skipped
fasta:LuaJIT:default-lua                : 0.301
fasta:PyPy:default-python               : 3.311
fasta:V8:default-javascript             : 1.136
nbody:C:default-c                       : 0.368
nbody:CPython:default-python            : skipped
nbody:Graal:default-java                : 0.189
nbody:HHVM:default-php                  : 3.287
nbody:Hotspot:default-java              : 0.155
nbody:JRubyTruffle:default-ruby         : 0.675
nbody:LuaJIT:default-lua                : 0.266
nbody:PyPy:default-python               : 1.618
nbody:V8:default-javascript             : 0.251
richards:C:default-c                    : 0.681
richards:CPython:default-python         : skipped
richards:Graal:default-java             : 0.247
richards:HHVM:default-php               : 23.302
richards:Hotspot:default-java           : 0.229
richards:JRubyTruffle:default-ruby      : 1.887
richards:LuaJIT:default-lua             : 2.607
richards:PyPy:default-python            : 1.049
richards:V8:default-javascript          : 0.450
spectralnorm:C:default-c                : 0.475
spectralnorm:CPython:default-python     : skipped
spectralnorm:Graal:default-java         : 0.421
spectralnorm:HHVM:default-php           : 1.487
spectralnorm:Hotspot:default-java       : 0.554
spectralnorm:JRubyTruffle:default-ruby  : 1.058
spectralnorm:LuaJIT:default-lua         : 0.421
spectralnorm:PyPy:default-python        : 0.468
spectralnorm:V8:default-javascript      : 0.421
------------------------------------------------------------------------------
exec estimate:                          : 35.517 hours

Should we skip HHVM/richards? This accounts for about 13 or the 36 hours per execution.

Cheers

[snim2]

Really nice to have a script to compute the expected time! If we set this off now we would have an answer early next week, right? So ISTM that we could keep HHVM in.

[vext01]

OK?