Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tooling] 'display aggregates only' support. #674

Merged
merged 1 commit into from Sep 17, 2018
Merged

[Tooling] 'display aggregates only' support. #674

merged 1 commit into from Sep 17, 2018

Conversation

LebedevRI
Copy link
Collaborator

This builds upon #658 and #665.
Works as you'd expect, both for the benchmark binaries, and
the jsons (naturally, since the tests need to work):

$ ~/src/googlebenchmark/tools/compare.py benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpYoui5H
2018-09-12 20:23:47
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.447302 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.738M Pixels/WallTime=117.738M Raws/CPUTime=2.23562 Raws/WallTime=2.23564 WallTime,s=0.4473
2K4A9927.CR2/threads:8/real_time               444 ms        444 ms          2 CPUTime,s=0.444228 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.552M Pixels/WallTime=118.553M Raws/CPUTime=2.2511 Raws/WallTime=2.25111 WallTime,s=0.444226
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=0.999999 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446171 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.037M Pixels/WallTime=118.038M Raws/CPUTime=2.24131 Raws/WallTime=2.24132 WallTime,s=0.44617
2K4A9927.CR2/threads:8/real_time_median        447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.69052m CPUTime/WallTime=3.53737u Pixels=0 Pixels/CPUTime=448.178k Pixels/WallTime=448.336k Raws/CPUTime=8.51008m Raws/WallTime=8.51308m WallTime,s=1.6911m
2K4A9928.CR2/threads:8/real_time               563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999824 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.561328 CPUTime/WallTime=0.999917 Pixels=27.9936M Pixels/CPUTime=49.8703M Pixels/WallTime=49.8662M Raws/CPUTime=1.78149 Raws/WallTime=1.78134 WallTime,s=0.561375
2K4A9928.CR2/threads:8/real_time               570 ms        570 ms          1 CPUTime,s=0.570423 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.0752M Pixels/WallTime=49.0691M Raws/CPUTime=1.75308 Raws/WallTime=1.75287 WallTime,s=0.570493
2K4A9928.CR2/threads:8/real_time_mean          565 ms        565 ms          1 CPUTime,s=0.564754 CPUTime/WallTime=0.999872 Pixels=27.9936M Pixels/CPUTime=49.5703M Pixels/WallTime=49.564M Raws/CPUTime=1.77077 Raws/WallTime=1.77055 WallTime,s=0.564826
2K4A9928.CR2/threads:8/real_time_median        563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time_stddev          5 ms          5 ms          1 CPUTime,s=4.945m CPUTime/WallTime=46.3459u Pixels=0 Pixels/CPUTime=431.997k Pixels/WallTime=432.061k Raws/CPUTime=0.015432 Raws/WallTime=0.0154343 WallTime,s=4.94686m
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.306476 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.5957M Pixels/WallTime=40.5941M Raws/CPUTime=3.2629 Raws/WallTime=3.26277 WallTime,s=0.306488
2K4A9929.CR2/threads:8/real_time               310 ms        310 ms          2 CPUTime,s=0.309978 CPUTime/WallTime=0.999939 Pixels=12.4416M Pixels/CPUTime=40.1371M Pixels/WallTime=40.1347M Raws/CPUTime=3.22604 Raws/WallTime=3.22584 WallTime,s=0.309996
2K4A9929.CR2/threads:8/real_time               309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999987 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_mean          309 ms        309 ms          2 CPUTime,s=0.308628 CPUTime/WallTime=0.999962 Pixels=12.4416M Pixels/CPUTime=40.3136M Pixels/WallTime=40.3121M Raws/CPUTime=3.24023 Raws/WallTime=3.24011 WallTime,s=0.308639
2K4A9929.CR2/threads:8/real_time_median        309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.88354m CPUTime/WallTime=23.7788u Pixels=0 Pixels/CPUTime=246.821k Pixels/WallTime=246.914k Raws/CPUTime=0.0198384 Raws/WallTime=0.0198458 WallTime,s=1.88442m
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpRShmmf
2018-09-12 20:23:55
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.445589 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.19M Pixels/WallTime=118.19M Raws/CPUTime=2.24422 Raws/WallTime=2.24422 WallTime,s=0.445589
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time               448 ms        448 ms          2 CPUTime,s=0.447763 CPUTime/WallTime=0.999994 Pixels=52.6643M Pixels/CPUTime=117.616M Pixels/WallTime=117.616M Raws/CPUTime=2.23332 Raws/WallTime=2.23331 WallTime,s=0.447766
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446453 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=117.962M Pixels/WallTime=117.962M Raws/CPUTime=2.23988 Raws/WallTime=2.23989 WallTime,s=0.446453
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=1.15367m CPUTime/WallTime=6.48501u Pixels=0 Pixels/CPUTime=304.437k Pixels/WallTime=305.025k Raws/CPUTime=5.7807m Raws/WallTime=5.79188m WallTime,s=1.15591m
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time               560 ms        560 ms          1 CPUTime,s=0.560023 CPUTime/WallTime=1.00001 Pixels=27.9936M Pixels/CPUTime=49.9865M Pixels/WallTime=49.9872M Raws/CPUTime=1.78564 Raws/WallTime=1.78567 WallTime,s=0.560015
2K4A9928.CR2/threads:8/real_time               562 ms        562 ms          1 CPUTime,s=0.562337 CPUTime/WallTime=0.999994 Pixels=27.9936M Pixels/CPUTime=49.7808M Pixels/WallTime=49.7805M Raws/CPUTime=1.77829 Raws/WallTime=1.77828 WallTime,s=0.56234
2K4A9928.CR2/threads:8/real_time_mean          561 ms        561 ms          1 CPUTime,s=0.561072 CPUTime/WallTime=1 Pixels=27.9936M Pixels/CPUTime=49.8932M Pixels/WallTime=49.8933M Raws/CPUTime=1.78231 Raws/WallTime=1.78231 WallTime,s=0.561071
2K4A9928.CR2/threads:8/real_time_median        561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=1.17202m CPUTime/WallTime=10.7929u Pixels=0 Pixels/CPUTime=104.164k Pixels/WallTime=104.612k Raws/CPUTime=3.721m Raws/WallTime=3.73701m WallTime,s=1.17706m
2K4A9929.CR2/threads:8/real_time               305 ms        305 ms          2 CPUTime,s=0.305436 CPUTime/WallTime=0.999926 Pixels=12.4416M Pixels/CPUTime=40.7339M Pixels/WallTime=40.7309M Raws/CPUTime=3.27401 Raws/WallTime=3.27376 WallTime,s=0.305459
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time               307 ms        307 ms          2 CPUTime,s=0.30724 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.4947M Pixels/WallTime=40.4944M Raws/CPUTime=3.25478 Raws/WallTime=3.25475 WallTime,s=0.307243
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.306145 CPUTime/WallTime=0.999972 Pixels=12.4416M Pixels/CPUTime=40.6398M Pixels/WallTime=40.6387M Raws/CPUTime=3.26645 Raws/WallTime=3.26635 WallTime,s=0.306154
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=961.851u CPUTime/WallTime=40.2708u Pixels=0 Pixels/CPUTime=127.481k Pixels/WallTime=126.577k Raws/CPUTime=0.0102463 Raws/WallTime=0.0101737 WallTime,s=955.105u
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time                       -0.0038         -0.0038           447           446           447           446
2K4A9927.CR2/threads:8/real_time                       +0.0031         +0.0031           444           446           444           446
2K4A9927.CR2/threads:8/real_time                       -0.0031         -0.0031           447           446           447           446
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  +0.0006         +0.0006           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                -0.0022         -0.0022           447           446           447           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.3161         -0.3175             2             1             2             1
2K4A9928.CR2/threads:8/real_time                       -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time                       -0.0009         -0.0008           561           561           561           561
2K4A9928.CR2/threads:8/real_time                       -0.0169         -0.0168           570           561           570           561
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  -0.0066         -0.0065           565           561           565           561
2K4A9928.CR2/threads:8/real_time_median                -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time_stddev                -0.7620         -0.7630             5             1             5             1
2K4A9929.CR2/threads:8/real_time                       -0.0034         -0.0034           306           305           306           305
2K4A9929.CR2/threads:8/real_time                       -0.0146         -0.0146           310           305           310           305
2K4A9929.CR2/threads:8/real_time                       -0.0128         -0.0129           309           305           309           305
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  -0.0081         -0.0080           309           306           309           306
2K4A9929.CR2/threads:8/real_time_median                -0.0119         -0.0119           309           306           309           306
2K4A9929.CR2/threads:8/real_time_stddev                -0.4931         -0.4894             2             1             2             1
$ ~/src/googlebenchmark/tools/compare.py -a benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpjrD2I0
2018-09-12 20:24:11
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446077 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.061M Pixels/WallTime=118.062M Raws/CPUTime=2.24177 Raws/WallTime=2.24178 WallTime,s=0.446076
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445676 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.167M Pixels/WallTime=118.168M Raws/CPUTime=2.24378 Raws/WallTime=2.2438 WallTime,s=0.445673
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=820.275u CPUTime/WallTime=3.51513u Pixels=0 Pixels/CPUTime=216.876k Pixels/WallTime=217.229k Raws/CPUTime=4.11809m Raws/WallTime=4.12478m WallTime,s=821.607u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.561843 CPUTime/WallTime=0.999983 Pixels=27.9936M Pixels/CPUTime=49.8247M Pixels/WallTime=49.8238M Raws/CPUTime=1.77986 Raws/WallTime=1.77983 WallTime,s=0.561852
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.561745 CPUTime/WallTime=0.999995 Pixels=27.9936M Pixels/CPUTime=49.8333M Pixels/WallTime=49.8307M Raws/CPUTime=1.78017 Raws/WallTime=1.78008 WallTime,s=0.561774
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=788.58u CPUTime/WallTime=30.2578u Pixels=0 Pixels/CPUTime=69.914k Pixels/WallTime=69.4978k Raws/CPUTime=2.4975m Raws/WallTime=2.48263m WallTime,s=783.873u
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.305718 CPUTime/WallTime=1.00001 Pixels=12.4416M Pixels/CPUTime=40.6964M Pixels/WallTime=40.6967M Raws/CPUTime=3.271 Raws/WallTime=3.27102 WallTime,s=0.305716
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.305737 CPUTime/WallTime=1 Pixels=12.4416M Pixels/CPUTime=40.6939M Pixels/WallTime=40.6945M Raws/CPUTime=3.27079 Raws/WallTime=3.27084 WallTime,s=0.305732
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=584.969u CPUTime/WallTime=7.87539u Pixels=0 Pixels/CPUTime=77.8769k Pixels/WallTime=77.9118k Raws/CPUTime=6.2594m Raws/WallTime=6.2622m WallTime,s=585.232u
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpN9Xhk2
2018-09-12 20:24:18
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.445771 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.142M Pixels/WallTime=118.143M Raws/CPUTime=2.2433 Raws/WallTime=2.24332 WallTime,s=0.445769
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445719 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.156M Pixels/WallTime=118.156M Raws/CPUTime=2.24356 Raws/WallTime=2.24358 WallTime,s=0.445717
2K4A9927.CR2/threads:8/real_time_stddev          0 ms          0 ms          2 CPUTime,s=184.23u CPUTime/WallTime=1.69441u Pixels=0 Pixels/CPUTime=48.8185k Pixels/WallTime=49.0086k Raws/CPUTime=926.975u Raws/WallTime=930.585u WallTime,s=184.946u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.562071 CPUTime/WallTime=0.999969 Pixels=27.9936M Pixels/CPUTime=49.8045M Pixels/WallTime=49.803M Raws/CPUTime=1.77914 Raws/WallTime=1.77908 WallTime,s=0.562088
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.56237 CPUTime/WallTime=0.999986 Pixels=27.9936M Pixels/CPUTime=49.7779M Pixels/WallTime=49.7772M Raws/CPUTime=1.77819 Raws/WallTime=1.77816 WallTime,s=0.562378
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=967.38u CPUTime/WallTime=35.0024u Pixels=0 Pixels/CPUTime=85.7806k Pixels/WallTime=84.0545k Raws/CPUTime=3.06429m Raws/WallTime=3.00263m WallTime,s=947.993u
2K4A9929.CR2/threads:8/real_time_mean          307 ms        307 ms          2 CPUTime,s=0.306511 CPUTime/WallTime=0.999995 Pixels=12.4416M Pixels/CPUTime=40.5924M Pixels/WallTime=40.5922M Raws/CPUTime=3.26264 Raws/WallTime=3.26262 WallTime,s=0.306513
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.306169 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6364M Pixels/WallTime=40.637M Raws/CPUTime=3.26618 Raws/WallTime=3.26622 WallTime,s=0.306164
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=2.25763m CPUTime/WallTime=21.4306u Pixels=0 Pixels/CPUTime=298.503k Pixels/WallTime=299.126k Raws/CPUTime=0.0239924 Raws/WallTime=0.0240424 WallTime,s=2.26242m
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  -0.0007         -0.0007           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                +0.0001         +0.0001           446           446           446           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.7746         -0.7749             1             0             1             0
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  +0.0004         +0.0004           562           562           562           562
2K4A9928.CR2/threads:8/real_time_median                +0.0011         +0.0011           562           562           562           562
2K4A9928.CR2/threads:8/real_time_stddev                +0.2091         +0.2270             1             1             1             1
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  +0.0026         +0.0026           306           307           306           307
2K4A9929.CR2/threads:8/real_time_median                +0.0014         +0.0014           306           306           306           306
2K4A9929.CR2/threads:8/real_time_stddev                +2.8652         +2.8585             1             2             1             2

@LebedevRI
[Tooling] 'display aggregates only' support.
ded5ecb 
Works as you'd expect, both for the benchmark binaries, and
the jsons (naturally, since the tests need to work):
```
$ ~/src/googlebenchmark/tools/compare.py benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpYoui5H
2018-09-12 20:23:47
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.447302 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.738M Pixels/WallTime=117.738M Raws/CPUTime=2.23562 Raws/WallTime=2.23564 WallTime,s=0.4473
2K4A9927.CR2/threads:8/real_time               444 ms        444 ms          2 CPUTime,s=0.444228 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.552M Pixels/WallTime=118.553M Raws/CPUTime=2.2511 Raws/WallTime=2.25111 WallTime,s=0.444226
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=0.999999 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446171 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.037M Pixels/WallTime=118.038M Raws/CPUTime=2.24131 Raws/WallTime=2.24132 WallTime,s=0.44617
2K4A9927.CR2/threads:8/real_time_median        447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.69052m CPUTime/WallTime=3.53737u Pixels=0 Pixels/CPUTime=448.178k Pixels/WallTime=448.336k Raws/CPUTime=8.51008m Raws/WallTime=8.51308m WallTime,s=1.6911m
2K4A9928.CR2/threads:8/real_time               563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999824 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.561328 CPUTime/WallTime=0.999917 Pixels=27.9936M Pixels/CPUTime=49.8703M Pixels/WallTime=49.8662M Raws/CPUTime=1.78149 Raws/WallTime=1.78134 WallTime,s=0.561375
2K4A9928.CR2/threads:8/real_time               570 ms        570 ms          1 CPUTime,s=0.570423 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.0752M Pixels/WallTime=49.0691M Raws/CPUTime=1.75308 Raws/WallTime=1.75287 WallTime,s=0.570493
2K4A9928.CR2/threads:8/real_time_mean          565 ms        565 ms          1 CPUTime,s=0.564754 CPUTime/WallTime=0.999872 Pixels=27.9936M Pixels/CPUTime=49.5703M Pixels/WallTime=49.564M Raws/CPUTime=1.77077 Raws/WallTime=1.77055 WallTime,s=0.564826
2K4A9928.CR2/threads:8/real_time_median        563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time_stddev          5 ms          5 ms          1 CPUTime,s=4.945m CPUTime/WallTime=46.3459u Pixels=0 Pixels/CPUTime=431.997k Pixels/WallTime=432.061k Raws/CPUTime=0.015432 Raws/WallTime=0.0154343 WallTime,s=4.94686m
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.306476 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.5957M Pixels/WallTime=40.5941M Raws/CPUTime=3.2629 Raws/WallTime=3.26277 WallTime,s=0.306488
2K4A9929.CR2/threads:8/real_time               310 ms        310 ms          2 CPUTime,s=0.309978 CPUTime/WallTime=0.999939 Pixels=12.4416M Pixels/CPUTime=40.1371M Pixels/WallTime=40.1347M Raws/CPUTime=3.22604 Raws/WallTime=3.22584 WallTime,s=0.309996
2K4A9929.CR2/threads:8/real_time               309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999987 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_mean          309 ms        309 ms          2 CPUTime,s=0.308628 CPUTime/WallTime=0.999962 Pixels=12.4416M Pixels/CPUTime=40.3136M Pixels/WallTime=40.3121M Raws/CPUTime=3.24023 Raws/WallTime=3.24011 WallTime,s=0.308639
2K4A9929.CR2/threads:8/real_time_median        309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.88354m CPUTime/WallTime=23.7788u Pixels=0 Pixels/CPUTime=246.821k Pixels/WallTime=246.914k Raws/CPUTime=0.0198384 Raws/WallTime=0.0198458 WallTime,s=1.88442m
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpRShmmf
2018-09-12 20:23:55
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.445589 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.19M Pixels/WallTime=118.19M Raws/CPUTime=2.24422 Raws/WallTime=2.24422 WallTime,s=0.445589
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time               448 ms        448 ms          2 CPUTime,s=0.447763 CPUTime/WallTime=0.999994 Pixels=52.6643M Pixels/CPUTime=117.616M Pixels/WallTime=117.616M Raws/CPUTime=2.23332 Raws/WallTime=2.23331 WallTime,s=0.447766
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446453 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=117.962M Pixels/WallTime=117.962M Raws/CPUTime=2.23988 Raws/WallTime=2.23989 WallTime,s=0.446453
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=1.15367m CPUTime/WallTime=6.48501u Pixels=0 Pixels/CPUTime=304.437k Pixels/WallTime=305.025k Raws/CPUTime=5.7807m Raws/WallTime=5.79188m WallTime,s=1.15591m
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time               560 ms        560 ms          1 CPUTime,s=0.560023 CPUTime/WallTime=1.00001 Pixels=27.9936M Pixels/CPUTime=49.9865M Pixels/WallTime=49.9872M Raws/CPUTime=1.78564 Raws/WallTime=1.78567 WallTime,s=0.560015
2K4A9928.CR2/threads:8/real_time               562 ms        562 ms          1 CPUTime,s=0.562337 CPUTime/WallTime=0.999994 Pixels=27.9936M Pixels/CPUTime=49.7808M Pixels/WallTime=49.7805M Raws/CPUTime=1.77829 Raws/WallTime=1.77828 WallTime,s=0.56234
2K4A9928.CR2/threads:8/real_time_mean          561 ms        561 ms          1 CPUTime,s=0.561072 CPUTime/WallTime=1 Pixels=27.9936M Pixels/CPUTime=49.8932M Pixels/WallTime=49.8933M Raws/CPUTime=1.78231 Raws/WallTime=1.78231 WallTime,s=0.561071
2K4A9928.CR2/threads:8/real_time_median        561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=1.17202m CPUTime/WallTime=10.7929u Pixels=0 Pixels/CPUTime=104.164k Pixels/WallTime=104.612k Raws/CPUTime=3.721m Raws/WallTime=3.73701m WallTime,s=1.17706m
2K4A9929.CR2/threads:8/real_time               305 ms        305 ms          2 CPUTime,s=0.305436 CPUTime/WallTime=0.999926 Pixels=12.4416M Pixels/CPUTime=40.7339M Pixels/WallTime=40.7309M Raws/CPUTime=3.27401 Raws/WallTime=3.27376 WallTime,s=0.305459
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time               307 ms        307 ms          2 CPUTime,s=0.30724 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.4947M Pixels/WallTime=40.4944M Raws/CPUTime=3.25478 Raws/WallTime=3.25475 WallTime,s=0.307243
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.306145 CPUTime/WallTime=0.999972 Pixels=12.4416M Pixels/CPUTime=40.6398M Pixels/WallTime=40.6387M Raws/CPUTime=3.26645 Raws/WallTime=3.26635 WallTime,s=0.306154
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=961.851u CPUTime/WallTime=40.2708u Pixels=0 Pixels/CPUTime=127.481k Pixels/WallTime=126.577k Raws/CPUTime=0.0102463 Raws/WallTime=0.0101737 WallTime,s=955.105u
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time                       -0.0038         -0.0038           447           446           447           446
2K4A9927.CR2/threads:8/real_time                       +0.0031         +0.0031           444           446           444           446
2K4A9927.CR2/threads:8/real_time                       -0.0031         -0.0031           447           446           447           446
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  +0.0006         +0.0006           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                -0.0022         -0.0022           447           446           447           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.3161         -0.3175             2             1             2             1
2K4A9928.CR2/threads:8/real_time                       -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time                       -0.0009         -0.0008           561           561           561           561
2K4A9928.CR2/threads:8/real_time                       -0.0169         -0.0168           570           561           570           561
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  -0.0066         -0.0065           565           561           565           561
2K4A9928.CR2/threads:8/real_time_median                -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time_stddev                -0.7620         -0.7630             5             1             5             1
2K4A9929.CR2/threads:8/real_time                       -0.0034         -0.0034           306           305           306           305
2K4A9929.CR2/threads:8/real_time                       -0.0146         -0.0146           310           305           310           305
2K4A9929.CR2/threads:8/real_time                       -0.0128         -0.0129           309           305           309           305
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  -0.0081         -0.0080           309           306           309           306
2K4A9929.CR2/threads:8/real_time_median                -0.0119         -0.0119           309           306           309           306
2K4A9929.CR2/threads:8/real_time_stddev                -0.4931         -0.4894             2             1             2             1
$ ~/src/googlebenchmark/tools/compare.py -a benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpjrD2I0
2018-09-12 20:24:11
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446077 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.061M Pixels/WallTime=118.062M Raws/CPUTime=2.24177 Raws/WallTime=2.24178 WallTime,s=0.446076
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445676 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.167M Pixels/WallTime=118.168M Raws/CPUTime=2.24378 Raws/WallTime=2.2438 WallTime,s=0.445673
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=820.275u CPUTime/WallTime=3.51513u Pixels=0 Pixels/CPUTime=216.876k Pixels/WallTime=217.229k Raws/CPUTime=4.11809m Raws/WallTime=4.12478m WallTime,s=821.607u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.561843 CPUTime/WallTime=0.999983 Pixels=27.9936M Pixels/CPUTime=49.8247M Pixels/WallTime=49.8238M Raws/CPUTime=1.77986 Raws/WallTime=1.77983 WallTime,s=0.561852
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.561745 CPUTime/WallTime=0.999995 Pixels=27.9936M Pixels/CPUTime=49.8333M Pixels/WallTime=49.8307M Raws/CPUTime=1.78017 Raws/WallTime=1.78008 WallTime,s=0.561774
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=788.58u CPUTime/WallTime=30.2578u Pixels=0 Pixels/CPUTime=69.914k Pixels/WallTime=69.4978k Raws/CPUTime=2.4975m Raws/WallTime=2.48263m WallTime,s=783.873u
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.305718 CPUTime/WallTime=1.00001 Pixels=12.4416M Pixels/CPUTime=40.6964M Pixels/WallTime=40.6967M Raws/CPUTime=3.271 Raws/WallTime=3.27102 WallTime,s=0.305716
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.305737 CPUTime/WallTime=1 Pixels=12.4416M Pixels/CPUTime=40.6939M Pixels/WallTime=40.6945M Raws/CPUTime=3.27079 Raws/WallTime=3.27084 WallTime,s=0.305732
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=584.969u CPUTime/WallTime=7.87539u Pixels=0 Pixels/CPUTime=77.8769k Pixels/WallTime=77.9118k Raws/CPUTime=6.2594m Raws/WallTime=6.2622m WallTime,s=585.232u
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpN9Xhk2
2018-09-12 20:24:18
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.445771 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.142M Pixels/WallTime=118.143M Raws/CPUTime=2.2433 Raws/WallTime=2.24332 WallTime,s=0.445769
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445719 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.156M Pixels/WallTime=118.156M Raws/CPUTime=2.24356 Raws/WallTime=2.24358 WallTime,s=0.445717
2K4A9927.CR2/threads:8/real_time_stddev          0 ms          0 ms          2 CPUTime,s=184.23u CPUTime/WallTime=1.69441u Pixels=0 Pixels/CPUTime=48.8185k Pixels/WallTime=49.0086k Raws/CPUTime=926.975u Raws/WallTime=930.585u WallTime,s=184.946u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.562071 CPUTime/WallTime=0.999969 Pixels=27.9936M Pixels/CPUTime=49.8045M Pixels/WallTime=49.803M Raws/CPUTime=1.77914 Raws/WallTime=1.77908 WallTime,s=0.562088
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.56237 CPUTime/WallTime=0.999986 Pixels=27.9936M Pixels/CPUTime=49.7779M Pixels/WallTime=49.7772M Raws/CPUTime=1.77819 Raws/WallTime=1.77816 WallTime,s=0.562378
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=967.38u CPUTime/WallTime=35.0024u Pixels=0 Pixels/CPUTime=85.7806k Pixels/WallTime=84.0545k Raws/CPUTime=3.06429m Raws/WallTime=3.00263m WallTime,s=947.993u
2K4A9929.CR2/threads:8/real_time_mean          307 ms        307 ms          2 CPUTime,s=0.306511 CPUTime/WallTime=0.999995 Pixels=12.4416M Pixels/CPUTime=40.5924M Pixels/WallTime=40.5922M Raws/CPUTime=3.26264 Raws/WallTime=3.26262 WallTime,s=0.306513
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.306169 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6364M Pixels/WallTime=40.637M Raws/CPUTime=3.26618 Raws/WallTime=3.26622 WallTime,s=0.306164
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=2.25763m CPUTime/WallTime=21.4306u Pixels=0 Pixels/CPUTime=298.503k Pixels/WallTime=299.126k Raws/CPUTime=0.0239924 Raws/WallTime=0.0240424 WallTime,s=2.26242m
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  -0.0007         -0.0007           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                +0.0001         +0.0001           446           446           446           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.7746         -0.7749             1             0             1             0
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  +0.0004         +0.0004           562           562           562           562
2K4A9928.CR2/threads:8/real_time_median                +0.0011         +0.0011           562           562           562           562
2K4A9928.CR2/threads:8/real_time_stddev                +0.2091         +0.2270             1             1             1             1
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  +0.0026         +0.0026           306           307           306           307
2K4A9929.CR2/threads:8/real_time_median                +0.0014         +0.0014           306           306           306           306
2K4A9929.CR2/threads:8/real_time_stddev                +2.8652         +2.8585             1             2             1             2
```
@coveralls
Copy link

Coverage Status

Coverage remained the same at 88.351% when pulling ded5ecb on LebedevRI:tooling-display-aggregates-only into c614dfc on google:master.

@AppVeyorBot
Copy link

❌ Build benchmark 1432 failed (commit 4e336bc522 by @LebedevRI)

@LebedevRI
Copy link
Collaborator Author

LebedevRI commented Sep 13, 2018 •
edited

As a follow-up, i think i need to somehow rework the generate_difference_report(),
to better handle differences in the count of runs between benchmarks, at least for the U test needs.

@LebedevRI LebedevRI mentioned this pull request Sep 17, 2018
Merged
@LebedevRI
Copy link
Collaborator Author

Thank you for the review!

@LebedevRI LebedevRI merged commit a5e9c06 into google:master Sep 17, 2018
@LebedevRI LebedevRI deleted the tooling-display-aggregates-only branch September 17, 2018 08:59
JBakamovic pushed a commit to JBakamovic/benchmark that referenced this pull request Dec 6, 2018
@LebedevRI
[Tooling] 'display aggregates only' support. (google#674)
4916098 
Works as you'd expect, both for the benchmark binaries, and
the jsons (naturally, since the tests need to work):
```
$ ~/src/googlebenchmark/tools/compare.py benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpYoui5H
2018-09-12 20:23:47
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.447302 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.738M Pixels/WallTime=117.738M Raws/CPUTime=2.23562 Raws/WallTime=2.23564 WallTime,s=0.4473
2K4A9927.CR2/threads:8/real_time               444 ms        444 ms          2 CPUTime,s=0.444228 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.552M Pixels/WallTime=118.553M Raws/CPUTime=2.2511 Raws/WallTime=2.25111 WallTime,s=0.444226
2K4A9927.CR2/threads:8/real_time               447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=0.999999 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446171 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.037M Pixels/WallTime=118.038M Raws/CPUTime=2.24131 Raws/WallTime=2.24132 WallTime,s=0.44617
2K4A9927.CR2/threads:8/real_time_median        447 ms        447 ms          2 CPUTime,s=0.446983 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=117.822M Pixels/WallTime=117.822M Raws/CPUTime=2.23722 Raws/WallTime=2.23722 WallTime,s=0.446984
2K4A9927.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.69052m CPUTime/WallTime=3.53737u Pixels=0 Pixels/CPUTime=448.178k Pixels/WallTime=448.336k Raws/CPUTime=8.51008m Raws/WallTime=8.51308m WallTime,s=1.6911m
2K4A9928.CR2/threads:8/real_time               563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999824 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.561328 CPUTime/WallTime=0.999917 Pixels=27.9936M Pixels/CPUTime=49.8703M Pixels/WallTime=49.8662M Raws/CPUTime=1.78149 Raws/WallTime=1.78134 WallTime,s=0.561375
2K4A9928.CR2/threads:8/real_time               570 ms        570 ms          1 CPUTime,s=0.570423 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.0752M Pixels/WallTime=49.0691M Raws/CPUTime=1.75308 Raws/WallTime=1.75287 WallTime,s=0.570493
2K4A9928.CR2/threads:8/real_time_mean          565 ms        565 ms          1 CPUTime,s=0.564754 CPUTime/WallTime=0.999872 Pixels=27.9936M Pixels/CPUTime=49.5703M Pixels/WallTime=49.564M Raws/CPUTime=1.77077 Raws/WallTime=1.77055 WallTime,s=0.564826
2K4A9928.CR2/threads:8/real_time_median        563 ms        563 ms          1 CPUTime,s=0.562511 CPUTime/WallTime=0.999876 Pixels=27.9936M Pixels/CPUTime=49.7654M Pixels/WallTime=49.7567M Raws/CPUTime=1.77774 Raws/WallTime=1.77743 WallTime,s=0.56261
2K4A9928.CR2/threads:8/real_time_stddev          5 ms          5 ms          1 CPUTime,s=4.945m CPUTime/WallTime=46.3459u Pixels=0 Pixels/CPUTime=431.997k Pixels/WallTime=432.061k Raws/CPUTime=0.015432 Raws/WallTime=0.0154343 WallTime,s=4.94686m
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.306476 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.5957M Pixels/WallTime=40.5941M Raws/CPUTime=3.2629 Raws/WallTime=3.26277 WallTime,s=0.306488
2K4A9929.CR2/threads:8/real_time               310 ms        310 ms          2 CPUTime,s=0.309978 CPUTime/WallTime=0.999939 Pixels=12.4416M Pixels/CPUTime=40.1371M Pixels/WallTime=40.1347M Raws/CPUTime=3.22604 Raws/WallTime=3.22584 WallTime,s=0.309996
2K4A9929.CR2/threads:8/real_time               309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999987 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_mean          309 ms        309 ms          2 CPUTime,s=0.308628 CPUTime/WallTime=0.999962 Pixels=12.4416M Pixels/CPUTime=40.3136M Pixels/WallTime=40.3121M Raws/CPUTime=3.24023 Raws/WallTime=3.24011 WallTime,s=0.308639
2K4A9929.CR2/threads:8/real_time_median        309 ms        309 ms          2 CPUTime,s=0.30943 CPUTime/WallTime=0.999961 Pixels=12.4416M Pixels/CPUTime=40.2081M Pixels/WallTime=40.2076M Raws/CPUTime=3.23175 Raws/WallTime=3.23171 WallTime,s=0.309434
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=1.88354m CPUTime/WallTime=23.7788u Pixels=0 Pixels/CPUTime=246.821k Pixels/WallTime=246.914k Raws/CPUTime=0.0198384 Raws/WallTime=0.0198458 WallTime,s=1.88442m
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_out=/tmp/tmpRShmmf
2018-09-12 20:23:55
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.445589 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.19M Pixels/WallTime=118.19M Raws/CPUTime=2.24422 Raws/WallTime=2.24422 WallTime,s=0.445589
2K4A9927.CR2/threads:8/real_time               446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time               448 ms        448 ms          2 CPUTime,s=0.447763 CPUTime/WallTime=0.999994 Pixels=52.6643M Pixels/CPUTime=117.616M Pixels/WallTime=117.616M Raws/CPUTime=2.23332 Raws/WallTime=2.23331 WallTime,s=0.447766
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446453 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=117.962M Pixels/WallTime=117.962M Raws/CPUTime=2.23988 Raws/WallTime=2.23989 WallTime,s=0.446453
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.446008 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.079M Pixels/WallTime=118.08M Raws/CPUTime=2.24211 Raws/WallTime=2.24213 WallTime,s=0.446005
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=1.15367m CPUTime/WallTime=6.48501u Pixels=0 Pixels/CPUTime=304.437k Pixels/WallTime=305.025k Raws/CPUTime=5.7807m Raws/WallTime=5.79188m WallTime,s=1.15591m
2K4A9928.CR2/threads:8/real_time               561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time               560 ms        560 ms          1 CPUTime,s=0.560023 CPUTime/WallTime=1.00001 Pixels=27.9936M Pixels/CPUTime=49.9865M Pixels/WallTime=49.9872M Raws/CPUTime=1.78564 Raws/WallTime=1.78567 WallTime,s=0.560015
2K4A9928.CR2/threads:8/real_time               562 ms        562 ms          1 CPUTime,s=0.562337 CPUTime/WallTime=0.999994 Pixels=27.9936M Pixels/CPUTime=49.7808M Pixels/WallTime=49.7805M Raws/CPUTime=1.77829 Raws/WallTime=1.77828 WallTime,s=0.56234
2K4A9928.CR2/threads:8/real_time_mean          561 ms        561 ms          1 CPUTime,s=0.561072 CPUTime/WallTime=1 Pixels=27.9936M Pixels/CPUTime=49.8932M Pixels/WallTime=49.8933M Raws/CPUTime=1.78231 Raws/WallTime=1.78231 WallTime,s=0.561071
2K4A9928.CR2/threads:8/real_time_median        561 ms        561 ms          1 CPUTime,s=0.560856 CPUTime/WallTime=0.999996 Pixels=27.9936M Pixels/CPUTime=49.9123M Pixels/WallTime=49.9121M Raws/CPUTime=1.78299 Raws/WallTime=1.78298 WallTime,s=0.560858
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=1.17202m CPUTime/WallTime=10.7929u Pixels=0 Pixels/CPUTime=104.164k Pixels/WallTime=104.612k Raws/CPUTime=3.721m Raws/WallTime=3.73701m WallTime,s=1.17706m
2K4A9929.CR2/threads:8/real_time               305 ms        305 ms          2 CPUTime,s=0.305436 CPUTime/WallTime=0.999926 Pixels=12.4416M Pixels/CPUTime=40.7339M Pixels/WallTime=40.7309M Raws/CPUTime=3.27401 Raws/WallTime=3.27376 WallTime,s=0.305459
2K4A9929.CR2/threads:8/real_time               306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time               307 ms        307 ms          2 CPUTime,s=0.30724 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.4947M Pixels/WallTime=40.4944M Raws/CPUTime=3.25478 Raws/WallTime=3.25475 WallTime,s=0.307243
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.306145 CPUTime/WallTime=0.999972 Pixels=12.4416M Pixels/CPUTime=40.6398M Pixels/WallTime=40.6387M Raws/CPUTime=3.26645 Raws/WallTime=3.26635 WallTime,s=0.306154
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.30576 CPUTime/WallTime=0.999991 Pixels=12.4416M Pixels/CPUTime=40.6908M Pixels/WallTime=40.6908M Raws/CPUTime=3.27054 Raws/WallTime=3.27054 WallTime,s=0.30576
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=961.851u CPUTime/WallTime=40.2708u Pixels=0 Pixels/CPUTime=127.481k Pixels/WallTime=126.577k Raws/CPUTime=0.0102463 Raws/WallTime=0.0101737 WallTime,s=955.105u
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time                       -0.0038         -0.0038           447           446           447           446
2K4A9927.CR2/threads:8/real_time                       +0.0031         +0.0031           444           446           444           446
2K4A9927.CR2/threads:8/real_time                       -0.0031         -0.0031           447           446           447           446
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  +0.0006         +0.0006           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                -0.0022         -0.0022           447           446           447           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.3161         -0.3175             2             1             2             1
2K4A9928.CR2/threads:8/real_time                       -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time                       -0.0009         -0.0008           561           561           561           561
2K4A9928.CR2/threads:8/real_time                       -0.0169         -0.0168           570           561           570           561
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  -0.0066         -0.0065           565           561           565           561
2K4A9928.CR2/threads:8/real_time_median                -0.0031         -0.0029           563           561           563           561
2K4A9928.CR2/threads:8/real_time_stddev                -0.7620         -0.7630             5             1             5             1
2K4A9929.CR2/threads:8/real_time                       -0.0034         -0.0034           306           305           306           305
2K4A9929.CR2/threads:8/real_time                       -0.0146         -0.0146           310           305           310           305
2K4A9929.CR2/threads:8/real_time                       -0.0128         -0.0129           309           305           309           305
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  -0.0081         -0.0080           309           306           309           306
2K4A9929.CR2/threads:8/real_time_median                -0.0119         -0.0119           309           306           309           306
2K4A9929.CR2/threads:8/real_time_stddev                -0.4931         -0.4894             2             1             2             1
$ ~/src/googlebenchmark/tools/compare.py -a benchmarks ~/rawspeed/build-{0,1}/src/utilities/rsbench/rsbench --benchmark_repetitions=3 *
RUNNING: /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpjrD2I0
2018-09-12 20:24:11
Running /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.446077 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.061M Pixels/WallTime=118.062M Raws/CPUTime=2.24177 Raws/WallTime=2.24178 WallTime,s=0.446076
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445676 CPUTime/WallTime=1 Pixels=52.6643M Pixels/CPUTime=118.167M Pixels/WallTime=118.168M Raws/CPUTime=2.24378 Raws/WallTime=2.2438 WallTime,s=0.445673
2K4A9927.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=820.275u CPUTime/WallTime=3.51513u Pixels=0 Pixels/CPUTime=216.876k Pixels/WallTime=217.229k Raws/CPUTime=4.11809m Raws/WallTime=4.12478m WallTime,s=821.607u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.561843 CPUTime/WallTime=0.999983 Pixels=27.9936M Pixels/CPUTime=49.8247M Pixels/WallTime=49.8238M Raws/CPUTime=1.77986 Raws/WallTime=1.77983 WallTime,s=0.561852
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.561745 CPUTime/WallTime=0.999995 Pixels=27.9936M Pixels/CPUTime=49.8333M Pixels/WallTime=49.8307M Raws/CPUTime=1.78017 Raws/WallTime=1.78008 WallTime,s=0.561774
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=788.58u CPUTime/WallTime=30.2578u Pixels=0 Pixels/CPUTime=69.914k Pixels/WallTime=69.4978k Raws/CPUTime=2.4975m Raws/WallTime=2.48263m WallTime,s=783.873u
2K4A9929.CR2/threads:8/real_time_mean          306 ms        306 ms          2 CPUTime,s=0.305718 CPUTime/WallTime=1.00001 Pixels=12.4416M Pixels/CPUTime=40.6964M Pixels/WallTime=40.6967M Raws/CPUTime=3.271 Raws/WallTime=3.27102 WallTime,s=0.305716
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.305737 CPUTime/WallTime=1 Pixels=12.4416M Pixels/CPUTime=40.6939M Pixels/WallTime=40.6945M Raws/CPUTime=3.27079 Raws/WallTime=3.27084 WallTime,s=0.305732
2K4A9929.CR2/threads:8/real_time_stddev          1 ms          1 ms          2 CPUTime,s=584.969u CPUTime/WallTime=7.87539u Pixels=0 Pixels/CPUTime=77.8769k Pixels/WallTime=77.9118k Raws/CPUTime=6.2594m Raws/WallTime=6.2622m WallTime,s=585.232u
RUNNING: /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench --benchmark_repetitions=3 2K4A9927.CR2 2K4A9928.CR2 2K4A9929.CR2 --benchmark_display_aggregates_only=true --benchmark_out=/tmp/tmpN9Xhk2
2018-09-12 20:24:18
Running /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Run on (8 X 4000 MHz CPU s)
CPU Caches:
  L1 Data 16K (x8)
  L1 Instruction 64K (x4)
  L2 Unified 2048K (x4)
  L3 Unified 8192K (x1)
-----------------------------------------------------------------------------------------------
Benchmark                                        Time           CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_mean          446 ms        446 ms          2 CPUTime,s=0.445771 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.142M Pixels/WallTime=118.143M Raws/CPUTime=2.2433 Raws/WallTime=2.24332 WallTime,s=0.445769
2K4A9927.CR2/threads:8/real_time_median        446 ms        446 ms          2 CPUTime,s=0.445719 CPUTime/WallTime=1.00001 Pixels=52.6643M Pixels/CPUTime=118.156M Pixels/WallTime=118.156M Raws/CPUTime=2.24356 Raws/WallTime=2.24358 WallTime,s=0.445717
2K4A9927.CR2/threads:8/real_time_stddev          0 ms          0 ms          2 CPUTime,s=184.23u CPUTime/WallTime=1.69441u Pixels=0 Pixels/CPUTime=48.8185k Pixels/WallTime=49.0086k Raws/CPUTime=926.975u Raws/WallTime=930.585u WallTime,s=184.946u
2K4A9928.CR2/threads:8/real_time_mean          562 ms        562 ms          1 CPUTime,s=0.562071 CPUTime/WallTime=0.999969 Pixels=27.9936M Pixels/CPUTime=49.8045M Pixels/WallTime=49.803M Raws/CPUTime=1.77914 Raws/WallTime=1.77908 WallTime,s=0.562088
2K4A9928.CR2/threads:8/real_time_median        562 ms        562 ms          1 CPUTime,s=0.56237 CPUTime/WallTime=0.999986 Pixels=27.9936M Pixels/CPUTime=49.7779M Pixels/WallTime=49.7772M Raws/CPUTime=1.77819 Raws/WallTime=1.77816 WallTime,s=0.562378
2K4A9928.CR2/threads:8/real_time_stddev          1 ms          1 ms          1 CPUTime,s=967.38u CPUTime/WallTime=35.0024u Pixels=0 Pixels/CPUTime=85.7806k Pixels/WallTime=84.0545k Raws/CPUTime=3.06429m Raws/WallTime=3.00263m WallTime,s=947.993u
2K4A9929.CR2/threads:8/real_time_mean          307 ms        307 ms          2 CPUTime,s=0.306511 CPUTime/WallTime=0.999995 Pixels=12.4416M Pixels/CPUTime=40.5924M Pixels/WallTime=40.5922M Raws/CPUTime=3.26264 Raws/WallTime=3.26262 WallTime,s=0.306513
2K4A9929.CR2/threads:8/real_time_median        306 ms        306 ms          2 CPUTime,s=0.306169 CPUTime/WallTime=0.999999 Pixels=12.4416M Pixels/CPUTime=40.6364M Pixels/WallTime=40.637M Raws/CPUTime=3.26618 Raws/WallTime=3.26622 WallTime,s=0.306164
2K4A9929.CR2/threads:8/real_time_stddev          2 ms          2 ms          2 CPUTime,s=2.25763m CPUTime/WallTime=21.4306u Pixels=0 Pixels/CPUTime=298.503k Pixels/WallTime=299.126k Raws/CPUTime=0.0239924 Raws/WallTime=0.0240424 WallTime,s=2.26242m
Comparing /home/lebedevri/rawspeed/build-0/src/utilities/rsbench/rsbench to /home/lebedevri/rawspeed/build-1/src/utilities/rsbench/rsbench
Benchmark                                                 Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------
2K4A9927.CR2/threads:8/real_time_pvalue                 0.6428          0.6428      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9927.CR2/threads:8/real_time_mean                  -0.0007         -0.0007           446           446           446           446
2K4A9927.CR2/threads:8/real_time_median                +0.0001         +0.0001           446           446           446           446
2K4A9927.CR2/threads:8/real_time_stddev                -0.7746         -0.7749             1             0             1             0
2K4A9928.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9928.CR2/threads:8/real_time_mean                  +0.0004         +0.0004           562           562           562           562
2K4A9928.CR2/threads:8/real_time_median                +0.0011         +0.0011           562           562           562           562
2K4A9928.CR2/threads:8/real_time_stddev                +0.2091         +0.2270             1             1             1             1
2K4A9929.CR2/threads:8/real_time_pvalue                 0.0636          0.0636      U Test, Repetitions: 3. WARNING: Results unreliable! 9+ repetitions recommended.
2K4A9929.CR2/threads:8/real_time_mean                  +0.0026         +0.0026           306           307           306           307
2K4A9929.CR2/threads:8/real_time_median                +0.0014         +0.0014           306           306           306           306
2K4A9929.CR2/threads:8/real_time_stddev                +2.8652         +2.8585             1             2             1             2
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmah42 dmah42 dmah42 approved these changes

@EricWF EricWF Awaiting requested review from EricWF

Assignees
No one assigned
Labels
cla: yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@LebedevRI @coveralls @AppVeyorBot @dmah42 @googlebot