Add --subtract option

[jcgruenhage]

This has been suggested as a possible solution in #170, and would also sort of solve #39. My usecase for this is that I want to benchmark page loads, for which I'm using headless chromium, but want to subtract the time that chromium takes to start up. Here a small example of what I mean, this is how it looks today:

╭─jcgruenhage@dreamy-tesla.jcg.re ~/dev/peshming  ‹master› 
╰─➤  hyperfine "chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://gitlab.com/jcgruenhage/peshming/-/blob/master/Cargo.lock" "chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://git.sr.ht/\~jcgruenhage/peshming/tree/master/Cargo.lock"
Benchmark #1: chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://gitlab.com/jcgruenhage/peshming/-/blob/master/Cargo.lock
  Time (mean ± σ):      4.293 s ±  0.373 s    [User: 1.481 s, System: 0.382 s]
  Range (min … max):    3.867 s …  4.945 s    10 runs
 
Benchmark #2: chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://git.sr.ht/\~jcgruenhage/peshming/tree/master/Cargo.lock
  Time (mean ± σ):      2.201 s ±  0.172 s    [User: 849.7 ms, System: 264.5 ms]
  Range (min … max):    2.076 s …  2.651 s    10 runs
 
Summary
  'chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://git.sr.ht/\~jcgruenhage/peshming/tree/master/Cargo.lock' ran
    1.95 ± 0.23 times faster than 'chromium --headless --disk-cache-dir=/dev/null --disable-gpu --download-whole-document --deterministic-fetch https://gitlab.com/jcgruenhage/peshming/-/blob/master/Cargo.lock'
╭─jcgruenhage@dreamy-tesla.jcg.re ~/dev/peshming  ‹master› 
╰─➤  hyperfine 'chromium --headless'
Benchmark #1: chromium --headless
  Time (mean ± σ):     625.0 ms ±  58.9 ms    [User: 441.6 ms, System: 184.4 ms]
  Range (min … max):   548.4 ms … 740.7 ms    10 runs

I can do the math myself for now, but adding this in hyperfine would be really cool.

As for how I think this could be implemented: Currently, the shell spawn time is measured and removed (#15), and a --subtract option could just be a more generic version of that, with an empty command being the default and serving as the replacement for the shell spawn time. How the subtraction is calculated into the result can just be taken from the shell spawn time subtraction for now, but IMO that should change: Right now, you just subtract the mean shell spawn time from every single execution time, but wouldn't it make more sense to measure all the execution times and then calculate the shell spawn times into mean, min and max? I mean the shell spawn time shouldn't change enough that this is actually a problem, but it feels a bit off, and when things like chromium start up are measured, it starts to make a small difference.

[sharkdp]

Thank you for the feedback.

As for how I think this could be implemented: Currently, the shell spawn time is measured and removed (#15), and a --subtract option could just be a more generic version of that, with an empty command being the default and serving as the replacement for the shell spawn time.

Interesting idea!

but wouldn't it make more sense to measure all the execution times and then calculate the shell spawn times into mean, min and max? I mean the shell spawn time shouldn't change enough that this is actually a problem, but it feels a bit off, and when things like chromium start up are measured, it starts to make a small difference.

Oh, you mean we would measure initialization_{mean,min,max} (where initialization would be shell spawning or shell spawning + --subtract command) and the compute the command_{mean,min,max} times from the actually measured total_{mean,min,max} via something like:

• command_min = total_min - initialization_max
• command_max = total_max - initialization_min
• command_mean = total_mean - initialization_mean

Sounds like a great idea to me. The only issue I see is that we would have to deal with negative times in some way or another.

This would be quite difficult to communicate to the user, I guess. Setting negative times to 0 could be an option, but also doesn't feel quite right, if we do not print a warning or similar.

[jcgruenhage]

Oh, right, that can result in negative times, hadn't thought of that. Maybe adding a note about this to the man page/--help output is enough? And then printing a warning if the init times are too large (and vary too much) relatively to the benchmarked command

[sharkdp]

Yeah, I think that could work.

I'm inclined to accept this as a new feature request if it can be cleanly implemented within the current codebase (i.e. without requiring very large scale changes to the program logic).

[mralusw]

I've worked around this by creating a custom "shell" (which is just a shell script, but can be used as a -S argument). The script simply weighs down all benchmarked commands, except for the one (./myprog) that has a sizable startup time. This assumes myprog has some sort of input that makes it quit immediately after startup. For any command other than myprog, add myprog quit as an overhead:

#!/bin/sh
case "$2" in ('./myprog'*) ;; (*) ./myprog 'quit' ;; esac
exec /bin/sh "$@"