--shell=none always gives the same performance

[RolfSievert]

Background
I use hyperfine to benchmark some custom cpp algorithms, and to do that I usually pipe the input to the executable binary like this:
./binary < sample_input.txt
Most of the time these run at 5 ms or faster and hyperfine suggests to run without the shell, therefore I benchmark with the following command,
hyperfine --shell=none './binary < sample_input.txt'

The issue
However, this seem to always give approximately (varying with ~0.1ms) the same result (~0.5ms mean, ~0.4ms min), even for ridiculously large input.
When I remove the --shell=none option it takes around 100s to complete one run of said large input, and that's the time I get when I run the command without hyperfine. Seem to be independent on what binary I run, but I can provide an example if it's useful.
Am I misunderstanding something or is this a bug? I expect that benchmarks with and without --shell=none would have similar performance and not 0.4ms vs. 100s.

Additional info
I tried to pipe the output of the binary
hyperfine --shell=none './binary < sample_input.txt > output.txt'
but the output only appears if I remove the --shell=none option.

I've used the command with absolute paths, and tried to print the output of the binary with --show-output without any results (only prints without --shell=none). I'm on a linux system, I use zsh, and my hyperfine version is 1.16.1-1.

I suspect that I misunderstand what "disabling the shell" really means, but I couldn't find anything that explains this behavior.

Thanks in advance!

[sharkdp]

therefore I benchmark with the following command,
hyperfine --shell=none './binary < sample_input.txt'

That does not work. You can not use shell syntax like <-redirection when using --shell=none. I tried to describe this in the --help text:

Finally, this can also be set to "none" to disable the shell. In this case, commands will be executed directly. They can still have arguments, but more complex things like "sleep 0.1; sleep 0.2" are not possible without a shell.

Maybe we should include a shell command with pipes here in the help text, to make this more clear.

What happens instead is that your ./binary will be called with two arguments: '<' and 'sample_input_txt'. This might explain the weird results you're seeing. What you can do instead is to use the new --input option:

hyperfine --shell=none --input=sample_input.txt ./binary

I tried to pipe the output of the binary
hyperfine --shell=none './binary < sample_input.txt > output.txt'
but the output only appears if I remove the --shell=none option.

Yes, this also doesn't work. But note that we have also have an output option:

hyperfine --shell=none --input=sample_input.txt --output=/tmp/output.log ./binary

I suspect that I misunderstand what "disabling the shell" really means, but I couldn't find anything that explains this behavior.

Did you see the help text that I quoted above? If not, where did you search for information? I'm always trying to improve the documentation.

[RolfSievert]

Thank you for your answer! That explains everything I needed to know.

This seems to be entirely on me, I mostly use man pages for documentation-lookup and just assumed that it would show the same information that --help outputs. A lot is missing there and it's not nearly as descriptive. Today I learned something 😄

Thank you very much for taking your time, much appreciated!

[sharkdp]

Thank you. Let's keep this open until the man page is updated

[sharkdp]

The man page is now up to date and synced with the --help text.

https://github.com/sharkdp/hyperfine/releases/tag/v1.17.0