Add new --parameter-list option for non-numeric parametrized benchmarks

[sharkdp]

Add a new option

--parameter-list <VAR> <VALUES>

that can be used like this:

hyperfine \
  --parameter-list branch_name master,feature1,bugfix,my-test-feature \
  --prepare "git checkout {branch_name}; make clean" \
  "make"

This would work just like the --parameter-scan option, but with non-numeric parameter values.

Maybe there is a better name for this command-line option.

see #216

[JuanPotato]

Hi! I'm interested in taking this on if you aren't yet. I feel like --parameter-list is a fine name, --parameter-values could be another option.

[sharkdp]

I'm interested in taking this on if you aren't yet.

Sounds great!

[JuanPotato]

Should parameter-list be able to be used with parameter-scan? Along the same lines, should parameter-list be able to be used twice? That way you could do something like hyperfine --parameter-list filename a,b,c --parameter-list ext csv,json,txt cat {filename}.{ext}

But then the question would be do we combine the two parameter lists in a zip fashion or just all combinations? (a.csv, b.json, c.txt or a.csv, b.csv, c.csv, a.json, b.json, c.json, a.txt, b.txt, c.txt). I think it would be simpler to only allow either list, or scan, but not both.

[sharkdp]

Disallowing --parameter-list and --parameter-scan to be used at the same time sounds fine for me.

Allowing multiple --parameter-list options could be useful, but I think we can skip that for now. If you want to implement it, I'd probably go with the zip-version, not the all-combinations-version. If a user really wants all combinations, they would have to be provided by hand.

[JuanPotato]

For turning the list of parameters into an actual list, master,bugfix,feature1 -> ["master", "bugfix", "feature1"], a split on comma would work but wouldn't allow list items with commas in them. Is there an existing format or tokenizer that is used in hyperfine for this sort of thing?