Save fuzzer/benchmark version in database, show them in the report

[ammaraskar]

With multiple reports over a lot of different days, this should help keep track of if the latest fuzzer was used in a particular experiment.

(Maybe even link the github commit to make it even more convenient.)

[lszekeres]

Yes! We are planning to indicate the versions of the fuzzers, and the benchmarks too.

[jonathanmetzman]

@inferno-chromium @mbarbella-chromium
(@lszekeres @lmrs2 also please share thoughts)

I can think of two ways to determine version.

1. Make users set an environment variable (e.g. FUZZBENCH_FUZZER_VERSION) at build time that gets saved.
2. Use the method OSS-Fuzz uses for finding out git repo versions automagically (I think it searches directories in SRC iirc).

I lean towards option 2 because it doesn't make integration harder but option 1 is nice for cases like entropic where we apply a patch to a repo.

[lszekeres]

I think we should automatically save the repo, branch, and the hash of the checked out version (e.g., git rev-parse HEAD). We should avoid relying on users manually setting versions also because we might want to track and benchmark some fuzzers at the HEAD of their master branch.

[jonathanmetzman]

I think we should automatically save the repo, branch, and the hash of the checked out version (e.g., git rev-parse HEAD). We should avoid relying on users manually setting versions also because we might want to track and benchmark some fuzzers at the HEAD of their master branch.

I forgot that forcing users to set the version means that the version is pinned and must be updated.
This seems like it might be a good thing though since results can be easily reproduced if we aren't simply building from master. WDYT?

[lszekeres]

I think less stable or less maintained fuzzers are better pinned to a specific working version. However for stable and actively maintained fuzzers (like libfuzzer, honggfuzz, afl, afl++ ...) it's probably easier to use the head of master by default, as it'd be good to automatically track regressions/improvements in these, and that way we don't need to manually bump versions all the time. (With the versions saved, reproducing an earlier experiment shouldn't be an issue.)

[inferno-chromium]

I dont think we can autoroll new versions/hashes as fuzzers will break with benchmark over time. We do this in OSS-Fuzz where project author have a responsibility to fix their build. We don't have that responsibility here, so manually bumping might be the way to go, for atleast next six months or so.

But i do agree that fuzzer repo hash is better rather than with some weirdly explicitly specified / hardcoded version.

[lszekeres]

Yes, it's a good idea to have fuzzer maintainers explicitly opt in for fuzzbench auto tracking, with responsibility to fix broken builds.