migrate runner.py features into oss-fuzz/infra/helper.py

[DavidKorczynski]

Making an issue of #525 (comment)

oss_fuzz_integration/runner.py has a few features that are convenient for building and running fuzzers by way of oss-fuzz, including:

• automatically downloading public corpus, which can be used to construct full coverage reports
• run commands such as: python3 ../runner.py {coverage | introspector} proj_name exec_sec which will build fuzzers of proj_name with the default sanitizer, run the fuzzers for exec_sec seconds and then generate a coverage or introspector report.

The features are useful when improving fuzzers for a given project as it makes the workflow fast.

Some of these features would make sense to add to OSS-Fuzz, in particular coverage generation using public corpus, generation of fuzz introspector reports for a given project and also generation just coverage for a given project.

[oliverchang]

+1 to this!

Note that we already have a coverage command in helper.py which has some functionality to download corpora and generate coverage reports: https://github.com/google/oss-fuzz/blob/f2d79d7877f74b6086abf6e3aa97d8a89b41923c/infra/helper.py#L885

Is the difference here just the ability to generate an introspector report?

@evverx

[evverx]

Note that we already have a coverage command in helper.py which has some functionality to download corpora and generate coverage reports

helper.py makes it possible to download the latest corpora but it requires setting up gsutil, tokens and so on. I tend to run helper.py in throw-away VMs so there it's just easier to download public corpora and point helper.py to them without having to deal with any tokens. It also helps to build coverage/FI reports for projects I don't have access to.

[evverx]

Is the difference here just the ability to generate an introspector report?

Apart from downloading public corpora automatically I think it is. Right now to build FI reports it's necessary to clone the fuzz-introspector repository and then run runner.py (which does a few things using helper.py under the hood). It works but I think ideally it should be included in helper.py. Among other things it would automatically make it possible to pass paths to source code when fuzz targets are built (i.e. ./helper.py build_fuzzers project ~/path/to/project/source/code) to figure out whether changes/patches have improved anything.

[evverx]

Before I forget if it's decided that it should be possible to download public corpora via helper.py it should construct urls by analogy with https://github.com/google/oss-fuzz/blob/7797279c274d10197d62841dc43834238fd483a1/infra/cifuzz/clusterfuzz_deployment.py#L295-L298

    project_qualified_fuzz_target_name = target_name
    qualified_name_prefix = self.config.oss_fuzz_project_name + '_'
    if not target_name.startswith(qualified_name_prefix):
      project_qualified_fuzz_target_name = qualified_name_prefix + target_name

I don't think there are a lot of projects where it would be necessary but as far as I can remember libidn2 would be one of them.

[evverx]

The features are useful when improving fuzzers for a given project as it makes the workflow fast.

I'd add that it would make it easier to fuzz projects that aren't integrated into OSS-Fuzz. For example I fuzzed a small library today so I put its fuzz target in the OSS-Fuzz repository to make it easier to fuzz it locally using the OSS-Fuzz toolchain. Once a couple of shallow bugs were fixed there I ran the fuzz target for a couple of hours to build up a corpora and was on the way to use them to build FI reports to figure out how it fared in general. I ended up cloning the FI repository, copying the OSS-Fuzz repository with that local project to the oss_fuzz_integration directory and pointing the runner script to various places. In the end I built the report but it would have been much easier if I could have just run ./infra/helper.py introspector directly from inside my local copy of the OSS-Fuzz repository.

[DavidKorczynski]

@evverx I'll start this in the coming days!

[evverx]

@DavidKorczynski good to know. Thanks!

FWIW I'm not sure how to fix that "infra/helper.py" bug where it can't be run a few times in a row with local build directories. Unfortunately the patch where I added the ":O" option to get it to destroy any changes to build directories works with podman only. On the other hand if the script is going to be integrated into helper.py directly maybe it can avoid that bug somehow.

[DavidKorczynski]

Before I forget if it's decided that it should be possible to download public corpora via helper.py it should construct urls by analogy with https://github.com/google/oss-fuzz/blob/7797279c274d10197d62841dc43834238fd483a1/infra/cifuzz/clusterfuzz_deployment.py#L295-L298

    project_qualified_fuzz_target_name = target_name
    qualified_name_prefix = self.config.oss_fuzz_project_name + '_'
    if not target_name.startswith(qualified_name_prefix):
      project_qualified_fuzz_target_name = qualified_name_prefix + target_name

I don't think there are a lot of projects where it would be necessary but as far as I can remember libidn2 would be one of them.

Thanks for the link here. I got this started in google/oss-fuzz#9155 and will iterate over this issue in a few PRs on the oss-fuzz side incrementally!

[evverx]

@DavidKorczynski now that FI is integrated into helper.py (and the documentation is updated) I think this issue can be closed. The only feature I'd add to make it perfect would be a way to point helper.py to directories where FI reports should be stored to make it easier to compare several FI reports produced by subsequent runs but it's probably a separate feature request and I can probably send a PR sooner or later. Thanks!

[DavidKorczynski]

@DavidKorczynski now that FI is integrated into helper.py (and the documentation is updated) I think this issue can be closed.

Great! Am glad the recent additions improved things!

The only feature I'd add to make it perfect would be a way to point helper.py to directories where FI reports should be stored to make it easier to compare several FI reports produced by subsequent runs but it's probably a separate feature request and I can probably send a PR sooner or later. Thanks!

Sounds good! Let's track diffing of runs here #734 -- fyi this is in my todo list for the near future, and we should probably have some updates in mid january!