Follow fuzzing best practices in CI

[morehouse]

It is good practice to run each fuzz target on its seed corpus in CI (just a run, no fuzzing). This turns the seed corpus into a set of regression tests that should be kept up-to-date with the best coverage found by each fuzz target. Especially when bugs are found, the triggering input should be added to the seed corpus.

The seed corpora could be stored in a separate repo similar to bitcoin-core.

There are several benefits of adopting this practice:

• Prevent fuzz target bit rot.
• More test coverage in CI.
• Greater visibility of fuzz tests, encouraging contributions to the seed corpus and tests themselves.

Bonus

Continuous fuzzing is also a good security/testing practice to find bugs over time. There are many ways to accomplish it, but probably the easiest are

• OSS-Fuzz to run on Google's platform.
• ClusterFuzzLite for a more self-sovereign solution.

[rustyrussell]

❤️

I have a branch where I tried to use AFL for better fuzzing. But it requires us to rewrite all our fuzz tests, which then ties us to AFL ;( Perhaps this is a better approach (though many of our fuzz tests are incomplete, or useless, anyway). It quickly complained about bad use of aliases inside ccan/tal, which is serious work to fix...

I don't think the corpora will be giant, though perhaps you're right about a separate repo! What do you need to do this?

[morehouse]

I have a branch where I tried to use AFL for better fuzzing. But it requires us to rewrite all our fuzz tests, which then ties us to AFL ;( Perhaps this is a better approach (though many of our fuzz tests are incomplete, or useless, anyway). It quickly complained about bad use of aliases inside ccan/tal, which is serious work to fix...

AFL actually does support libFuzzer-style fuzz targets with LLVM instrumentation, so we wouldn't need to be tied down to AFL. I can take a look at that in the future as well, probably with a different issue to track progress.

I don't think the corpora will be giant, though perhaps you're right about a separate repo! What do you need to do this?

If you're fine with storing corpora in-tree, I can get started now. Otherwise we'll need an official repo to use for fuzz corpora.

[rustyrussell]

I have a branch where I tried to use AFL for better fuzzing. But it requires us to rewrite all our fuzz tests, which then ties us to AFL ;( Perhaps this is a better approach (though many of our fuzz tests are incomplete, or useless, anyway). It quickly complained about bad use of aliases inside ccan/tal, which is serious work to fix...

AFL actually does support libFuzzer-style fuzz targets with LLVM instrumentation, so we wouldn't need to be tied down to AFL. I can take a look at that in the future as well, probably with a different issue to track progress.

It does, but you don't want to do that. You want to use the AFL infra which is much much faster (shared memory and process snapshots).

See my AFL fuzzing branch (be warned, it's a bit of a mess and unfinished!).

I don't think the corpora will be giant, though perhaps you're right about a separate repo! What do you need to do this?

If you're fine with storing corpora in-tree, I can get started now. Otherwise we'll need an official repo to use for fuzz corpora.

[morehouse]

It does, but you don't want to do that. You want to use the AFL infra which is much much faster (shared memory and process snapshots).

I would think in-process fuzzing libFuzzer-style would be faster than restoring process snapshots every iteration. AFL++ folks at least don't discourage this mode.

Regardless, if we want to use multiple fuzzing engines, the LLVM-style driver is the way to go. It's compatible with libFuzzer, AFL++, honggfuzz, etc.