[feature]: make fuzz tests first-class citizens

[morehouse]

Problem

LND's fuzz tests failed to build last year for at least 4 months, maybe longer. This wasn't noticed until @Crypt-iQ ran the tests locally.

This bit rot was possible because LND's fuzz tests do not run in CI.

Solution

It is good practice to run each fuzz target on its seed corpus in CI (just a test run, no fuzzing). This turns the seed corpus into a set of regression tests that should be kept up-to-date with the best coverage found by each fuzz target. Especially when bugs are found, the triggering input should be added to the seed corpus.

The seed corpora could be stored in the LND repo or a separate repo similar to bitcoin-core.

Bonus

It would also be very good to set up continuous fuzzing somewhere offline if this isn't already being done. We don't want an attacker to be able to discover and exploit a bug simply because they ran our existing fuzz tests more than us.

#5965 is one continuous fuzzing option, or we could do something homegrown.

[Crypt-iQ]

The seed corpora could be stored in the LND repo or a separate repo similar to bitcoin-core.

These should be stored in a separate repo so that git clone github.com/lightningnetwork/lnd doesn't download GBs of seeds. Do you want to work on the open-source CI part?

[morehouse]

I'm happy to work on the CI. Just let me know once you have a GitHub repo set up for the corpora.

[morehouse]

After some tinkering, I think we should use libFuzzer as our fuzzing engine instead of Go's native engine.

As of Go 1.20, the native engine has practically no configuration options, and no ability to minimize a corpus or merge new inputs into an existing corpus. The latter makes maintaining the https://github.com/lightninglabs/lnd-fuzz repo more difficult: we'd need to build some new tool to do our own minimization.

Using libFuzzer instead gets us easy minimization and corpus merging, plus all the config options, at the cost of slightly more complexity to build and run the fuzz tests. The complexity is similar (or slightly less) to our previous method using go-fuzz.

Note that other projects serious about fuzzing also use libFuzzer mode for similar reasons (e.g., OSS-Fuzz).

[morehouse]

After some tinkering, I think we should use libFuzzer as our fuzzing engine instead of Go's native engine.

I played around with https://github.com/AdamKorcz/go-118-fuzz-build as a way of using libFuzzer with Go native fuzz targets, and I'm pretty disappointed by it. I ran into bugs (e.g., input bytes dropped) and limitations (e.g., only 1 byte integers), giving a rather unpolished feeling to the whole project.

It seems native Go fuzzing will be less of a headache for us. I've implemented a corpus merging/minimization script that we can use until Go supports merging natively. It seems to be working pretty well so far.

[morehouse]

We now have

• seed corpora for all fuzz tests at https://github.com/lightninglabs/lnd-fuzz
• fuzz tests run on seeds during CI
• a corpus merging script
• a short fuzzing contributor's guide

This satisfies the main purpose of this issue. A continuous fuzzing solution would still be good, but we can use #5965 or a new issue to track that.