Debug RGBDS with a fuzzer (afl? libfuzzer?)

[ISSOtm]

meithecatte did it back then I think with AFL, but LLVM also has libfuzzer, which seems to be easier to set up.

[aaaaaa123456789]

Basic setup:

• Build the programs as a library (i.e., no main)
• Create the fuzzer entry point (int LLVMFuzzerTestOneInput (const unsigned char *, size_t); must accept any input, run it and return 0)
• Link the whole thing and build with -fsanitize=fuzzer and you're done: the resulting binary fuzzes the program

[Rangi42]

libfuzzer doesn't look appropriate for testing rgbasm/rgblink since the "fuzz target" only takes const uint8_t *data, size_t size as input (i.e. none of our CLI flags) and must not exit() (which we do in many cases of fatal errors).

[aaaaaa123456789]

libfuzzer doesn't look appropriate for testing rgbasm/rgblink since the "fuzz target" only takes const uint8_t *data, size_t size as input (i.e. none of our CLI flags) and must not exit() (which we do in many cases of fatal errors).

There are easy ways around these problems.
The exit issue is solvable by having the fuzz build not do that. You can add a macro at compile time that defines exit to call another function if needed.

As for the input, you're allowed to preprocess it! Define some sort of "file delimiter", and split the input into command line and flags before feeding it to RGBASM:

<CLI options go here>
============
<file 0.asm>
============
<file 1.asm>

You can do all of this preprocessing in your fuzz driver.

[ISSOtm]

Due to the pervasiveness of exit() calls everywhere, I believe such a setup would basically require redirecting exit() to std::thread::exit(), and spawning a new thread that runs the appropriate program. (And due to the pervasiveness of global variables, either you make all of them thread-local, or you cap the number of jobs at 1!)

I judged that too complex, so I gave up and moved on the the Rust port, which has “no global variables” as a design point notably to enable this.

[Rangi42]

If I really wanted to try libfuzzer, I'd replace the exit calls with an exit_wrapper, and then for the fuzz build exit_wrapper would actually throw an exception to be caught by the fuzz driver. That still leaves actually writing the fuzz driver, including somehow replacing/dispatching into all the setup that each tool's own main function currently does with CLI args.