fuzz test the compiler

[chriseth]

Set up a server to continuously fuzz test the compiler with the goal to find crashes.

There are two kinds of crashes:

1. Actual crashes (segmentation fault and others)
2. internal compiler errors

For the second kind, it is hard to find "unanticipated" crashes because some internal compiler errors correctly signal non-implemented features. We might split this exception class into two different classes in the future to be able to detect "real" errors.

[pirapira]

Very similar project CSmith.

Some months ago I was running solc under American Fuzzy Lop, but didn't find a crash back then.

[axic]

We might split this exception class into two different classes in the future to be able to detect "real" errors.

I think this should be done in any case.

[chriseth]

I am using libfuzzer (from the llvm project) manually now and it detects quite a lot of crashes. The problem is that it exits for the first crash, not sure if that is intended use.

[pirapira]

Yes that is the intended use.

[holiman]

I am running an AFL-fuzz now, found it pretty straight-forward to set up.

Build and install AFL

[~/tools/afl/afl-2.35b]
#make
[~/tools/afl/afl-2.35b]
#sudo make install

Instrument the solidity source:

[~/workspace/solidity/build]
#cmake -DCMAKE_CXX_COMPILER=afl-g++ -DCMAKE_C_COMPILER=afl-gcc .. && make

Create a sample input file, and an output directory, then do fuzzing

#afl-fuzz -i /home/martin/workspace/solidity/testcases/input -o /home/martin/workspace/solidity/testcases/output /home/martin/workspace/solidity/build/solc/solc

It's running, no crashes yet.

[chriseth]

@holiman we do not only want to detect segmentation faults but also InternalCompilerErrors. The problem with the latter is that this is still blocked by #1133

Furthermore, if you want to get better example input, you can use https://github.com/ethereum/solidity/blob/develop/scripts/isolateTests.py to extract the end to end test cases from the .cpp file.

[chriseth]

Closing this as we have the necessary infrastructure in place now.