Investigate soltix

[axic]

See https://medium.com/chainsecurity/soltix-scalable-testing-of-solidity-compilers-57be270e802b

SOLTIX is a framework for automated testing of Solidity compilers.

It seems to have two modes:

• random code generation
• code mutation

Maybe we can run it as part of ossfuzz?

Apparently soltix was used to disover #5130 and #4893.

[bshastry]

Sorry for the late update. Progress made is as follows

• Took a look at the soltix architecture
• Successfully built and ran soltix

Regarding running it as part of ossfuzz: soltix seems to be a test framework of its own that is written in Java. It creates test input (solidity contracts) and executes it on an EVM implementation. If I understand correctly it uses Truffle and Ganache/geth to generate solidity programs and execute the resulting byte code respectively.

So, it is not clear to me how we interface with ossfuzz. One option worth trying is to use the programs generated by soltix as seeds for our existing solidity fuzzers. However, I'm sceptical if this is what we should really be doing because these fuzzers target parsing/code generation issues (mostly flag internal compiler exceptions) and not semantic errors like #4893

OTOH, tangentially to soltix, we could create an ossfuzz based test harness to catch issues like #4893. Here's what we would need for that

• some form of execution tracing support in evmone (like the trace as json option of geth)
• a protobuf grammar for solidity programs (I started work on this in this branch: https://github.com/ethereum/solidity/tree/solProto)

Then, we do something like

string solidityProgram = convertProtoToSol(solidityProto);
string unoptimizedTrace = evmone::runAndTrace(solidityProgram);
// Potentially do two runs: one with standard opt and one with yul
string optimizedTrace = evmone::runAndTrace(compileContract(solidityProgram));
solAssert(unoptimizedTrace == optimizedTrace);
return;

What do you think?

[github-actions]

Hi everyone! This issue has been closed due to inactivity.
If you think this issue is still relevant in the latest Solidity version and you have something to contribute, feel free to reopen.
However, unless the issue is a concrete proposal that can be implemented, we recommend starting a language discussion on the forum instead.