Csmith is a random generator of C programs. It's primary purpose is to find compiler bugs with random programs, using differential testing as the test oracle.
Csmith can be used outside of the field of compiler testing. If your application needs a test suite of C programs and you don't bother to write them, feel free to give Csmith a try.
Csmith outputs C programs free of undefined behaviors (believe us, that's not trivial), and the statistics of each generated program.
You can install Csmith from tarballs downloaded from here (coming soon), or you can build it from the source. The following commands apply to Ubuntu.
git clone https://github.com/csmith-project/csmith.git cd csmith sudo apt install g++ cmake m4 cmake -DCMAKE_INSTALL_PREFIX=<INSTALL-PREFIX> . make && make install
Please see specific instructions for building on Windows.
Suppose Csmith is installed to
$HOME/csmith locally. You can simply
generate, compile, and execute a test case by:
export PATH=$PATH:$HOME/csmith/bin csmith > random1.c gcc random1.c -I$HOME/csmith/include -o random1 ./random1
To add differential testing into the picture, we need to install another compiler, e.g., another version of gcc or clang. And repeat the process of:
csmith > random2.c gcc random2.c -I$HOME/csmith/include -o random2_gcc clang random2.c -I$HOME/csmith/include -o random2_clang ./random2_gcc > gcc_output.txt ./random2_clang > clang_output.txt
If there is any difference in
aha, you have found a bug in either gcc or clang, or, in the
unlikely case, a bug in Csmith itself.
You could write scripts in your favorite language to repeat the above process to amplify the power of random differential testing.
The generate programs might contain infinite loops. The best practice is to apply timeout to their executions.
csmith -h or
csmith -hh to see lists of command line options that you
can pass to Csmith and customize the random generation.
Here is a slightly outdated but still relevant document about using Csmith for compiler testing.
Csmith was originally developed at the University of Utah by:
Csmith was open sourced in 2009. We try to keep maintaining it as an open source project using discretionary times. As much, the response to bug reports or feature requests might be delayed.
Please use github issues to report bugs or suggestions.
We have a mailing list for discussing Csmith. Please visit here to subscribe.