Test runner for file-based regression testing
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
doc
examples
plugins
sisyphus
.gitignore
.lvimrc
LICENCE
MANIFEST.in
README.mdwn
setup.py
sis
testsuite.py

README.mdwn

Sisyphus

A compiler regression testing tool.

How to Use

We assume you have a directory full of test programs. To configure sisyphus for your test programs, you have to write a plugin, unless our included plugins are sufficient. Put the plugin into the program directory as well, sisyphus fill find it automatically.

Now let sisyphus run. It will test all your programs according to your plugin.

Why

A simple shell script can do the job just as well, does it? Dependending on your requirements that might be true. We also started like that. Then the script got more and more complicated until it was rewritten in Python. A few more extensions and rewrites and sisyphus emerged. A few features your shell script might not have:

  • An expectation file, possibly from a URL, to distuingish between expected and unexpected failures. This means you notice, when you accidentally fix a long standing bug.
  • Parallel execution of test, but limited to the number of cores on the machine.
  • Options for cross-compilation, qemu, valgrind, and performance checks.
  • Allows to retry to accomodate tests which sometimes fail.

Concepts

A single test is represented by a file. Test factories generate a test for each file. Those factories also associate a list of steps to perform with the file. Each step in turn has a list of checks associated.

There might be additional files (reference output, header files, etc), which should be ignored by the factories.

Licence

MIT, see LICENCE file for full text