batterseapower edited this page Sep 13, 2010 · 9 revisions
Clone this wiki locally

Test Framework

You can help improve this README with extra snippets and advice by using the GitHub wiki.


To just install the library:

runghc Setup.lhs configure
runghc Setup.lhs build
sudo runghc Setup.lhs install

If you want to build the example, in order to check it’s all working you will need to download it from GitHub. Once you have it follow the instructions in the file itself as to how to build it.


An extensible test framework, with support for HUnit and QuickCheck tests through the test-framework-hunit and test-framework-quickcheck packages. The main benefit of using this framework is that you get a nice console based test runner with the following features:

  • Run tests in parallel but report results in a deterministic order (to aid diff-based analysis of test output)
  • Progress reporting for individual QuickCheck properties and for the whole suite being run
  • Filter the tests to be run using patterns specified on the command line
  • Hierarchical, colored display of test results
  • Reporting of test statistics (number of tests run, number failed etc.)
  • Extensibility: add your own test providers above and beyond those provided
  • Seed reporting upon a failed QuickCheck run, so you can reproduce the failure if necessary


You can view the most recent version of the testsuite online at GitHub, as described in the “Installing” section. The example is not currently available via Hackage.

There are two essential components to getting running with the test framework: setting up the tests to be run, and making the program run the tests in the provided console test runner.

You specify the tests to run in your code like so:

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck

tests = [
        testGroup "Sorting Group 1" [
                testProperty "sort1" prop_sort1,
                testProperty "sort2" prop_sort2,
                testProperty "sort3" prop_sort3
        testGroup "Sorting Group 2" [
                testProperty "sort4" prop_sort4,
                testProperty "sort5" prop_sort5,
                testProperty "sort6" prop_sort6,
                testCase "sort7" test_sort7,
                testCase "sort8" test_sort8

And set up the console runner by including this in your Main module:

main = defaultMain tests

Console Runner Manual

A description of the options available can be obtained by using the option --help on the command line.

The test-selection syntax for use with the -s command line option is based on that of shell globs or Git .gitignore files. Test patterns are treated as follows:

  • An optional prefix ! which negates the pattern
  • If the pattern ends with a slash, it is removed for the purpose of the following description, but it would only find a match with a test group. In other words, foo/ will match a group called foo and any tests underneath it, but will not match a regular test foo.
  • If the pattern does not contain a slash /, the framework checks for a match against any single component of the path
  • Otherwise, the pattern is treated as a glob, where the wildcard * matches anything within a single path component (i.e. foo but not foo/bar), two wildcards ** matches anything (i.e. foo and foo/bar) and anything else matches exactly that text in the path (i.e. foo would only match a component of the test path called foo (or a substring of that form). For example, group/*1 matches group/test1 but not group/subgroup/test1, whereas both examples would be matched by group/**1. A leading slash matches the beginning of the test path; for example, /test* matches test1 but not group/test1.

A test will be run if it matches any of the patterns supplied with -s.