State Machine Testing
This is in progress course material for a course on property-based state machine testing, using the Hedgehog package.
- You would like an automated minion to unleash hell on your application, breaking it in strange and fascinating ways.
What is State Machine Testing?
State Machine Testing extends property-based testing to provide a toolkit for building randomised tests of stateful systems. Like property-based testing, state machine tests use random generators to create test cases and shrink failing tests to minimal counter-examples. The difference with state machine testing is that the random input is now a sequence of commands to perform instead of arguments to pure functions.
How do we know that our stateful system is behaving itself? We build a model of the system being tested, and use it in a few ways:
Not all actions make sense at all times (e.g., what should happen if you try to log in when you're already logged-in?). When hedgehog generates a command sequence, we update the model being tested and use it to limit the actions we generate.
When we run tests, we perform commands both on the model and the system being tested, and check that their results agree.
The system we're testing is a vending machine for hot drinks, defined
src/CoffeeMachine.hs. You can select which drink you'd like,
insert or remove a mug, add milk or sugar, insert coins and dispense a
beverage. We'll be testing these features at different levels of the
The course itself is broken apart into several levels. Because this is
a course about testing, each level is a separate
test-suite in the
.cabal file, and its own directory.
Solutions are on the
solutions branch, one commit per level.
Running tests with
ghcid is a helpful tool that
helps to automate the 'edit-save-build' workflow.
ghcid can also
execute code or perform other actions after a successful build. In
this case we're going to setup
ghcid to run our tests whenever our
code is in a buildable state.
- Create a
dev.ghciscript in the root of this project to prepare our repl and load the required modules:
:set -isrc:levelNN :load levelNN/Main.hs
The first line indicates which folders
ghci will include when looking for any
of our code. We can provide multiple directories by providing a colon (
The second line loads the module that contains the function we want to execute
when the code is buildable. In this case it is
main :: IO () function that
runs our tests.
ghcidto use our
dev.ghciand also what command to run when everything is 'All good'. Additionally we will instruct
ghcidto ignore warnings:
$ ghcid -c 'ghci -ghci-script="dev.ghci"' --test=:main -W
ghci instead of
cabal new-repl so we can provide the
setup our repl environment. The
--test=:main is the repl command that will be
ghcid to ignore any warnings from compilation.
Running with stack
You will need to run a few additional commands when using stack:
# Initialise $ stack init # Replace `level01` with the level you are working on $ stack test :level01 # REPL, if you want it $ stack ghci :level01
- Setting Up
- First tests
- Terminology and
- Some simple commands
- Discussion of test feedback and interpreting errors
- More Commands
- Positive & Negative Testing
- Lensy Models