test.check is a tool for writing property-based tests. This differs from traditional unit-testing, where you write individual test-cases. With test.check you write universal quantifications, properties that should hold true for all input. For example, for all vectors, reversing the vector should preserve the count. Reversing it twice should equal the input. In this guide, we'll cover the thought process for coming up with properties, as well as the practice of writing the tests themselves.
First, let's start with an example, suppose we want to test a sort function. It's easy to come up with some trivial properties for our function, namely that the output should be in ascending order. We also might want to make sure that the count of the input is preserved. Our test might look like:
(require '[clojure.test.check :as tc])
(require '[clojure.test.check.generators :as gen])
(require '[clojure.test.check.properties :as prop])
(defn ascending?
"clojure.core/sorted? doesn't do what we might expect, so we write our
own function"
[coll]
(every? (fn [[a b]] (<= a b))
(partition 2 1 coll)))
(def property
(prop/for-all [v (gen/vector gen/int)]
(let [s (sort v)]
(and (= (count v) (count s))
(ascending? s)))))
;; test our property
(sc/quick-check 100 property)
;; => {:result true, :num-tests 100, :seed 1381894143051}What if we were to forget to actually sort our vector? The test will fail, and
then test.check will try and find 'smaller' inputs that still cause the test
to fail. For example, the function might originally fail with input:
[5 4 2 2 2], but test.check will shrink this down to [0 -1] (or [1 0]).
(def bad-property
(prop/for-all [v (gen/vector gen/int)]
(ascending? v)))
(sc/quick-check 100 bad-property)
;; => {:result false, :failing-size 7, :num-tests 8, :fail [[-2 4 -7 5 -2 7 -4]],
;; => :shrunk {:total-nodes-visited 19, :depth 8, :result false,
;; => :smallest [[0 -1]]}}This process of shrinking is done automatically, even for our more complex generators that we write ourselves.
In order to write our property, we'll use generators. A generator knows how to
generate random values for a specific type. The test.check.generators
namespace has many built-in generators, as well as combinators for creating
your own new generators. You can write sophisticated generators just by
combining the existing generators with the given combinators. As we write
generators, we can see them in practice with the sample function:
(require 'clojure.test.check.generators :as gen])
(gen/sample gen/int)
;; => (0 1 -1 0 -1 4 4 2 7 1)we can ask for more samples:
(gen/sample gen/int 20)
;; => (0 1 1 0 2 -4 0 5 -7 -8 4 5 3 11 -9 -4 6 -5 -3 0)or get a lazy-seq of values:
(take 1 (gen/sample-seq gen/int))
;; => 0You may notice that as you ask for more values, the 'size' of the generated values increases. As test.check generates more values, it increases the 'size' of the generated values. This allows tests to fail early, for simple values, and only increase the size as the test continues to pass.
Some generators take other generators as arguments. For example the vector
and list generator:
(gen/sample (gen/vector gen/nat))
;; => ([] [] [1] [1] [] [] [5 6 6 2 0 1] [3 7 5] [2 0 0 6 2 5 8] [9 1 9 3 8 3 5])
(gen/sample (gen/list gen/boolean))
;; => (() () (false) (false true false) (false true) (false true true true) (true) (false false true true) () (true))
(gen/sample (gen/map gen/keyword gen/boolean) 5)
;; => ({} {:z false} {:k true} {:v8Z false} {:9E false, :3uww false, :2s true})Sometimes we'll want to create heterogeneous collections. The tuple generator
allows us to to do this:
(gen/sample (gen/tuple gen/nat gen/boolean gen/ratio))
;; => ([0 false 0] [1 false 0] [0 false 2] [0 false -1/3] [1 true 2] [1 false 0] [2 false 3/5] [3 true -1] [3 true -5/3] [6 false 9/5])There are several generator combinators, we'll take a look at fmap,
such-that and bind.
fmap allows us to create a new generator by applying a function to the
values generated by another generator. Let's say we want to to create a set of
natural numbers. We can create a set by calling set on a vector. So let's
create a vector of natural numbers (using the nat generator), and then use
fmap to call set on the values:
(gen/sample (gen/fmap set (gen/vector gen/nat)))
;; => (#{} #{1} #{1} #{3} #{0 4} #{1 3 4 5} #{0 6} #{3 4 5 7} #{0 3 4 5 7} #{1 5})Imagine you have a record, that has a convenience creation function, foo. You
can create random foos by generated the types of the arguments to foo with
tuple, and then using (fmap foo (tuple ...)).
such-that allows us to create a generator that passes a predicate. Imagine we
wanted to generate non-empty lists, we can use such-that to filter out empty
lists:
(gen/sample (gen/such-that not-empty (gen/list gen/boolean)))
;; => ((true) (true) (false) (true false) (false) (true) (false false true true) (false) (true) (false))bind allows us to create a new generator based on the value of a previously
created generator. For example, say we wanted to generate a vector of keywords,
and then choose a random element from it, and return both the vector and the
random element. bind takes a generator, and a function that takes a value
from that generator, and creates a new generator.
(def keyword-vector (gen/such-that not-empty (gen/vector gen/keyword)))
(def vec-and-elem
(gen/bind keyword-vector
(fn [v] (gen/tuple (gen/elements v) (gen/return v)))))
(gen/sample vec-and-elem 4)
;; => ([:va [:va :b4]] [:Zu1 [:w :Zu1]] [:2 [:2]] [:27X [:27X :KW]])This allows us to build quite sophisticated generators.
Let's go through an example of generating random values of our own
defrecords. Let's create a simple user record:
(defrecord User [user-name user-id email active?])
;; recall that a helper function is automatically generated
;; for us
(->User "reiddraper" 15 "reid@example.com" true)
;; #user.User{:user-name "reiddraper",
;; :user-id 15,
;; :email "reid@example.com",
;; :active? true}We can use the ->User helper function to construct our user. First, let's
look at the generators we'll use for the arguments. For the user-name, we can
just use an alpha-numeric string, user IDs will be natural numbers, we'll
construct our own simple email generator, and we'll use booleans to denote
whether the user account is active. Let's write a simple email address
generator:
(def domain (gen/elements ["gmail.com" "hotmail.com" "computer.org"]))
(def email-gen
(gen/fmap (fn [[name domain-name]]
(str name "@" domain-name))
(gen/tuple (gen/not-empty gen/string-alpha-numeric) domain)))
(last (gen/sample email-gen))
;; => "CW6161Q6@hotmail.com"To put it all together, we'll use fmap to call our record constructor, and
tuple to create a vector of the arguments:
(def user-gen
(gen/fmap (partial apply ->User)
(gen/tuple (gen/not-empty gen/string-alpha-numeric)
gen/nat
email-gen
gen/boolean)))
(last (gen/sample user-gen))
;; => #user.User{:user-name "kWodcsE2",
;; :user-id 1,
;; :email "r2ed3VE@computer.org",
;; :active? true}Imagine we have some tree data-type that we want to generate: an (unbalanced) binary tree of natural numbers. Our tree representation uses vectors of count three for nodes ([value, left-tree, right-tree]), and raw integers for leaf nodes. Some example trees are
7
[5 [2 2] 1]
[10 nil 3]
[3 [1 [3 4 3] [1 5 nil]] [5 4 3]]Let's try and rephrase our goals a little more formally:
- A tree is either a natural number, or a natural number and two children.
- A child is either
nil, or a tree.
Here's a first translation of our requirements:
(def tree
(gen/one-of ;; choose either a natural number, or a node
[gen/nat
(gen/tuple gen/nat
(gen/one-of [(gen/return nil) tree])
(gen/one-of [(gen/return nil) tree]))]))And if we try and sample our generator:
(gen/sample tree)
;; => NullPointerException test.check.generators/gen-bind/fn--1244 (generators.clj:147)It turns out, we can't create recursive values (tree refers to itself in it's definition). Well, we can simply wrap it in a nullary function:
(defn tree
[]
(gen/one-of ;; choose either a natural number, or a node
[gen/nat
(gen/tuple gen/nat
(gen/one-of [(gen/return nil) (tree)])
(gen/one-of [(gen/return nil) (tree)]))]))And now if we try and sample:
(gen/sample (tree)) ;; we now have to 'call' tree to get our generator
;; => StackOverflowError test.check.generators/return (generators.clj:161)Progress. It turns out, we don't have a deterministic way to stop our
recursion. Our tree can just be created deeper and deeper. What we'd like is
some way to control the maximum depth of the tree. Fortunately, test.check
provides a function to help: sized. sized takes a function that takes an
integer size, and returns a generator based on this size. We can use this size
parameter to decide when to stop recurring. We'll say that when size is 0,
we'll only return leaf nodes. Further, when we recur, calling our own
generator, we'll decrease the size (half it, in this case). We can do this
with the resize function, which allows us to create a new generator with
size always bound to a particular value. Let's see how this looks:
(defn tree
[size]
(if (= size 0)
gen/nat
(let [new-size (quot size 2)
smaller-tree (gen/resize new-size (gen/sized tree))]
(gen/one-of ;; choose either a natural number, or a node
[gen/nat
(gen/tuple gen/nat
(gen/one-of [(gen/return nil) smaller-tree])
(gen/one-of [(gen/return nil) smaller-tree]))]))))Now let's see where we are:
(gen/sample (gen/sized tree))
;; => (0 [1 nil 0] [1 nil 0] [1 nil 1] 1 1 4 3 8 [7 nil [1 nil nil]])Excellent. We see ten trees printed. But we're seeing lots of one-element
trees. And if we do some more sampling, we'll see that we see lots of nils as
well. We can replace some of our calls to one-of with calls to frequency
to start controlling the likelihood of generating different bits of our tree.
Try playing with this yourself.