The let%expect_test equivalent for quickcheck - a syntax extension for writing
quickcheck tests. For example:
open! Core
let%quick_test "int comparison is transitive" =
fun (a : int) (b : int) (c : int) -> assert (if a > b && b > c then a > c else true)
;;The following construct is now a valid structure item:
let%quick_test "name" = fun (<param> : <type>) ... (<param> : <type>) -> <body>- We may write
_instead of"name"for anonymous tests - There must be at least one provided parameter
- Types must be provided for all parameters
- Types must implement
sexp_of_t - The
<body>tests some property of the supplied parameters. - The
<body>returns auniton a success, and raises on a failure. Ifppx_quick_test_asyncis opened, it returns aunit Deferred.t.
Add ppx_quick_test as a preprocessor to your build file. For example
(library ((name your_lib) (preprocess (pps (ppx_quick_test)))))NOTE: If you are using dune outside of Jane Street, the syntax is:
(library (name your_lib) (preprocess (pps ppx_quick_test)))If quick test finds a failing input, ppx_quick_test can "remember" failing inputs with
[@remember_failures]. Using this requires your type to additionally have t_of_sexp.
This feature is opt-in as it might not make sense to remember every test case that has
failed, specially the failures that arise as you are writing your generator. After you've
finished writing your generator, and would like to remember future failures, you can add
[@remember_failures].
After the attribute is added, every time a failures occurs, a .corrected file will be
produced that adds the failing input next to the [@remember_failures] attribute. The
regression test is added after you accept the corrected test output. Details are
provided below, in the Attributes section.
Test scoped attributes are used to configure a quick test. They are attached to the test name. For example:
let%quick_test "name" [@first] [@second] =
fun (param1 : type1) (param2 : type2) -> <body>
let%quick_test _ [@first] [@second] =
fun (param1 : type1) (param2 : type2) -> <body>Available test scoped attributes are:
[@config <EXPR>]
- Sets the quickcheck configuration to use in the test
- EXPR Type =
Base_quickcheck.Test.Config.t - EXPR Default =
Base_quickcheck.Test.default_config
let%quick_test (_ [@config
{ Base_quickcheck.Test.default_config with
seed = Deterministic "seed"
}])
=
fun (a : int) (b : int) -> assert (a + b = b + a)
;;[@trials <INT>]
- Number of quickcheck trials to run.
- If provided, overrides the [test_count] value in the quickcheck config, even if a [@config] attribute is present.
- INT Type = positive integer literal
- INT Default = [config.test_count] value
let%quick_test (_ [@trials 500]) = fun (a : int) (b : int) -> assert (a + b = b + a)[@cr <EXPR>]
- Sets the CR level to print when a test is failing
- EXPR Type =
Expect_test_helpers_base.CR.t - EXPR Default =
Expect_test_helpers_base.CR.CR
[@examples <EXPR>]
- User provided example inputs to test your property
- EXPR Type =
(t1 * ...* tn) list - EXPR Default =
[]
let%quick_test (_ [@examples [ 1, 0; 0, 0 ]]) =
fun (a : int) (b : int) -> assert (a + b = b + a)
;;[@rembember_failures <SEXP 1> ... <SEXP N>]
- Machine provided example inputs to test your property
- Requires
t_of_sexpto convert the examples to the concrete types - Sexp examples are autogenerated when a test fails from a non-example input
- Corrected files are produced containing the generated sexp example
- SEXP Type =
stringliteral representing aSexp.trepresenting a(t1 * ... * tn)
let%quick_test _ [@remember_failures {|(1 0)|} {|(0 0)|}] =
fun (a : int) (b : int) -> assert (a + b = b + a)
;;[@tags <EXPR>]
- Tags to pass along to the generated expect test, useful to specify that a quick_test runs in "no-js", "fast-flambda", etc. See the list of supported tags in the ppx_inline_test docs
- EXPR Type =
(t1 * ...* tn) list - EXPR Default =
[]
let%quick_test (_ [@tags "no-js"]) = fun (a : int) (b : int) -> assert (a + b = b + a)Type scoped attributes used to configure a property of a specific input type. They are attached to the parameter's type. For example:
let%quick_test _ =
fun (param1 : type1 [@first] [@second])
(param2 : type2 [@third])
-> <body>Available type scoped attributes are:
[@generator <EXPR>]
- Use when you want an alternative/custom quickcheck generator or when you have no quickcheck generator avilable on your types
- EXPR Type =
your_type Base_quickcheck.Generator.t - EXPR Default =
[%quickcheck.generator: your_type]
let%quick_test _ = fun (a : int [@generator Int.gen_incl 0 10])
(b : int)
-> assert (a + b = b + a)[@shrinker <EXPR>]
- Use when you want to use an alternative/custom quickcheck shrinker
- EXPR Type =
your_type Base_quickcheck.Shrinker.t - EXPR Default =
[%quickcheck.shrinker: your_type] - NOTE: If you do not want a shrinker, you can default to
Base_quickcheck.Shrinker.atomic
let%quick_test _ = fun (a : int) (b : int) -> assert (a + b = b + a)The following examples shows the usage of all available attributes set to their default values.
let%quick_test (_ [@config Base_quickcheck.Test.default_config]
[@cr Expect_test_helpers_base.CR.CR]
[@hide_positions false]
[@generator [%quickcheck.generator: int * int * int]]
[@shrinker [%quickcheck.shrinker: int * int * int]]
[@examples []]
(* [@remember_failures] *))
=
fun (a : int) (b : int) (c : int) -> assert (if a < b && b < c then a < c else true)
;;If you'd like to write a quick_test that uses Async, you can use the library
ppx_quick_test_async by adding it to your jbuild's libraries field and opening it at
the top of your test file. This will make your let%quick_test need to return a unit Deferred.t instead of a unit.
Q: Why do the input types appear as a tuple in some contexts (e.g. quickcheck generation and example passing) even though we provide a function that takes in multiple parameters (instead of a single parameter of the tuple-typed)?
A: We translate the provided multi-parameter function to a single parameter function that takes in the tuple of input types. We perform this translation to make the function syntax easier to remember and read (no commas required between function arguments). Under the hood, we are always dealing with the tupled version of the input arguments.
Q: How do I resolve error: Error: unbound value <your_type>.t_of_sexp?
A: This is caused by having a (likely autogenerated) sexp example in [@remember_failures <SEXP>]
that can't be parsed because <your_type> doesn't have an t_of_sexp function.
Options:
- Add
t_of_sexpto<your_type>by using[@@deriving sexp]or specifying it manually - Move the failing example to
[@examples]by replacing[@remember_failures <SEXP>]with[@examples [<VALUE REPRESENTED BY SEXP>]]
Q : How do I resolve error: Error: unbound value <your_type>.quickcheck_generator?
A: This is caused by a failure to generate a quickcheck generator on your_type.
Options:
- Add
quickcheck_generatorto<your_type>by using[@@deriving quickcheck ~generator]or specifying it manually ([%quickcheck.generator : <type>]could be helpful) - Provide a generator to use using the
[@generator]attribute (see the Attributes section)
Q : How do I resolve error: Error: unbound value <your_type>.quickcheck_shrinker?
A: This is caused by a failure to generate a quickcheck shrinker on your_type.
Options:
- Add
quickcheck_shrinkerto<your_type>by using[@@deriving quickcheck ~shrinker]or specifying it manually ([%quickcheck.shrinker : <type>]could be helpful) - Provide shrinker to use using the
[@shrinker]attribute (see the Attributes section) - Disable shrinking by providing the atomic shrinker:
[@shrinker Base_quickcheck.Shrinker.atomic]
Q : Why don't you support syntax let%quick_test "name" (a : int) (b : int) = ... instead of
let%quick_test "name" = fun (a : int) (b : int) -> ...?
A: This is caused by a limitation in the underlying parser - the former syntax is unable to be parsed, even though it is a valid AST representation.
The ppx below:
let%quick_test "relation is transitive" (a : int) (b : int) (c : int) =
assert (if relation a b && relation b c then relation a c else true)will expand into something like:
let%expect_test "relation is transitive" =
Expect_test_helpers_base.quickcheck_m
[%here]
(module struct
type t = int * int * int [@@deriving quickcheck, sexp_of]
end)
(fun (a, b, c) ->
assert (if relation a b && relation b c then relation a c else true))Note: Rather than using quickcheck_m we use the a custom method in the runtime library with
similar functionality.
- Find all instances of
let%quick_tests in the tree. Extract them and spend time running them outside the context of a build step using additional randomized inputs (maybe integration AFL)?