objective-c
The runner is how Fox executes properties, seeds data generation, and triggers shrinking. Although you can create and use the runner directly via FOXRunner
, it is more common to use wrappers -- such as FOXAssert
.
FOXAssert
takes a property assertion generator (only FOXForAll
for now). An exception is raised when the property fails to hold.
A more flexible FOXAssertWithOptions
can be used to provide parameters that is normally accepted by the runner.
The primary operation of Fox's runner is to create and execute tests. There are three parameters to configure Fox's test generation:
- The seed allows you to seed the random number generator Fox uses. This allows you to set the PRNG to help reproduce failures that Fox may have generated during a test run. Setting this to the default (
0
) will make Fox generate a seed based on the current time. - The number of tests indicates the number of tests Fox will generate for this particular property. More tests generated will more thoroughly cover the property at the cost of time. Setting this to the default (
0
) will make Fox run500
tests. - The maximum size indicates the maximum size factor Fox will use when generating tests. Generators use this size factor as a hint to produce data of the appropriate sizes. For example,
FOXInteger
will generate integers within the range of 0 tomaximumSize
andFOXArray
will generate arrays whose number of elements are in the range of 0 tomaximumSize
. Setting this to the default (0
) will make Fox run with amaximumSize
of200
. If you know this property's data generation can tolerate larger sizes, feel free to increase this. Large collection generation can be prohibitively expensive.
Please note that seed, number of tests, and maximum size should all be recorded to reproduce a failure that Fox may have generated.
By default, Fox will generate 500 tests per assertion with a maximum size of 200 and a random seed. By changing FOXAssert
to FOXAssertWithOptions
we can provide optional configuration by using the FOXOptions
:
FOXAssertWithOptions(FOXForAll(...), (FOXOptions){
seed=5, // default: time(NULL)
numberOfTests=1000, // default: 500
maximumSize=100, // default: 200
});
Values can be overridden using environment variables to globally change the defaults.
- Setting
FOX_SEED
can specify a specific seed to run for all properties. - Setting
FOX_NUM_TESTS
sets the number of tests to generate for each property. - Setting
FOX_MAX_SIZE
sets the maximum size factor Fox will use to when generating tests.
Fox provides a hook for custom pseudo-random number generation. This is also what generators receive as their first argument.
The runner uses FOXDeterministicRandom
which uses C++ random by default. This keeps randomization state isolated from any other potential system that uses a global PRNG. But you can implement the FOXRandom
protocol to support custom random schemes.
Another implementation Fox provides out of the box is FOXConstantRandom
, which always generates a constant number. This can be useful for testing generators with example-based tests.
The runner also provides a way to observe its operation via a reporter. Reporters are a delegate to the runner. They are invoked synchronously, so be careful about its performance impact on execution.
Delegates cannot influence the execution of the runner, but can provide useful user-facing output about the progress of Fox's execution.
By default, Fox runners do not have a reporter assigned to it. But Fox does provide a couple reporters:
FOXStandardReporter
reports in a rspec-like dot reporter.FOXDebugReporter
reports more information about the execution.
The default reporter can be changed by setting it from the instance FOXAssert
uses: [[FOXRunner assertInstance] setReporter:reporter]
.