Hypothesis tries to have good defaults for its behaviour, but sometimes that's not enough and you need to tweak it.
:func:`@given <hypothesis.given>` invocation is as follows:
from hypothesis import given, settings @given(integers()) @settings(max_examples=500) def test_this_thoroughly(x): pass
This uses a :class:`~hypothesis.settings` object which causes the test to receive a much larger set of examples than normal.
This may be applied either before or after the given and the results are the same. The following is exactly equivalent:
from hypothesis import given, settings @settings(max_examples=500) @given(integers()) def test_this_thoroughly(x): pass
.. autoclass:: hypothesis.settings :members: :exclude-members: register_profile, get_profile, load_profile
Controlling what runs
Hypothesis divides tests into logically distinct phases:
- Running explicit examples :ref:`provided with the @example decorator <providing-explicit-examples>`.
- Rerunning a selection of previously failing examples to reproduce a previously seen error
- Generating new examples.
- Mutating examples for :ref:`targeted property-based testing <targeted-search>`.
- Attempting to shrink an example found in previous phases (other than phase 1 - explicit examples cannot be shrunk). This turns potentially large and complicated examples which may be hard to read into smaller and simpler ones.
- Attempting to explain the cause of the failure, by identifying suspicious lines of code (e.g. the earliest lines which are never run on passing inputs, and always run on failures). This relies on :func:`python:sys.settrace`, and is therefore automatically disabled on PyPy or if you are using :pypi:`coverage` or a debugger. If there are no clearly suspicious lines of code, :pep:`we refuse the temptation to guess <20>`.
The phases setting provides you with fine grained control over which of these run, with each phase corresponding to a value on the :class:`~hypothesis.Phase` enum:
Phase.explicitcontrols whether explicit examples are run.
Phase.reusecontrols whether previous examples will be reused.
Phase.generatecontrols whether new examples will be generated.
Phase.targetcontrols whether examples will be mutated for targeting.
Phase.shrinkcontrols whether examples will be shrunk.
Phase.explaincontrols whether Hypothesis attempts to explain test failures.
The phases argument accepts a collection with any subset of these. e.g.
settings(phases=[Phase.generate, Phase.shrink]) will generate new examples
and shrink them, but will not run explicit examples or reuse previous failures,
settings(phases=[Phase.explicit]) will only run the explicit
Seeing intermediate result
To see what's going on while Hypothesis runs your tests, you can turn up the verbosity setting.
>>> from hypothesis import find, settings, Verbosity >>> from hypothesis.strategies import lists, integers >>> @given(lists(integers())) ... @settings(verbosity=Verbosity.verbose) ... def f(x): ... assert not any(x) ... f() Trying example:  Falsifying example: [-1198601713, -67, 116, -29578] Shrunk example to [-1198601713] Shrunk example to [-1198601600] Shrunk example to [-1191228800] Shrunk example to [-8421504] Shrunk example to [-32896] Shrunk example to [-128] Shrunk example to  Shrunk example to  Shrunk example to  Shrunk example to  Shrunk example to  Shrunk example to  Shrunk example to  Shrunk example to  
The four levels are quiet, normal, verbose and debug. normal is the default, while in quiet mode Hypothesis will not print anything out, not even the final falsifying example. debug is basically verbose but a bit more so. You probably don't want it.
If you are using :pypi:`pytest`, you may also need to :doc:`disable output capturing for passing tests <pytest:capture>`.
Building settings objects
Settings can be created by calling :class:`~hypothesis.settings` with any of the available settings values. Any absent ones will be set to defaults:
>>> from hypothesis import settings >>> settings().max_examples 100 >>> settings(max_examples=10).max_examples 10
You can also pass a 'parent' settings object as the first argument, and any settings you do not specify as keyword arguments will be copied from the parent settings:
>>> parent = settings(max_examples=10) >>> child = settings(parent, deadline=None) >>> parent.max_examples == child.max_examples == 10 True >>> parent.deadline 200 >>> child.deadline is None True
At any given point in your program there is a current default settings,
settings.default. As well as being a settings object in its own
right, all newly created settings objects which are not explicitly based off
another settings are based off the default, so will inherit any values that are
not explicitly set from it.
You can change the defaults by using profiles.
Depending on your environment you may want different default settings. For example: during development you may want to lower the number of examples to speed up the tests. However, in a CI environment you may want more examples so you are more likely to find bugs.
Hypothesis allows you to define different settings profiles. These profiles can be loaded at any time.
.. automethod:: hypothesis.settings.register_profile
.. automethod:: hypothesis.settings.get_profile
.. automethod:: hypothesis.settings.load_profile
Loading a profile changes the default settings but will not change the behaviour of tests that explicitly change the settings.
>>> from hypothesis import settings >>> settings.register_profile("ci", max_examples=1000) >>> settings().max_examples 100 >>> settings.load_profile("ci") >>> settings().max_examples 1000
Instead of loading the profile and overriding the defaults you can retrieve profiles for specific tests.
>>> settings.get_profile("ci").max_examples 1000
Optionally, you may define the environment variable to load a profile for you. This is the suggested pattern for running your tests on CI. The code below should run in a conftest.py or any setup/initialization section of your test suite. If this variable is not defined the Hypothesis defined defaults will be loaded.
>>> import os >>> from hypothesis import settings, Verbosity >>> settings.register_profile("ci", max_examples=1000) >>> settings.register_profile("dev", max_examples=10) >>> settings.register_profile("debug", max_examples=10, verbosity=Verbosity.verbose) >>> settings.load_profile(os.getenv(u"HYPOTHESIS_PROFILE", "default"))
If you are using the hypothesis pytest plugin and your profiles are registered
by your conftest you can load one with the command line option
$ pytest tests --hypothesis-profile <profile-name>