Brownie utilizes the hypothesis
framework to allow for property-based testing.
Much of the content in this section is based on the official hypothesis.works website. To learn more about property-based testing, you may wish to read this series of introductory articles or view the official Hypothesis documentation.
Property-based testing is a powerful tool for locating edge cases and discovering faulty assumptions within your code.
The core concept behind property-based testing is that rather than writing a test for a single scenario, you write tests that describe a range of scenarios and then let your computer explore the possibilities for you rather than having to hand-write every one yourself.
The basic process consists of:
- Choose a function within your smart contract that you wish to test.
- Specify a range of inputs for this function that should always yield the same result.
- Call the function with random data from your specification.
- Make an assertion about the result.
Using this technique, each test is run many times with different arbitrary data. If an example is found where the assertion fails, an attempt is made to find the simplest case possible that still causes the problem. This example is then stored in a database and repeated in each subsequent tests to ensure that once the issue is fixed, it stays fixed.
To begin writing property-based tests, import the following two methods:
from brownie.test import given, strategy
given
is a decorator that converts a test function that accepts arguments into a randomized test. This is a thin wrapper around :func:`hypothesis.given <hypothesis.given>`, the API is identical.strategy
is a method for creating :ref:`test strategies<hypothesis-strategies>` based on ABI types.
A test using Hypothesis consists of two parts: A function that looks like a normal pytest test with some additional arguments, and a @given
decorator that specifies how to those arguments are provided.
Here is a basic example, testing the transfer
function of an ERC20 token contract.
from brownie import accounts
from brownie.test import given, strategy
@given(value=strategy('uint256', max_value=10000))
def test_transfer_amount(token, value):
balance = token.balanceOf(accounts[0])
token.transfer(accounts[1], value, {'from': accounts[0]})
assert token.balanceOf(accounts[0]) == balance - value
When this test runs:
- The setup phase of all pytest fixtures are executed in their regular order.
- A snapshot of the current chain state is taken.
strategy
generates a random integer value and assigns it to theamount
keyword argument.- The test is executed.
- The chain is reverted to the snapshot taken in step 2.
- Steps 3-5 are repeated 50 times, or until the test fails.
- The teardown phase of all pytest fixtures are executed in their normal order.
It is possible to supply multiple strategies via @given
. In the following example, we add a to
argument using an address strategy.
from brownie import accounts
from brownie.test import given, strategy
@given(
to=strategy('address', exclude=accounts[0]),
value=strategy('uint256', max_value=10000),
)
def test_transfer_amount(token, to, value):
balance = token.balanceOf(accounts[0])
token.transfer(to, value, {'from': accounts[0]})
assert token.balanceOf(accounts[0]) == balance - value
assert token.balanceOf(to) == value
The key object in every test is a strategy. A strategy is a recipe for describing the sort of data you want to generate. Brownie provides a strategy
method that generates strategies for any given ABI type.
>>> from brownie.test import strategy
>>> strategy('uint8')
integers(min_value=0, max_value=255)
Each strategy object contains an example
method that you can call in the console to explore the types of data that will be generated.
>>> st = strategy('uint8')
>>> st.example()
243
>>> st.example()
77
strategy
accepts different keyword arguments depending on the ABI type.
The following strategies correspond to types within Solidity and Vyper.
Base strategy: :func:`hypothesis.strategies.sampled_from <hypothesis.strategies.sampled_from>`
address
strategies yield :func:`Account <brownie.network.account.Account>` objects from the :func:`Accounts <brownie.network.account.Accounts>` container.
Optional keyword arguments:
excludes
: An object, iterable or callable used to filter strategy results.
>>> strategy('address')
sampled_from(accounts)
>>> strategy('address').example()
<Account '0x33A4622B82D4c04a53e170c638B944ce27cffce3'>
Base strategy: :func:`hypothesis.strategies.booleans <hypothesis.strategies.booleans>`
bool
strategies yield True
or False
.
This strategy does not accept any keyword arguments.
>>> strategy('bool')
booleans()
>>> strategy('bool').example()
True
Base strategy: :func:`hypothesis.strategies.binary <hypothesis.strategies.binary>`
bytes
strategies yield byte strings.
All bytes
strategies accept the following keyword arguments:
excludes
: An object, iterable or callable used to filter strategy results.
For fixed length values (bytes1`...``bytes32
) the strategy always generates bytes of exactly the given length. For dynamic bytes arrays (bytes
), the minimum and maximum length may be specified using keyord arguments:
min_size
: Minimum length for each returned value. The default value is1
.max_size
: Maximum length for each returned value. The default value is64
.
>>> strategy('bytes32')
binary(min_size=32, max_size=32)
>>> strategy('bytes', max_size=16)
binary(min_size=1, max_size=16)
>>> strategy('bytes8').example()
b'\xb8\xd6\xaa\xcbR\x0f\xb88'
Base strategy: :func:`hypothesis.strategies.decimals <hypothesis.strategies.decimals>`
decimal
strategies yield :py:class:`decimal.Decimal <decimal.Decimal>` instances.
Optional keyword arguments:
min_value
: The maximum value to return. The default is-2**127
(the lower bound of Vyper'sdecimal
type). The given value is converted to :func:`Fixed <brownie.convert.datatypes.Fixed>`.max_value
: The maximum value to return. The default is2**127-1
(the upper bound of Vyper'sdecimal
type). The given value is converted to :func:`Fixed <brownie.convert.datatypes.Fixed>`.places
: The number of decimal points to include. The default value is10
.excludes
: An object, iterable or callable used to filter strategy results.
>>> strategy('decimal')
decimals(min_value=-170141183460469231731687303715884105728, max_value=170141183460469231731687303715884105727, places=10)
>>> strategy('decimal').example()
Decimal('44.8234019327')
Base strategy: :func:`hypothesis.strategies.integers <hypothesis.strategies.integers>`
int
and uint
strategies yield integer values.
Optional keyword arguments:
min_value
: The maximum value to return. The default is the lower bound for the given type. The given value is converted to :func:`Wei <brownie.convert.datatypes.Wei>`.max_value
: The maximum value to return. The default is the upper bound for the given type. The given value is converted to :func:`Wei <brownie.convert.datatypes.Wei>`.excludes
: An object, iterable or callable used to filter strategy results.
>>> strategy('uint32')
integers(min_value=0, max_value=4294967295)
>>> strategy('int8')
integers(min_value=-128, max_value=127)
>>> strategy('uint', min_value="1 ether", max_value="25 ether")
integers(min_value=1000000000000000000, max_value=25000000000000000000)
>>> strategy('uint').example()
156806085
Base strategy: :func:`hypothesis.strategies.text <hypothesis.strategies.text>`
string
strategies yield unicode text strings.
Optional keyword arguments:
min_size
: Minimum length for each returned value. The default value is0
.max_size
: Maximum length for each returned value. The default value is64
.excludes
: An object, iterable or callable used to filter strategy results.
>>> strategy('string')
text(max_size=64)
>>> strategy('string', min_size=12, max_size=23)
text(min_size=12, max_size=23)
>>> strategy('string').example()
'\x02\x14\x01\U0009b3c5'
Along with the core strategies, Brownie also offers strategies for generating array or tuple sequences.
Base strategy: :func:`hypothesis.strategies.lists <hypothesis.strategies.lists>`
Array strategies yield lists of strategies for the base array type. It is possible to generate arrays of both fixed and dynamic length, as well as multidimensional arrays.
Optional keyword arguments:
min_length
: The minimum number of items inside a dynamic array. The default value is1
.max_length
: The maximum number of items inside a dynamic array. The default value is8
.unique
: IfTrue
, each item in the list will be unique.
For multidimensional dynamic arrays, min_length
and max_length
may be given as a list where the length is equal to the number of dynamic dimensions.
You can also include keyword arguments for the base type of the array. They will be applied to every item within the generated list.
>>> strategy('uint32[]')
lists(elements=integers(min_value=0, max_value=4294967295), min_length=1, max_length=8)
>>> strategy('uint[3]', max_value=42)
lists(elements=integers(min_value=0, max_value=42), min_length=3, max_length=3)
>>> strategy('uint[3]', max_value=42).example()
[16, 23, 14]
Base strategy: :func:`hypothesis.strategies.tuples <hypothesis.strategies.tuples>`
Tuple strategies yield tuples of mixed strategies according to the given type string.
This strategy does not accept any keyword arguments.
>>> strategy('(int16,bool)')
tuples(integers(min_value=-32768, max_value=32767), booleans())
>>> strategy('(uint8,(bool,bytes4))')
tuples(integers(min_value=0, max_value=255), tuples(booleans(), binary(min_size=4, max_size=4)))
>>> strategy('(uint16,bool)').example()
(47628, False)
All of the strategies that Brownie provides are based on core strategies from the hypothesis.strategies
library. If you require something more specific or complex than Brownie offers, you can also directly use hypothesis strategies.
See the Hypothesis strategy documentation for more information on available strategies and how they can be customized.
Depending on the scope and complexity of your tests, it may be necessary to modify the default settings for how property-based tests are run.
The mechanism for doing this is the :py:class:`hypothesis.settings <hypothesis.settings>` object. You can set up a @given
based test to use this using a settings decorator:
from brownie.test import given
from hypothesis settings
@given(strategy('uint256'))
@settings(max_examples=500)
def test_this_thoroughly(x):
pass
You can also affect the settings permanently by adding a hypothesis
field to your project's brownie-config.yaml
file:
hypothesis:
max_examples: 500
See the :ref:`Configuration File<config>` documentation for more information.
Note
See the Hypothesis settings documentation for a complete list of available settings. This section only lists settings where the default value has been changed from the Hypothesis default.
.. py:attribute:: deadline The number of milliseconds that each individual example within a test is allowed to run. Tests that take longer than this time will be considered to have failed. Because Brownie test times can vary widely, this property has been disabled by default. default-value: ``None``
.. py:attribute:: max_examples The maximum number of times a test will be run before considering it to have passed. For tests involving many complex transactions you may wish to reduce this value. default-value: ``50``
.. py:attribute:: stateful_step_count The maximum number of rules to execute in a stateful program before ending the run and considering it to have passed. For more complex state machines you may wish to increase this value - however you should keep in mind that this can result in siginificantly longer execution times. default-value: ``10``