Runs tests multiple times to expose flakiness.
- When enabled it will 'multiply' your tests so that they get run multiple times without having to restart pytest. This helps you find flakiness in your tests.
- You can limit your flake runs to a particular timeout value.
Install with setup.py:
python setup.py install
For best flake-finding combine with pytest-xdist:
pip install pytest-xdist
Enable plugin for tests:
This will run every test the default, 50, times. Every test is run independently and you can even use xdist to send tests to multiple processes.
To configure the number of runs:
pytest --flake-finder --flake-runs=runs
To find flakes in one test or a couple of tests you can use pytest's built in test selectiong.
Finding flakes in one test:
pytest -k test_maybe_flaky --flake-finder
When used with xdist the flake finder can expose many timing related flakes.
When using flake-finder as part of a CI run it might be useful to limit the amount of time it will run.
Running with timeout:
pytest --flake-finder --flake-max-minutes=minutes
Tests started after the timeout will be skipped.
Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.
If you encounter any problems, please file an issue along with a detailed description.