Saves previous test runs and allow re-execute previous pytest runs to reproduce crashes or flaky tests
This plugin helps to reproduce random or flaky behavior when running tests with xdist.
pytest-xdist executes tests
in a non-predictable order, making it hard to reproduce a behavior seen in CI locally because there's no convenient way
to track which test executed in which worker.
This plugin records the executed node ids by each worker in the directory given by
--replay=<file> can be used to re-run the tests from a previous run. For example:
$ pytest -n auto --replay-record-dir=build/tests/replay
This will generate files with the node id of each test executed by each worker, for example worker
gw1 will generate
.pytest-replay-gw1.txt with contents like this:
test_foo.py::test test_foo.py::test test_foo.py::test test_foo.py::test test_foo.py::test
If there is a crash or a flaky failure in the tests of the worker
gw1, one can take that file from the CI server and
execute the tests in the same order with:
$ pytest --replay=.pytest-replay-gw1.txt
Hopefully this will make it easier to reproduce the problem and fix it.
pytesthas its own cache, why use a different mechanism?
The internal cache saves its data using
json, which is not suitable in the advent of a crash because the file will not be readable.
Shouldn't the ability of selecting tests from a file be part of the
Sure, but let's try to use this a bit as a separate plugin before proposing its inclusion into the core.
$ pip install pytest-replay
Or with conda:
$ conda install -c conda-forge pytest-replay
Contributions are very welcome.
Tests can be run with tox if you are using a native Python installation.
To run tests with conda, first create a virtual environment and execute tests from there (conda with Python 3.5+ in the root environment):
$ python -m venv .env $ .env\scripts\activate $ pip install -e . pytest-xdist $ pytest tests
Follow these steps to make a new release:
- Create a new branch
- Open a PR;
- After it is green and approved, push a new tag in the format
Travis will deploy to PyPI automatically.
Afterwards, update the recipe in conda-forge/pytest-replay-feedstock.
Distributed under the terms of the MIT license.
If you encounter any problems, please file an issue along with a detailed description.