New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-deterministic test collection order for deterministic fixtures makes xdist fail #920

Closed
aldanor opened this Issue Aug 6, 2015 · 9 comments

Comments

Projects
None yet
6 participants
@aldanor

aldanor commented Aug 6, 2015

Issues that may be related: #437, #594, #596, #669.

pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:

import pytest

@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
    return request.param

@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
    return request.param

def test_1(fixture1, fixture2):
    pass

Sometimes it works:

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling

foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]

========================= 4 passed in 0.60 seconds ==========================

And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1

+++ gw0

@@ -1,4 +1,4 @@

 foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
 foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================

@aldanor aldanor changed the title from Non-deterministic test collection order for determinstic fixtures makes xdist fail to Non-deterministic test collection order for deterministic fixtures makes xdist fail Aug 6, 2015

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 6, 2015

interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings

it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)

@aldanor

This comment has been minimized.

aldanor commented Aug 6, 2015

Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).

By the way, I wonder why should it care about the order at all if the sets are the same?

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 7, 2015

That goes a few years back, I recall its a workaround for duplicate test ids or something like that

@jstasiak

This comment has been minimized.

jstasiak commented May 5, 2016

@aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Jun 18, 2016

Hmm perhaps we can pass master's PYTHON_SEED value to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).

@hectorv

This comment has been minimized.

hectorv commented Jul 1, 2016

I'm seeing this while trying to move away from nosetests on a legacy test suite:

$ py.test -sxv -rw -n4 common/tests

this is my environment:

Python 2.7.11
pytest==2.9.2
pytest-xdist==1.14

Edit: to be clear, this works fine:

$ py.test -sxv -rw common/tests

@wence- wence- referenced this issue Jul 25, 2017

Merged

Fix nondeterminism in fixture collection order #2617

4 of 4 tasks complete

wence- added a commit to wence-/pytest that referenced this issue Jul 26, 2017

wence- added a commit to wence-/pytest that referenced this issue Jul 26, 2017

Fix nondeterminism in fixture collection order
fixtures.reorder_items is non-deterministic because it reorders based
on iteration over an (unordered) set.  Change the code to use an
OrderedDict instead so that we get deterministic behaviour, fixes pytest-dev#920.

wence- added a commit to wence-/pytest that referenced this issue Jul 26, 2017

wence- added a commit to wence-/pytest that referenced this issue Jul 26, 2017

Fix nondeterminism in fixture collection order
fixtures.reorder_items is non-deterministic because it reorders based
on iteration over an (unordered) set.  Change the code to use an
OrderedDict instead so that we get deterministic behaviour, fixes pytest-dev#920.

wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017

jenkins: Temporary workaround for pytest-dev/pytest#920
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.

wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017

jenkins: Temporary workaround for pytest-dev/pytest#920
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.

wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017

jenkins: Temporary workaround for pytest-dev/pytest#920
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.

wence- added a commit to wence-/pytest that referenced this issue Jul 26, 2017

wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017

jenkins: Temporary workaround for pytest-dev/pytest#920
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.

@nicoddemus nicoddemus closed this in a546a61 Aug 1, 2017

@renefritze

This comment has been minimized.

renefritze commented Aug 2, 2017

I think we still see this in our project with pytest 3.2 and xdist 1.18.2:
https://travis-ci.org/pymor/pymor/jobs/260181761#L744

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 2, 2017

@renemilk the issue you show is a different problem that comes from your own self-made parametrization system

all your tests that differ in order may be in a different place based on pythonhashseed
as such pytest cant know the order itself since the methods share the order

@renefritze

This comment has been minimized.

renefritze commented Aug 2, 2017

Ah, ok. I misunderstood then. Setting PYTHONHASHSEED=0 in env indeed fixes the ordering issue for me locally. Thanks.

@QuLogic QuLogic referenced this issue Aug 8, 2018

Open

Adding subplotpars to the plot stack #11796

0 of 6 tasks complete

webknjaz referenced this issue in aio-libs/aiohttp Oct 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment