Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-deterministic test collection order for deterministic fixtures makes xdist fail #920

Closed
aldanor opened this issue Aug 6, 2015 · 12 comments
Closed

Comments

@aldanor
Copy link

@aldanor aldanor commented Aug 6, 2015

Issues that may be related: #437, #594, #596, #669.

pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:

import pytest

@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
    return request.param

@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
    return request.param

def test_1(fixture1, fixture2):
    pass

Sometimes it works:

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling

foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]

========================= 4 passed in 0.60 seconds ==========================

And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1

+++ gw0

@@ -1,4 +1,4 @@

 foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
 foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================
@aldanor aldanor changed the title Non-deterministic test collection order for determinstic fixtures makes xdist fail Non-deterministic test collection order for deterministic fixtures makes xdist fail Aug 6, 2015
@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Aug 6, 2015

interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings

it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)

@aldanor
Copy link
Author

@aldanor aldanor commented Aug 6, 2015

Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).

By the way, I wonder why should it care about the order at all if the sets are the same?

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Aug 7, 2015

That goes a few years back, I recall its a workaround for duplicate test ids or something like that

@jstasiak
Copy link

@jstasiak jstasiak commented May 5, 2016

@aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Jun 18, 2016

Hmm perhaps we can pass master's PYTHON_SEED value to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).

@hectorv
Copy link

@hectorv hectorv commented Jul 1, 2016

I'm seeing this while trying to move away from nosetests on a legacy test suite:

$ py.test -sxv -rw -n4 common/tests

this is my environment:

Python 2.7.11
pytest==2.9.2
pytest-xdist==1.14

Edit: to be clear, this works fine:

$ py.test -sxv -rw common/tests
@wence- wence- mentioned this issue Jul 25, 2017
4 of 4 tasks complete
wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.
wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.
wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.
wence- added a commit to firedrakeproject/firedrake that referenced this issue Jul 26, 2017
Fixture collection is not deterministic in the presence of hash
randomisation, so seed the hash so that we can run tests in parallel.
@nicoddemus nicoddemus closed this in a546a61 Aug 1, 2017
@renefritze
Copy link

@renefritze renefritze commented Aug 2, 2017

I think we still see this in our project with pytest 3.2 and xdist 1.18.2:
https://travis-ci.org/pymor/pymor/jobs/260181761#L744

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Aug 2, 2017

@renemilk the issue you show is a different problem that comes from your own self-made parametrization system

all your tests that differ in order may be in a different place based on pythonhashseed
as such pytest cant know the order itself since the methods share the order

@renefritze
Copy link

@renefritze renefritze commented Aug 2, 2017

Ah, ok. I misunderstood then. Setting PYTHONHASHSEED=0 in env indeed fixes the ordering issue for me locally. Thanks.

webknjaz referenced this issue in aio-libs/aiohttp Oct 21, 2018
@liiight
Copy link

@liiight liiight commented Jan 21, 2019

I think I may have the same issue. I dynamically generate parameterization based on a txt file from S3:

def parametrize_tenant_from_s3_file(metafunc):
    file_path = metafunc.config.getoption('--tenant-list-file')
    profile_name = metafunc.config.getoption('--aws-profile-name')
    aws_session = AWSSession(profile_name=profile_name)
    s3 = S3(aws_session.session)
    parsed_link = parse.urlparse(file_path)
    file_name = Path.cwd() / 'tenant_list.txt'
    s3.download_file(bucket=parsed_link.netloc, key=parsed_link.path.lstrip('/'), path=file_name)
    metafunc.parametrize('tenant_id_from_s3_file', file_name.read_text().splitlines())

Running this without -n works:

Launching pytest with arguments -m test_marker --collect-only --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
collecting ... collected 5096 items / 4550 deselected
<Package '/Users/orcarmi/PycharmProjects/shield'>
  <Module 'test_tsd_migration.py'>
    <Function 'test_tsd_customers_migration[tenant_0003a1ef4dfa4360bdcdc175c858e7c6]'>
    <Function 'test_tsd_customers_migration[tenant_001dececa0164f6ba44a1514c6369b42]'>
...

With -n it does not:

Launching pytest with arguments -m test_marker -n auto --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
[gw0] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw1] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw2] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw3] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw4] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw5] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw6] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw7] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw0] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw3] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw4] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw5] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw6] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw7] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0] / gw4 [0] / gw5 [0] / gw6 [0] / gw7 [0]

scheduling tests via LoadScheduling

 generated xml file: /Users/orcarmi/PycharmProjects/shield/reports/jreport.xml -
 generated html file: /Users/orcarmi/PycharmProjects/shield/reports/report.html 
========================= no tests ran in 6.89 seconds =========================

I tried setting PYTHONSEEDHASH to no avail.
Please advise, thanks in advance

EDIT: Realised I wasn't on latest xdist and pytest, upgrade both to latest, still same issue.

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Jan 21, 2019

@liiight based on what you show its a completely different issue please open new issues instead of recycling related looking but actually fixed issues

@liiight
Copy link

@liiight liiight commented Jan 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.