xdist: Different tests were collected

[mhils]

Steps to reproduce the problem:

1. Create some test cases that look as follows:

   def _test(a):
       def t(self):
           assert a == a
       return t
   
   class TestFoo(object):
       test_x = _test("a")
       test_y = _test("b")

2. Run py.test -n 4

What is the expected behavior?

The tests should pass.

What went wrong?

λ py.test test_foo.py -n 4
============================= test session starts =============================
platform win32 -- Python 3.5.0, pytest-2.8.0, py-1.4.30, pluggy-0.3.1
rootdir: C:\Users\user\git\netlib, inifile:
plugins: cov-2.1.0, xdist-1.13.1
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
collecting 0 items / 3 errors
=================================== ERRORS ====================================
____________________________ ERROR collecting gw0 _____________________________
Different tests were collected between gw2 and gw0. The difference is:
--- gw2

+++ gw0

@@ -1,2 +1,2 @@

+test_foo.py::TestFoo::()::test_y
 test_foo.py::TestFoo::()::test_x
-test_foo.py::TestFoo::()::test_y
=========================== 3 error in 1.17 seconds ===========================

This is the minimal test case I could break it down to. Looks like there's an issue if we're not defing functions. Any help would be appreciated! 😃

[RonnyPfannschmidt]

the easy way could be, to declare the python hash seed, so the unordered parts of the collection have a high chance for having the same order

the problem is, that when you use test items that have a indistinguishable source code location, order is left to mappings, and mapping order is undefined

alternatively you could use a type whose items are put into a ordereddict

[AbdealiLoKo]

As @RonnyPfannschmidt mentioned about the hash seed,
This gets solved for me if I use the pytest-env plugin to do:

env =
    PYTHONHASHSEED=0

[RonnyPfannschmidt]

Im closing this one AS invalid, please reopened if a New Detail comes up

[trtg]

This is still a problem for me, I have not been able to find a single project where I can successfully use more than 1 core without encountering a failure. I looked through the list of projects that use py.test and tried cloning a bunch of them and running py.test -n 2. For example tiddlyweb from here: git@github.com:tiddlyweb/tiddlyweb.git

if that project is tested with py.test -n 2 test/
3 failures appear:
______________________________________________________ ERROR collecting gw0 _______________________________________________________
Different tests were collected between gw1 and gw0. The difference is:

This happened on the pypy project and on the project I'm currently working on that I want to test faster (the test suite is currently 24 minutes long). So far xdist has been unusable for me, since specifying more than one core guarantees failure. Are there more special considerations to keep in mind if I want to parallelize with py.test -n 2 ?

[RonnyPfannschmidt]

@trtg the project you linked uses a demonstrably broken approach to generate tests - thats pretty much its own fault

[trtg]

Did the exploration of changing how tests are distributed discussed here:
#175
ever move forward?

Pierre-yves Rofes changes that allow assigning all the tests in one class
to the same core would be perfect for my use case.

On Sep 13, 2016 11:57 PM, "Ronny Pfannschmidt" notifications@github.com
wrote:

@trtg https://github.com/trtg the project you linked uses a demonstrably
broken approach to generate tests - thats pretty much its own fault

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1075 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQk-BiY2We9X391FZo_YGY9CZcclzTJks5qp5rlgaJpZM4GEYrm
.

[RonnyPfannschmidt]

@trtg the discussionadvanced in the xdist issue tracker, but no implementation did take place

[nicoddemus]

For the record, the discussion took place in pytest-dev/pytest-xdist#18.

[rawrgulmuffins]

@RonnyPfannschmidt My company ran into a similar issue today that I think is different. I wanted to check my understanding before opening a new issue.

-    filter_time = datetime.strftime(datetime.utcnow(), WINDOW_END_DATE_FORMAT)
+    date = datetime(2018, 10, 9)
+    filter_time = datetime.strftime(date, WINDOW_END_DATE_FORMAT)

 @pytest.mark.parametrize('test_name,'
                           # several sets of tuples to parameterize on
                           ('test_window_end_date', filter_time, None),
                           # More sets of tuples that are parameterized on
                           )

Which lead to interesting test failures because of slight variations in the datetime.

Different tests were collected between gw6 and gw5. The difference is:
--- gw6

+++ gw5

@@ -656,10 +656,10 @@

 apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[stage_4_filter-None-4]
 apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[stage_5_filter-None-5]
 apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[stage_without_findings-None-100]
-apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[test_window_end_date-2018-10-09T23:03:03.770890-None]
+apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[test_window_end_date-2018-10-09T23:03:03.637042-None]
 apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[test_window_end_no_findings-1901-01-01T00:00:00.000000-None]
 apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[no_filters-None-None]
-apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[all_filters-2018-10-09T23:03:03.770890-4]
+apt/presentation/results_backend_unit_test.py::test_get_findings_latest_state[all_filters-2018-10-09T23:03:03.637042-4]
 apt/test_support/packaging_unit_test.py::test_lambda
 apt/test_support/packaging_unit_test.py::test_function
 apt/test_support/packaging_unit_test.py::test_callable

Is this behavior covered by this issue?

If it is would you be willing to expand on or link us to what good test generation looks like in this case?

[RonnyPfannschmidt]

I believe docs on that need to be written

Basically its an issue when your tests depend on dynamic values

[RonnyPfannschmidt]

In your specific case it may be advisable to use "test-name" as I'd instead of part of the parameter set

[rawrgulmuffins]

@RonnyPfannschmidt Which docs do you need. I'll go write a PR for em.

[RonnyPfannschmidt]

@rawrgulmuffins thanks for signing up for that, i am currently not deeply aware of the state of the docs on what basically amounts to best practices wrt parameterize and declaring parameter-sets at all,

i believe @pfctdayelise and @nicoddemus have a more detailed awareness and might be able to answer that question directly

we might want to open up a followup issue to sort those details out as its not really fitting the issue at hand anymore

[nicoddemus]

@RonnyPfannschmidt definitely, created #4101 which explains the details. 👍

Thanks @rawrgulmuffins for volunteering. 🙇

[tokkoro]

In case someone is still having this issue. For me setting the PYTHONHASHSEED resulted one of my dict iterator still producing nondeterministic results, so I simply sorted the parametrized array after creating it.

From

@pytest.mark.parametrize(
    'things',
    [thing for thing in get_things().values()],
)

To

@pytest.mark.parametrize(
    'things',
    sorted([thing for thing in get_things().values()], key=lambda item: item.type),
)

the things have unique type string property that was used to get deterministic order.