8.4.0 test collection order is broken

[joniemi]

Expected behaviour

Using pytest 8.3.5, calling --collect-only lists test modules in command line input order and the contained test functions in definition order.

pytest src\tests\test_audio_levels.py src\tests\test_audio_io_page.py --collect-only

<Dir web-ui-test>
  <Dir src>
    <Package tests>
      <Module test_audio_levels.py>
        <Function test_card_header[chromium]>
        <Function test_nav_bar[chromium]>
        <Function test_six_graphs_exist[chromium]>
        <Function test_titles[chromium-Dante\xae Inputs]>
        <Function test_titles[chromium-Dante\xae Outputs]>
        <Function test_titles[chromium-MADI 1 Inputs]>
        <Function test_titles[chromium-MADI 1 Outputs]>
        <Function test_titles[chromium-MADI 2 Inputs]>
        <Function test_titles[chromium-MADI 2 Outputs]>
        <Function test_x_labels[chromium-Dante\xae Inputs]>
        <Function test_x_labels[chromium-Dante\xae Outputs]>
        <Function test_x_labels[chromium-MADI 1 Inputs]>
        <Function test_x_labels[chromium-MADI 1 Outputs]>
        <Function test_x_labels[chromium-MADI 2 Inputs]>
        <Function test_x_labels[chromium-MADI 2 Outputs]>
        <Function test_y_labels[chromium-Dante\xae Inputs]>
        <Function test_y_labels[chromium-Dante\xae Outputs]>
        <Function test_y_labels[chromium-MADI 1 Inputs]>
        <Function test_y_labels[chromium-MADI 1 Outputs]>
        <Function test_y_labels[chromium-MADI 2 Inputs]>
        <Function test_y_labels[chromium-MADI 2 Outputs]>
      <Module test_audio_io_page.py>
        <Function test_nav_bar[chromium]>
        <Function test_card_header[chromium]>
        <Function test_table_headers[chromium]>
        <Function test_channel_indices[chromium]>
        <Function test_number_of_rows[chromium]>
        <Function test_two_rows_per_port[chromium-Dante\xae]>
        <Function test_two_rows_per_port[chromium-MADI 1]>
        <Function test_two_rows_per_port[chromium-MADI 2]>
        <Function test_input_assignment[chromium]>
        <Function test_output_assignment[chromium]>

Actual behaviour

Same call with pytest 8.4.0 lists the test functions in a seemingly shuffled order jumping between the modules multiple times.

pytest src\tests\test_audio_levels.py src\tests\test_audio_io_page.py --collect-only

<Dir web-ui-test>
  <Dir src>
    <Package tests>
      <Module test_audio_levels.py>
        <Function test_card_header[chromium]>
        <Function test_nav_bar[chromium]>
        <Function test_six_graphs_exist[chromium]>
        <Function test_titles[chromium-Dante\xae Inputs]>
        <Function test_x_labels[chromium-Dante\xae Inputs]>
        <Function test_y_labels[chromium-Dante\xae Inputs]>
      <Module test_audio_io_page.py>
        <Function test_nav_bar[chromium]>
        <Function test_card_header[chromium]>
        <Function test_table_headers[chromium]>
        <Function test_channel_indices[chromium]>
        <Function test_number_of_rows[chromium]>
        <Function test_two_rows_per_port[chromium-Dante\xae]>
        <Function test_input_assignment[chromium]>
        <Function test_output_assignment[chromium]>
      <Module test_audio_levels.py>
        <Function test_titles[chromium-Dante\xae Outputs]>
        <Function test_x_labels[chromium-Dante\xae Outputs]>
        <Function test_y_labels[chromium-Dante\xae Outputs]>
      <Module test_audio_io_page.py>
        <Function test_two_rows_per_port[chromium-MADI 1]>
      <Module test_audio_levels.py>
        <Function test_titles[chromium-MADI 1 Inputs]>
        <Function test_x_labels[chromium-MADI 1 Inputs]>
        <Function test_y_labels[chromium-MADI 1 Inputs]>
      <Module test_audio_io_page.py>
        <Function test_two_rows_per_port[chromium-MADI 2]>
      <Module test_audio_levels.py>
        <Function test_titles[chromium-MADI 1 Outputs]>
        <Function test_x_labels[chromium-MADI 1 Outputs]>
        <Function test_y_labels[chromium-MADI 1 Outputs]>
        <Function test_titles[chromium-MADI 2 Inputs]>
        <Function test_x_labels[chromium-MADI 2 Inputs]>
        <Function test_y_labels[chromium-MADI 2 Inputs]>
        <Function test_titles[chromium-MADI 2 Outputs]>
        <Function test_x_labels[chromium-MADI 2 Outputs]>
        <Function test_y_labels[chromium-MADI 2 Outputs]>

Environment

Note: pytest-order plugin is installed but not used in the modules. Uninstalling pytest-order does not make a difference.

platform win32 -- Python 3.11.3, pytest-8.4.0, pluggy-1.5.0
plugins: base-url-2.1.0, html-4.1.1, metadata-3.1.1, order-1.3.0, playwright-0.7.0

pip list
Package            Version
------------------ ---------
artifacts-keyring  0.4.0
backports.tarfile  1.2.0
bcrypt             4.3.0
certifi            2024.8.30
cffi               1.17.1
charset-normalizer 3.4.0
colorama           0.4.6
cryptography       44.0.0
filetype           1.2.0
greenlet           3.1.1
idna               3.10
importlib_metadata 8.5.0
iniconfig          2.0.0
jaraco.classes     3.4.0
jaraco.context     6.0.1
jaraco.functools   4.1.0
Jinja2             3.1.4
keyring            25.5.0
MarkupSafe         3.0.2
more-itertools     10.5.0
msal               1.31.1
packaging          24.2
paramiko           3.5.1
pip                25.0.1
platformdirs       4.3.6
playwright         1.51.0
pluggy             1.5.0
pybatutils         0.4.0
pycparser          2.22
pycryptodome       3.21.0
pyee               12.0.0
Pygments           2.19.1
PyJWT              2.10.1
PyNaCl             1.5.0
pyserial           3.5
pytest             8.4.0
pytest-base-url    2.1.0
pytest-html        4.1.1
pytest-metadata    3.1.1
pytest-order       1.3.0
pytest-playwright  0.7.0
python-slugify     8.0.4
pywin32-ctypes     0.2.3
requests           2.32.3
setuptools         65.5.0
text-unidecode     1.3
typing_extensions  4.12.2
urllib3            2.2.3
zipp               3.21.0

[RonnyPfannschmidt]

Please provide details on the fixtures

There where some recent fixes in scope reordering

[joniemi]

Issue persists in 8.4.1 too.

[RonnyPfannschmidt]

The fix causes the new behavior

[joniemi]

setup_plan_8_3_5.txt
setup_plan_8_4_1.txt

I ran --setup-plan on the good and bad versions and attached as text files above.

The goto_page_and_login fixture is autouse and defined (copy-pasted) in both modules. Maybe that could throw something off? I'll try to construct a simpler example for reproduction when I have time.

Just a thought, if I remember correctly, pytest-playwright does some unorthodox things with its fixtures such as force-teardown/setup of session scope fixtures during a session when parametrizing with multiple browsers. This example used only one browser, however.

[joniemi]

I can reproduce the issue with a simple example. Seems like the issue is linked with pytest-playwright plugin.

evironment:

pip install playwright pytest pytest-playwright

test_a.py

import pytest

@pytest.mark.parametrize("param", ["apple", "orange", "pear"])
def test_1(page, param):
    assert True

test_b.py

import pytest

@pytest.mark.parametrize("param", ["dog", "cat", "mouse"])
def test_2(page, param):
    assert True

Calling pytest --collect-only results in

<Package tests>
  <Module test_a.py>
    <Function test_1[chromium-apple]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
  <Module test_a.py>
    <Function test_1[chromium-orange]>
  <Module test_b.py>
    <Function test_2[chromium-cat]>
  <Module test_a.py>
    <Function test_1[chromium-pear]>
  <Module test_b.py>
    <Function test_2[chromium-mouse]>

After removing the page fixture from one of the tests, the collection order is as expected:

<Package tests>
  <Module test_a.py>
    <Function test_1[apple]>
    <Function test_1[orange]>
    <Function test_1[pear]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
    <Function test_2[chromium-cat]>
    <Function test_2[chromium-mouse]>