Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The sorting of parameterized tests broke down in pytest 8.0.0 #11976

Closed
ShurikMen opened this issue Feb 14, 2024 · 1 comment
Closed

The sorting of parameterized tests broke down in pytest 8.0.0 #11976

ShurikMen opened this issue Feb 14, 2024 · 1 comment

Comments

@ShurikMen
Copy link
Contributor

In release 8.0.0, the order of test execution has changed.
The logic of the new order is not clear.
For illustration simple parameterized tests

import pytest


@pytest.mark.parametrize('proto', ['serial', 'telnet', 'ssh'], scope='class')
@pytest.mark.parametrize('unit', [1, 2, 3], scope='class')
class TestA:
    def test_one(self, proto, unit):
        pass

    def test_two(self, proto, unit):
        pass

Order in pytest <=7.4.4

<Package tests>
  <Module test_asd.py>
    <Class TestA>
      <Function test_one[1-serial]>
      <Function test_two[1-serial]>
      <Function test_one[1-telnet]>
      <Function test_two[1-telnet]>
      <Function test_one[1-ssh]>
      <Function test_two[1-ssh]>
      <Function test_one[2-serial]>
      <Function test_two[2-serial]>
      <Function test_one[2-telnet]>
      <Function test_two[2-telnet]>
      <Function test_one[2-ssh]>
      <Function test_two[2-ssh]>
      <Function test_one[3-serial]>
      <Function test_two[3-serial]>
      <Function test_one[3-telnet]>
      <Function test_two[3-telnet]>
      <Function test_one[3-ssh]>
      <Function test_two[3-ssh]>

Order in pytest >=8.0.0

<Dir pytest>
  <Package tests>
    <Module test_asd.py>
      <Class TestA>
        <Function test_one[1-serial]>
        <Function test_two[1-serial]>
        <Function test_one[2-serial]>
        <Function test_two[2-serial]>
        <Function test_one[2-telnet]>
        <Function test_two[2-telnet]>
        <Function test_one[1-telnet]>
        <Function test_two[1-telnet]>
        <Function test_one[3-telnet]>
        <Function test_two[3-telnet]>
        <Function test_one[3-serial]>
        <Function test_two[3-serial]>
        <Function test_one[3-ssh]>
        <Function test_two[3-ssh]>
        <Function test_one[2-ssh]>
        <Function test_two[2-ssh]>
        <Function test_one[1-ssh]>
        <Function test_two[1-ssh]>

@ShurikMen ShurikMen changed the title The sorting of parameterized tests broke down in 8.0.0 The sorting of parameterized tests broke down in pytest 8.0.0 Feb 14, 2024
@bluetech
Copy link
Member

This bug is already fixed, and will be included in pytest 8.0.1 (due in a few days).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants