Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add package scoped fixtures #2283 #3389

Merged
merged 19 commits into from Jul 6, 2018

Conversation

Projects
None yet
4 participants
@jonozzz
Copy link

jonozzz commented Apr 11, 2018

In response to #2283, adding package scoped fixtures, that follow python-nose's implementation.

package/
  __init__.py
  test_foo.py

# __init__.py
def setup_module():
  print("Setting up the package")

def teardown_module():
  print("Tearing down the package")

In this case teardown_module() will be executed when all tests in test_foo.py are done.

turturica added some commits Apr 11, 2018

turturica turturica
turturica turturica
@coveralls

This comment has been minimized.

Copy link

coveralls commented Apr 11, 2018

Coverage Status

Coverage increased (+0.07%) to 92.68% when pulling 027d233 on jonozzz:features into 3f5e06e on pytest-dev:features.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 12, 2018

note that while adding that we should sort the relation to pytest_collect_directory
in addition to that - package scope cant possibly be added in that way - since different levels of package sort the same way for fixture setup/teardown

its also not in any way clear how things should cache and order for package nesting

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Apr 12, 2018

its also not in any way clear how things should cache and order for package nesting

I've yet to review the PR, but I say if a package-scoped fixture is requested by the tests in core.views.business.test_foo.py, then the fixture will be tear-down when all tests below core.views.business finishe, IOW, it is bound to the closest package where the fixture was requested.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 12, 2018

@nicoddemus but that means we still need "synthetic" scope for every level of package and match fixtures to that in order to correctly set it up and in order to correctly order tests

else you have trouble when doing fixture ordering and for parameterization in different levels

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Apr 12, 2018

@jonozzz does this PR only supports setup_*-style setup/teardown and not fixtures?

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Apr 12, 2018

You can definitely have something like this in a conftest.py at a (sub)package level and if any of the tests within that package uses that fixture it'll apply to the closest package that wraps those tests.

# root/conftest.py
@pytest.fixture(scope='package')
def fixture():
    print('SETUP')
    yield
    print('TEARDOWN')

root/
  __init__.py
  conftest.py <-- package fixture defined here
  sub1/
    __init__.py
    sub1_1/  <-- fixture setup/teardown happens here
      __init__.py
      test_foo.py <-- def test_1(fixture): pass

But if you'd set it to autouse=True, it'll apply at each nested package level (not really useful, but that's the expected behavior).

turturica added some commits Apr 16, 2018

turturica turturica
turturica turturica
Don't stop at the first package when looking up package-scoped fixtures.
Example:
package1.subpackage1
package1.subpackage2

package1's setup/teardown were executed again when exiting subpackage1 and entering subpackage2.
turturica turturica
Collapse all parent nested package fixtures when pointing to a sub-node.
Example:
Given this hierarchy: p1.s1.s2.s3
I want to run pytest p1/s1/s2/s3/foo.py

If there are any package fixtures defined at p1..s2 levels, they should also be executed.
turturica turturica
turturica turturica
Build a stack of all previous packages instead of just the one closes…
…t to the initial argument(s).

Address #3358 by caching nodes in a session dict.
@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Apr 22, 2018

While working on this I noticed another issue with setup_*/teardown_*-style fixtures. If you're mixing those with @pytest.fixture-style fixtures then they won't be executed in the correct order. For example a setup_class() will be executed before a scope='module' pytest fixture.

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Apr 23, 2018

if you're mixing those with @pytest.fixture-style fixtures then they won't be executed in the correct order.

You are right, we plan to address this in #3094.

@@ -1448,6 +1448,39 @@ def test_x(one):
reprec = testdir.inline_run("..")
reprec.assertoutcome(passed=2)

def test_package_xunit_fixture(self, testdir):

This comment has been minimized.

Copy link
@nicoddemus

nicoddemus Apr 23, 2018

Member

Could you also add another test with a package-scoped fixture? I also would like to see a more complex test with all fixture scopes involved.

This comment has been minimized.

Copy link
@jonozzz

jonozzz Apr 24, 2018

Author

Added one and changed a few of the existing fixture tests to use package fixtures.

turturica added some commits Apr 24, 2018

turturica turturica
Added another package-scoped fixture test.
Changed existing complex tests to use package fixtures.
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 25, 2018

currently its still absolutely unclear to me how this interacts with deeper nesting and how this interacts with parameterization

take as an example the following structure

- tests
  - conftest.py
      - def fix -> scope=package, parms=[1,2,3])
      - def fun(fix) -> scope=module, params=[a,b,c]
  - test_a.py
    - test_a(fun)
  - test_other
    - test_b.py(fun)
  - test_c.py
    - fix meh(fun) (scope=module, params =x,y)
    - test_c(meh)

when does what fixture run, how will tests be ordered

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Apr 26, 2018

It's very similar to "def fix -> (scope=session, params=[1,2,3])" except that the teardown part of this fixture (after yield) is executed once "exiting" the package not at the end of the session, as a scope=session would.

For your example this is how the tests are ordered:

dir8/sub1/test_a.py::test_a[1-a] PASSED
dir8/sub1/test_a.py::test_a[1-b] PASSED
dir8/sub1/test_a.py::test_a[1-c] PASSED
dir8/sub1/test_b.py::test_b[1-a] PASSED
dir8/sub1/test_b.py::test_b[1-b] PASSED
dir8/sub1/test_b.py::test_b[1-c] PASSED
dir8/sub1/test_c.py::test_c[1-x-a] PASSED
dir8/sub1/test_c.py::test_c[1-y-a] PASSED
dir8/sub1/test_c.py::test_c[1-y-b] PASSED
dir8/sub1/test_c.py::test_c[1-x-b] PASSED
dir8/sub1/test_c.py::test_c[1-y-c] PASSED
dir8/sub1/test_c.py::test_c[1-x-c] PASSED
dir8/sub1/test_a.py::test_a[2-a] PASSED
dir8/sub1/test_a.py::test_a[2-b] PASSED
dir8/sub1/test_a.py::test_a[2-c] PASSED
dir8/sub1/test_b.py::test_b[2-a] PASSED
dir8/sub1/test_b.py::test_b[2-b] PASSED
dir8/sub1/test_b.py::test_b[2-c] PASSED
dir8/sub1/test_c.py::test_c[2-x-a] PASSED
dir8/sub1/test_c.py::test_c[2-y-a] PASSED
dir8/sub1/test_c.py::test_c[2-y-b] PASSED
dir8/sub1/test_c.py::test_c[2-x-b] PASSED
dir8/sub1/test_c.py::test_c[2-y-c] PASSED
dir8/sub1/test_c.py::test_c[2-x-c] PASSED
dir8/sub1/test_a.py::test_a[3-a] PASSED
dir8/sub1/test_a.py::test_a[3-b] PASSED
dir8/sub1/test_a.py::test_a[3-c] PASSED
dir8/sub1/test_b.py::test_b[3-a] PASSED
dir8/sub1/test_b.py::test_b[3-b] PASSED
dir8/sub1/test_b.py::test_b[3-c] PASSED
dir8/sub1/test_c.py::test_c[3-x-a] PASSED
dir8/sub1/test_c.py::test_c[3-y-a] PASSED
dir8/sub1/test_c.py::test_c[3-y-b] PASSED
dir8/sub1/test_c.py::test_c[3-x-b] PASSED
dir8/sub1/test_c.py::test_c[3-y-c] PASSED
dir8/sub1/test_c.py::test_c[3-x-c] PASSED
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 26, 2018

thansk for working out the example, i believe for better understanding i need a printout of setup-only as well,

thanks for going the effort to make that listing, if possible i'd like to see that file committed as an example (im planning to use example files in test runs more) and would like to build on it

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Apr 26, 2018

Where do you want the examples to be checked in? I couldn't find an existing dir.
Also, any idea what caused that failure in "py36-xdist"?

Here's the output of the --setup-only:

dir8/sub1/test_a.py 
  SETUP    P fix[1]
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_a.py::test_a[1-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_a.py::test_a[1-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_a.py::test_a[1-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_b.py 
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_b.py::test_b[1-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_b.py::test_b[1-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_b.py::test_b[1-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_c.py 
    SETUP    M fun (fixtures used: fix)[a]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[1-x-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[1-y-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[1-y-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[1-x-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[1-y-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[1-x-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_a.py 
  TEARDOWN P fix[1]
  SETUP    P fix[2]
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_a.py::test_a[2-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_a.py::test_a[2-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_a.py::test_a[2-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_b.py 
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_b.py::test_b[2-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_b.py::test_b[2-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_b.py::test_b[2-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_c.py 
    SETUP    M fun (fixtures used: fix)[a]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[2-x-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[2-y-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[2-y-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[2-x-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[2-y-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[2-x-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_a.py 
  TEARDOWN P fix[2]
  SETUP    P fix[3]
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_a.py::test_a[3-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_a.py::test_a[3-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_a.py::test_a[3-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_b.py 
    SETUP    M fun (fixtures used: fix)[a]
        tmp/root/dir8/sub1/test_b.py::test_b[3-a] (fixtures used: fix, fun)
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
        tmp/root/dir8/sub1/test_b.py::test_b[3-b] (fixtures used: fix, fun)
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
        tmp/root/dir8/sub1/test_b.py::test_b[3-c] (fixtures used: fix, fun)
    TEARDOWN M fun[c]                                                                                                                    [  0%]
dir8/sub1/test_c.py 
    SETUP    M fun (fixtures used: fix)[a]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[3-x-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[3-y-a] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    TEARDOWN M fun[a]
    SETUP    M fun (fixtures used: fix)[b]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[3-y-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[3-x-b] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[b]
    SETUP    M fun (fixtures used: fix)[c]
    SETUP    M meh (fixtures used: fun)[y]
        tmp/root/dir8/sub1/test_c.py::test_c[3-y-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[y]
    SETUP    M meh (fixtures used: fun)[x]
        tmp/root/dir8/sub1/test_c.py::test_c[3-x-c] (fixtures used: fix, fun, meh)
    TEARDOWN M meh[x]
    TEARDOWN M fun[c]
  TEARDOWN P fix[3]
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 26, 2018

the failure on the xdist call happens because of a breaking behaviour change introduced by the package scope

the package objects import modules in the file tree even if there are no test modules below,
the tasks package inside of the pytest repo is a package with no tests which only works under certain conditions

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Apr 26, 2018

Right and this was introduced recently and it's unrelated to the one-liner that I just checked in.
For that particular set of tests, the invocation command changed from pytest -n3 -ra testing to pytest -n8 -ra ..

Now if we look at tox.ini, for py36-xdist section, the changedir=testing is missing, which is present for py27-xdist and that explains why the 2.7 suite is working while 3.6 does not.

[testenv:py36-xdist]
deps = {[testenv:py27-xdist]deps}
commands = {[testenv:py27-xdist]commands}

Breaking change:
888fcbc#diff-b91f3d5bd63fcd17221b267e851608e8L63

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Apr 27, 2018

@jonozzz while that assessment is entirely correct wrt introducing that failure as configuration,
the new behaviour is still a regression on that part

i propose trying to prevent a Package's collection from importing the package itself until a test module below it is actually imported

how hard do you think that will be ?

turturica turturica
@@ -116,6 +116,7 @@ def ihook(self):
Function = _CompatProperty("Function")
File = _CompatProperty("File")
Item = _CompatProperty("Item")
Package = _CompatProperty("Package")

This comment has been minimized.

Copy link
@RonnyPfannschmidt

RonnyPfannschmidt Apr 27, 2018

Member

i missed his earlier, but those properties are scheduled for removal, i'd like to avoid adding more

turturica turturica
@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented May 10, 2018

Hey @jonozzz,

I just wanted to let you know that we will not merge this until we get 3.6 out (which should be soon), it is not that we forgot or don't care about this PR. We want to avoid introducing a number of large changes in minor releases, to avoid potential breakage. For 3.6 we already have a large scale mark refactoring in the works.

Thanks for the patience! 👍

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented May 10, 2018

No worries and thanks for the update. The code could use some cleanup though, especially in main.py. I also think that I'm breaking the logical separation between the code in main.py and python.py. I'm open to suggestions though and this should be a good opportunity to make some changes before 3.6! :)

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jun 5, 2018

@RonnyPfannschmidt and @jonozzz, I think it is time for us to resume working on this. 👍

@jonozzz

This comment has been minimized.

Copy link
Author

jonozzz commented Jun 14, 2018

Sorry for the late reply- I was on vacation until today. Is there anything that needs to be added/changed, besides merging the changes? I just noticed there are some conflicts now.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 19, 2018

due to other development and introducing black, there turn up a lot of conflicts, we should resolve those

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 19, 2018

im not sure if i can taek a look personally merging t this week,
i believe this is mergable just fine (minus the conflicts we introduced by adding black)

i would like to introduce this marked as "experimental" since i do fear that we end up finding a strange edge-case once this has been available to a wider audience

i would like to note that the implementation is great work,
my fear stems from the limiting base pytest provides for this
(aka imho @jonozzz did a really great job within the limits of pytest)

as this touches nodes, scopes and introduces a new type of node, i fear that it may trigger a underlying issue
(the test reordering is really gritty and while the demonstration that @jonozzz posted really helped in making me trust the work, i can't quite shake off a rest of fear)

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jul 5, 2018

Took the liberty of fixing the conflicts and pushing the changes 👍

(you have some time on your hands when you are on the phone trying to cancel your internet service... 1h and counting now 😠).

@RonnyPfannschmidt RonnyPfannschmidt merged commit 1e94ac7 into pytest-dev:features Jul 6, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jul 6, 2018

win 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.