Allow fixtures to explicitly state which other fixtures they invalidate

[SalmonMode]

Regarding the discussion on #3905, I'm making a new issue to target with a PR, as I don't think this would be considered a solution to the original issue.

In order to keep the logic of a test suite's fixtures straightforward and to avoid possibly having to repeat operations unnecessarily, I propose we add a new optional argument (invalidates) to pytest.fixture and pytest.mark.parametrize that allows them to specify what fixture should be considered invalid once they run, and should be finalized when the invalidating fixture is finalized.

Here's a quick example (pulled from the discussion in #3905):

@pytest.fixture(scope="class", autouse=True)
def page(driver):
    """Goes to and returns a page object for the page with a form on it."""
    driver.get("https://mysite.com/some_form")
    return FormPage(driver)

class TestDefault():

    # tests for the default state of the page

class TestBlankSubmit():
    @pytest.fixture(scope="class", autouse=True)
    def submit_form(self, page):
        page.form.submit()

    # tests for the blank submit state of the page

class TestBadDataEntry():
    @pytest.fixture(
        scope="class",
        params=["bad_data1", "bad_data2"],
        autouse=True,
        invalidates="page",
    )
    def submit_bad_data(request, page):
        page.form.data = request.param
        page.form.submit()

    # tests for the bad data entry state of the page

Normally, in this example, TestBadDataEntry could probably not be trusted, because page would only happen once for the 1st param set, but it would not happen a 2nd time for the 2nd param set, which would mean that the 2nd param set would likely not be starting off from the same point as the 1st param set. In order to fix this, you'd have to define another page fixture just for that class and custom tailor it to make sure it happens once per param set, but this would mean writing duplicate code, and making things more complicated than they need to be.

But with the invalidates argument, each time submit_bad_data is finalized, it makes sure page is finalized as well, and when the fixtures of the 2nd param set request page, it will be executed again, resetting the state of the page.

I would also make it so that invalidates accepts one or more fixtures much like how similar arguments are provided, i.e. either a list/tuple of strings of fixture names (e.g. ["fixture_a", "fixture_b"]), or a comma separated string of fixture names (e.g. "fixture_a,fixture_b").

edit: added autouse to page fixture

[asottile]

my 2c:

• this seems like it's going to confuse people. I myself had to read the proposed documentation nearly five times to get what it was talking about
• this makes the (imo already too-magical) magical fixtures more magical
• this seems like a crutch for bad fixtures -- ideally fixtures should be functional and immutable
• at worst, this is a potential place for performance issues, a fixture needed to be rerun many times (opaquely as well, there's no indication to the end user that this is happening) due to "invalidates" (if I'm reading correctly it could also cause an invalidation cycle? or even need a solver to properly avoid that?)
• shouldn't these "bad" fixtures be rewritten as a series of functions? Or a fixture returning a function?

I didn't read the whole discussion in #3905 so it's possible I'm missing something

Given the example, here's a few potential rewrites without this feature:

use a factory

a factory function allows you to reset the state directly

@pytest.fixture(scope="class")
def page_factory(driver):
    """Goes to and returns a page object for the page with a form on it."""
    def fn():
       driver.get("https://mysite.com/some_form")
       return FormPage(driver)
    return fn

class TestDefault():

    # tests for the default state of the page

class TestBlankSubmit():
    @pytest.fixture(scope="class", autouse=True)
    def submit_form(self, page_factory):
        page_factory().form.submit()

    # tests for the blank submit state of the page

class TestBadDataEntry():
    @pytest.fixture(
        scope="class",
        params=["bad_data1", "bad_data2"],
        autouse=True,
    )
    def submit_bad_data(request, page_factory):
        page = page_factory()
        page.form.data = request.param
        page.form.submit()

don't use class scope

each function gets its own page with reset state

@pytest.fixture
def page_factory(driver):
    """Goes to and returns a page object for the page with a form on it."""
    driver.get("https://mysite.com/some_form")
    return FormPage(driver)

class TestDefault():

    # tests for the default state of the page

class TestBlankSubmit():
    @pytest.fixture(scope="class", autouse=True)
    def submit_form(self, page):
        page.form.submit()

    # tests for the blank submit state of the page

class TestBadDataEntry():
    @pytest.fixture(
        scope="class",
        params=["bad_data1", "bad_data2"],
        autouse=True,
    )
    def submit_bad_data(request, page):
        page.form.data = request.param
        page.form.submit()

non-fixture functional approach

(this is roughly the approach implemented at my previous employer -- it allowed construction to be composed / reordered / and shared between two testing libraries. I'm a bit partial to this approach since it is much more flexible -- and in my opinion the most readable).

def navigate_to_form_page_factory(driver):
    """Goes to and returns a page object for the page with a form on it."""
    driver.get("https://mysite.com/some_form")
    return FormPage(driver)

class TestDefault():

    # tests for the default state of the page

class TestBlankSubmit():
    @pytest.fixture(scope="class", autouse=True)
    def submit_form(self, driver):
        navigate_to_form(driver).form.submit()

    # tests for the blank submit state of the page

class TestBadDataEntry():
    @pytest.fixture(
        scope="class",
        params=["bad_data1", "bad_data2"],
        autouse=True,
    )
    def submit_bad_data(request, driver):
        page = navigate_to_form(driver)
        page.form.data = request.param
        page.form.submit()

[SalmonMode]

Hey, @asottile! Thanks for the 2c! 😁

this seems like it's going to confuse people. I myself had to read the proposed documentation nearly five times to get what it was talking about

I can definitely see where you're coming from here. I think if someone is familiar with how scope levels work, then understanding what this would do or how to use it shouldn't be too confusing for them. However, I don't intend for this to be something beginners would use, and it should be used only in exceptional circumstances.

I'm not sure if you're referring to the description I put here or on my PR for this (#4478), though. If you haven't seen the documentation I added in the PR, that might be more clear. Either way, if you have any ideas for how to make it more clear in either spot, I would love to hear them 😄.

It basically lets you define your own ad-hoc scope level using a given fixture that impacts targeted fixtures above it, without those fixtures needing to be defined any differently. Maybe having something like that included (maybe even with some diagrams) could help clarify?

this makes the (imo already too-magical) magical fixtures more magical

I agree that pytest's fixtures are pretty darned magical, but I think this just gives you some more explicit control over that magic in the rare circumstances that you'd want/need it.

this seems like a crutch for bad fixtures -- ideally fixtures should be functional and immutable

Normally, I would 100% agree. However, this came about as an issue with parametrization (and IMO, that's the only time I would ever use this feature, but I didn't want to rule out any other potential uses).

The way pytest currently works, is that fixtures that happen "before" parametrization aren't impacted by that parametrization even if they share the same scope. So in this case, submit_bad_data happens "after" page (because it's dependent on page), but, even though they're both using the class scope level and all the tests are impacted by the parametrization, page only happens once, while submit_bad_data and all the tests are run once for each set of parameters.

(oddly enough, this is not the case when you parametrize at a function level, because of how caching fixtures works)

The intent in that design was to parametrize an entire scenario, page included, but there's currently no clean/simple way to do this. Some solutions would be making page more complicated, adding more fixtures into the mix, and/or introducing redundant code, but I don't consider these options clean.

Adding this feature lets common fixtures be written without needing to worry about how parametrization further down the line (that may or may not happen) will factor in to things. It shifts the burden of handling this from both the common fixture(s) and the fixture parametrizing things, to just the parametrizing one.

at worst, this is a potential place for performance issues, a fixture needed to be rerun many times (opaquely as well, there's no indication to the end user that this is happening) due to "invalidates" (if I'm reading correctly it could also cause an invalidation cycle? or even need a solver to properly avoid that?)

Luckily, that's not an issue.

My implementation just adds the fixtures that should be invalidated to the "finalizers" list of the fixture that should be invalidating them. This ties their finalization processes together in the same way that it works for the last test in a scope. When the invalidating fixture then has to go through its finalization process, the code that's already in place handles everything so all fixtures (and those dependent on them) are finalized in the right order and aren't finalized more than once.

As for performance, it could negatively impact it if you were using invalidates in a way that altering the scope of the fixture you want to invalidate to a smaller one could also solve the issue (or you just don't know what invalidates is meant for). But that would be a similar issue to using a scope level that's too small for how you intend to use the fixture.

use a factory
non-fixture functional approach

The end goal with both of these solutions is to bring the act of going to the url "after" the parametrization occurs, which could work. But even if they didn't break away from any established patterns for defining the page fixtures, these would still add redundant code, and complicate the fixtures needed to setup for the tests for every class, even though the only one(s) that need it are the minority.

don't use class scope

Unfortunately, I test multiple things per scenario, so I have multiple test methods in each class. If I didn't use the class scope, it would go to the page, create a page object, and any other setup steps for every test method.

---

I originally only wanted this to deal with odd circumstances surrounding parametrization, but decided to let it be used with non-parametrized fixtures as couldn't be absolutely sure there was no circumstance it could be practically used with a non-parametrized fixture. But looking back, maybe it would be best to have it strongly recommended in the documentation that this is only used with parametrizing fixtures.

[Zac-HD]

FWIW I'm with @asottile; fixtures are already contributing more than their share of confusion to test suites everywhere. I'd really prefer to simplify them rather than adding more features, especially where there are decent alternatives already.

[SalmonMode]

Hi @Zac-HD, thanks for the tags and the feedback! 😄

I was about to ask what alternatives you had in mind when it occurred to me that indirect can actually be used to accomplish what I originally had in mind. Here's what I came up with:

@pytest.fixture(scope="class", autouse=True)
def page(driver):
    """Goes to and returns a page object for the page with a form on it."""
    driver.get("https://mysite.com/some_form")
    return FormPage(driver)

class TestDefault():

    # tests for the default state of the page

class TestBlankSubmit():
    @pytest.fixture(scope="class", autouse=True)
    def submit_form(self, page):
        page.form.submit()

    # tests for the blank submit state of the page

@pytest.mark.parametrize(
    ("data", "page"), 
    [["bad_data1", None], ["bad_data2", None]],
    scope="class",
    indirect=["page"],
)
class TestBadDataEntry():

    @pytest.fixture(scope="class", autouse=True)
    def submit_bad_data(self, data, page):
        page.form.data = data
        page.form.submit()

    # tests for the bad data entry state of the page

It works for my uses, but I'm not quite sold on it. I'd love to hear your thoughts on this approach.

[Zac-HD]

Sounds like we have a working solution, even without ✨magic✨, so I'm closing this issue 🙂