integrate with coveragepy 5 who-tests-what feature: @pytest.mark.covers

[graingert]

Coveragepy 5 now supports coverage provenance in a "who-tests-what" feature. Currently this information is reported on but there's currently no easy way to provide assertions on the information

I'd like to be able to mark modules and classes with @pytest.mark.covers('path.to.some.sut', fail_under=...)

And have them a test suite fail if the coverage by the items contained in the node hierarchy are not sufficient to cover 'path.to.some.sut'

Design

Scope Grouping

The scope of the mark effects the grouping of the coverage assertion:

# testing/ham/spam/selenium.py
import pytest

from ham import spam
pytestmark = pytest.mark.covers(spam.__spec__.name)


def test_baz(browser): ...

@pytest.mark.covers(spam.eggs.__qualname__)
class TestEggs:
    def test_foo(self, browser): ...
    def test_bar(self, browser): ...

testing.ham.spam.selenium.test_baz, testing.ham.spam.selenium.TestEggs.test_foo and testing.ham.spam.selenium.TestEggs.test_bar should cover "ham.spam" completely on their own

testing.ham.spam.selenium.TestEggs.test_foo and testing.ham.spam.seleniumTestEggs.test_bar should cover ham.spam.eggs completely on their own - regardless of if testing.ham.spam.selenium.test_baz
happens to cover any of "ham.spam.eggs" or any other test in the suite happens to do so.

A new file

adding a new file, should not weaken any of the above coverage assertions:

# testing/ham/spam/unit.py
import pytest

from ham import spam
pytestmark = pytest.mark.covers(spam.__spec__.name)


def test_baz(db): ...

@pytest.mark.covers(spam.eggs.__qualname__)
class TestEggs:
    def test_foo(self, db): ...
    def test_bar(self, db): ...

In addition:
testing.ham.spam.unit.test_baz, testing.ham.spam.selenium.unit.test_foo and testing.ham.spam.unit.TestEggs.test_bar should cover "ham.spam" completely on their own

testing.ham.spam.unit.TestEggs.test_foo and testing.ham.spam.unit.test_bar should cover ham.spam.eggs completely on their own - regardless of if testing.ham.spam.unit.test_baz or any other test in the suite
happens to cover any of "ham.spam.eggs"

Grouping override

an groupe kwarg allows you to explicitly weaken the coverage assertion:

# testing/ham/eggs/selenium/marks.py
import pytest
from ham import eggs
covers_spam_together = pytest.mark.covers(eggs.__spec__.name, group=__spec__.name)

# testing/ham/eggs/selenium/homepage.py
from .marks import helpers

pytestmark = helpers.covers_spam_together

# testing/ham/eggs/selenium/cart.py
from .marks import helpers

pytestmark = helpers.covers_spam_together

Problems

I'm not particularly happy with my design for explicit grouping. Would sentinel objects be better?

UNIT = object()
SELENIUM = object()
covers_spam_unit = pytest.mark.covers(eggs.__spec__.name, group=UNIT)
covers_spam_selenium = pytest.mark.covers(eggs.__spec__.name, group=SELENIUM)

can mark instances themselves be a sentinel by default?

covers_spam_selenium = pytest.mark.covers(eggs.__spec__.name)
covers_spam_selenium = pytest.mark.covers(eggs.__spec__.name)

would it be better to disable scope-grouping entirely, and require a group kwarg?
this could result in spooky-action-at-a-distance in which adding testing.ham.spam.unit to a suite implicitly weakens testing.ham.spam.selenium