"Correct" way to create external tests, as a pytest-plugin?

[DanCardin]

So the context here is that pytest-alembic is a plugin which tests your migrations. It comes bundled with some number of tests which can automatically run against your migrations.

You can optionally, from pytest_alembic.tests import test_x, test_y, etc and pytest will do the rest.

Alternatively, you can invoke pytest like pytest --test-alembic tests, and it automatically adds the tests to the set of collected tests.

Today this works by something along the lines of

# in-plugin code
def pytest_collection_modifyitems(session, config, items):
    tests = collect_tests(session, config)
    session.items.append(tests)

def collect_tests(...):
    return [AlembicItem.from_parent(...)]

class AlembicItem(pytest.Item):
    ...

    def runtest(self):
         ... some fixture fulfilling rigamarole
         self.test_fn(**params)

This has historically been all well and good, although it does appear to use some internal pytest behavior that i'm not certain is what you're supposed to be doing.

What I think I've recently found is that this strategy may not be playing well with other pytest plugins which use the hooking system. In particular pytest-asyncio makes use of the pytest_pycollect_makeitem hook to perform some of its behaviors, which is leading to some issues where these test don't trigger pytest_pycollect_makeitem which causes downstream fixture resolution problems (i think).

---

So ultimately my question is: Is there a more canonical way of getting the expected behavior i'm looking for with the --test-alembic flag? (i.e. where tests not defined within the tests/ dir (for a pytest tests/ invocation) are still found/run)

• Am I supposed to be registering some kind of collector somehow?
• or somehow emitting pytest_pycollect_makeitem myself?
• or somehow altering the pytest session on pytest_collectstart to pick up the tests automatically, without my having to write a pytest.Item subclass at all?

[RonnyPfannschmidt]

We currently have no concept of external add on test collections

This "kinda" is on my todo, but timing wise it will happen at earliest in 14 months

[DanCardin]

this already "works" as-is, just by adding the items to the list. Is there a way I could fake support by manually invoking any sort of hooks or something to get it working for at least the specific hook interaction i know is not working?

I realize there's potentially a lot of detail you might require to answer that statement but any of:

• adding the file containing these tests to the like...cli options such that they get pulled in like a normal python file at pytest foo.py
• or being able to add my Items early enough that pytest_pycollect_makeitem gets automatically invoked (if that's how it works)
• or getting a handle on a PyCollector such that i could manually call pytest-asyncio's pytest_pycollect_makeitem myself
• other options i dont know enough to suggest?

if nothing above are actionable for me now, i guess worst case, i can document the incompatibility with pytest-asyncio and suggest the workaround 🤷

[RonnyPfannschmidt]

Currently it "works"

It would be nice if we had a hook to provide collection roots with fixture /parameter setups

[nicoddemus]

Just from the top of my head:

Can you implement pytest_collect_file and return a custom collector, which then builds your items? Or do you need to inspect the items already collected in order to generate your own custom items?

[DanCardin]

there's no interaction with other items, but also the items i produce don't correspond to any of the files being collected normally. Given the API for pytest_collect_file, i had assume it would be called once per collected file (i guess, per plugin/location which defines one).

Are you suggesting that I define a pytest_collect_file, and return a collector (and that wont interfere with anything else defining the same hook?). And then any items i return from that hook will go through the typical set of other hooks like pytest_pycollect_makeitem?

[nicoddemus]

there's no interaction with other items, but also the items i produce don't correspond to any of the files being collected normally. Given the API for pytest_collect_file, i had assume it would be called once per collected file (i guess, per plugin/location which defines one).

Yes it will be invoked once for each file in the collection tree, for each hook implementation.

Are you suggesting that I define a pytest_collect_file, and return a collector (and that wont interfere with anything else defining the same hook?)

It won't interfere... a file may have multiple collectors:

pytest/src/_pytest/main.py

Lines 569 to 591 in 42776c0

	def _collectfile(
	self, fspath: Path, handle_dupes: bool = True
	) -> Sequence[nodes.Collector]:
	assert (
	fspath.is_file()
	), "{!r} is not a file (isdir={!r}, exists={!r}, islink={!r})".format(
	fspath, fspath.is_dir(), fspath.exists(), fspath.is_symlink()
	)
	ihook = self.gethookproxy(fspath)
	if not self.isinitpath(fspath):
	if ihook.pytest_ignore_collect(collection_path=fspath, config=self.config):
	return ()
	if handle_dupes:
	keepduplicates = self.config.getoption("keepduplicates")
	if not keepduplicates:
	duplicate_paths = self.config.pluginmanager._duplicatepaths
	if fspath in duplicate_paths:
	return ()
	else:
	duplicate_paths.add(fspath)
	return ihook.pytest_collect_file(file_path=fspath, parent=self) # type: ignore[no-any-return]

You will notice that _collectfile returns a Sequence[nodes.Collector].

Note that I agree pytest should have better mechanisms to add new items regardless of files, but I'm trying to find ways to work your requirements with the current hooks, even if not ideal.

[RonnyPfannschmidt]

Strict speaking it's at best a hack when a custom additional collection tree can't be provided

[nicoddemus]

Yep

[DanCardin]

I've arrived at something like this:

def pytest_sessionstart(session):
    if session.config.option.pytest_alembic_enabled:
        plugin = PytestAlembicPlugin(session.config)
        session.config.pluginmanager.register(plugin, 'pytest-alembic')

@dataclass(eq=False)
class PytestAlembicPlugin:
    config: config.Config

    def pytest_collect_file(self, file_path, path, parent):
        # todo: skip everything but the root module
        return TestCollector.from_parent(parent, fspath=None, path=file_path)

class TestCollector(pytest.Module):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._nodeid += "::pytest-alembic"

    def collect(self):
        return TestOptionResolver.collect_tests(self, self.parent.config, self.path)


# class PytestAlembicItem(pytest.Item):
class PytestAlembicItem(pytest.Function):
    pass

1. i have no idea if this resembles what you were suggesting. I've tried to vaguely copy the setup of pytest-black/pytest-cov where applicable, but nothing else i've seen is really doing anything like this.

2. pytest_collect_file appears to be being called twice at the beginning with the same tests/__init__.py file. if i return the collector on the first call, it produces no tests. if i do it on the 2nd call, it works as expected. I can't detect any Session state difference to know programmatically the one i'm "supposed" to use.

3. this doesn't actually fix the problem. pytest-asyncio's pytest_pycollect_makeitem is not automatically called (i dont know if i should expect that it would); and "calling it" through ihook causes a failure elsewhere.

   It seems that the fixtures depended upon by these tests are not being automatically resolved. this is something i was manually doing in runtest before.