Have a way to control order of plugin loading

[ionelmc]

It appears that the order of pkg_resources.iter_entry_points is pretty arbitrary.

I would like pytest to order that so I can say pytest-cov must load before pytest-benchmark. Currently I cannot use pytest-cov to measure pytest-benchmark due to the ordering issue.

[RonnyPfannschmidt]

needs consideration in pluggy i think

we might want to add a own setuptools metadata writer to pluggy, to have more informative plugin specs

[ionelmc]

We could just sort the entrypoints before calling them, no?

Would there by any problem if I decide to rename my entrypoint?

[RonnyPfannschmidt]

in my oppinion odering by name or lexical order is flawed when dependencies come into play

[ionelmc]

Just for reference, my original issue was with the imports, I wanted pytest-cov to be imported before pytest-benchmark. I have found a workaround since (albeit a bit contrived), just to measure coverage ...

[RonnyPfannschmidt]

early plugin import is planned since a while, just never found a god way to implement

[The-Compiler]

I can see how this would be useful for pytest-cov, but it shouldn't block 3.0. Changing the milestone to 3.1 for now.

[dedsm]

I'm also a victim of this, I'm using pytest_env and pytest_django, my django settings read some variables from environment (that I want to set with pytest_env), but in some computers pytest_django loads before pytest_env, so tests run without the environment.

[nicoddemus]

Back to the topic: the idea is to just somehow "mark" a plugin so it tries to be loaded as early as possible? How one would mark such a plugin?

[dedsm]

I think the easiest way is making py.test load first the plugins passed with the -p command line option before the automatic loading using pkg_resources

[nicoddemus]

@dedsm thanks for the suggestion. I think it is a valid idea, but that does not cover (heh) pytest-cov though... it is a plugin which by definition should be loaded as early as possible, independently if users have passed it into the command line or not.

I'm wondering if people have something in mind already.

[ionelmc]

One idea is to introduce another alternative entrypoint name that supports ordering. The object (a module?) that a plugin exports though that entrypoint should have some sort of priority property/attribute. Old style plugins would have default priority at 100 and ordering is done in descending order (plugins with high priority loads first). How about that?

[ionelmc]

Actually scratch that, accessing the priority attribute would imply importing module and causing undue issues. Perhaps an entrypoint just for the priority number? Hmmmm

[nicoddemus]

Would we need to have priority numbers, or something like tryfirst (where we just pile every hook which is declared as tryfirst and hope that's enough)? Priority numbers would mean some coordination between plugin authors...

Not that I have a better idea, mind you. 😜

[The-Compiler]

The first thing which came to mind are pytest11_tryfirst and pytest11_trylast entrypoints 😆

As for that vs. a priority value, I've actually talked to Holger about that one for plugin hooks, and we agreed it's the much nicer (if less powerful) mechanism, because it requires no (hard to impossible) coordination between authors. As a counter-example, nose seems to use priority values, and apparently that didn't end up being a good idea 😉

[dedsm]

I like the tryfirst trylast idea, I think that would solve most of the cases.

for the more obscure ones, how about having a way to alter the order via the config file? a simple list that the developer can alter with a placeholder for the "rest"? something like

[pytest]
plugin_order =
plugin_a
plugin_z
all

that coupled with debug information about the order of loading would be enough for the rest of the cases.

[RonnyPfannschmidt]

i propose the concept of a pluginspec

instead of having just random modules without order, each plugin should have a spec object telling its dependencies, and the order for consideration for required/optional dependencies

tryfirst/last is imho pretty much a horrific mess

[dedsm]

the problem is none of these cases are because of lack of dependencies, the django plugin should never depend on the env plugin, same as in the case of the coverage plugin

[RonnyPfannschmidt]

so what will you do if both plugins become tryfirst?

tryfirst/last do not at all solve the problem, they just bolt a non-solution on top

what would help is a way to specifiy dependencies in plugins and out of them
if your pytest,ini tells the pluginmanager, oh, in my case django needs env, you specify a topology

also we need a topology anyway - for example py.tests internal plugins can never be loaded via setuptools, and tryfirst/last couldnt fix that

[dedsm]

that's why I agree on having a really simple ordering builtin, but ultimately the control should reside in the developer because as far as I can see this problem is not common and should be treated individually

[RonnyPfannschmidt]

for me the problem is very common and currently solved by not using setuptools, but relying purely on ordered pytest-plugins specification

[dedsm]

I agree on setuptools not being a solution, but how and who should decide the topology of plugins that don't have anything to do with each other? it's naiveness thinking that every plugin will specify that relationship with all the rest of the plugins, it's not maintainable

[RonnyPfannschmidt]

there is need for 2 mechanisms

a) plugin authors spelling dependencies they do know
b) plugin users spelling dependencies plugin authors cant know

[The-Compiler]

So how would pytest-cov declare that it should be loaded as early as possible?

[RonnyPfannschmidt]

i propose a priority value for an "absolute order" and before/after listing for putting topology on top of that

also a pytest.ini entry for adding extra topology "hints"

[virogenesis]

How about a dependency order setting in a plugin making it load after a certain plugin in case it's defined.

That would solve the problem I am having.

depends_on = [...]
Making it load after the ones defined in the list.

[RonnyPfannschmidt]

@vbarbaresi thats basically the starting point of the topology i was talking about before

[nickwilliams-eventbrite]

I would just like to throw in another user story here. We have a project using pytest-cov and pytest-django. On one machine, pytest-cov loads first, and all of our __init__.pys and settings files are marked as covered, resulting in coverage of about 90%. On another machine, pytest-django loads first, and all of our __init__.pys and settings files are marked as uncovered, resulting in a coverage of only 74%. This is a HUGE difference and makes it hard to set fail-if-coverage-under thresholds and be able to rely on them.

I really do like this suggestion I saw above:

[pytest]
plugin_order =
    plugin_a
    plugin_z
    all

That seems the most elegant to me, IMO.

[nickwilliams-eventbrite]

Related (and perhaps I should report an actual bug on this one):

If I look at the help with pytest --help, it says this:

  -p name               early-load given plugin (multi-allowed). To avoid
                        loading of plugins, use the `no:` prefix, e.g.
                        `no:doctest`.

Using that and a suggestion above, hoping I could early-load the coverage plugin, I tried that out:

$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 ...........

Using -p no:pytest_cov results in the same behavior. So It appears that -p name does the exact same thing as -p no:name, which is contrary to the documentation. You can't early-load plugins.

[RonnyPfannschmidt]

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

[nickwilliams-eventbrite]

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

That's incredible confusing, and also not documented. Also, if that's the case, I can't figure out what I'm supposed to use to make pytest-cov load early. Things I've tried:

# disables plugin
$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 --cov=.......

# does not recognize it
$ pytest -p pytest-cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'pytest-cov'

# does not recognize it
$ pytest -p cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'cov'

# finally no error, but does not load it early
$ pytest -p coverage
================= test session starts =================
platform linux -- Python 3.7.1, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
Django settings: xxxx.settings.dev_local (from environment variable)
rootdir: /srv/xxxx, inifile: setup.cfg
plugins: django-3.4.6, cov-2.6.1 .......
...
xxxx/settings/__init__.py                           36     36      0      0     0.0%   1-135
xxxx/settings/dev_local.py                          33     33      6      0     0.0%   1-125
xxxx/settings/jenkins_test.py                       24     24      4      0     0.0%   1-131
xxxx/settings/production_docker.py                  23     23      9      0     0.0%   1-125
...

# gives an error about its already being loaded
$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 11, in <module>
    sys.exit(main())
...
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140434823318496': <_pytest.config.PytestPluginManager object at 0x7fb987c20be0>, 'pytestconfig': <_pytest.config.Config object at 0x7fb98498c5c0> ......

Is -p just broken?

[RonnyPfannschmidt]

its simply not in sync with the setuptools plugin loading mechanism which registers plugins at a different name than their import name while -p works in terms of import names

so in a sense - -p is broken

[nickwilliams-eventbrite]

Want me to file a new bug about it?

[ionelmc]

Ok so I've tried patching iter_entry_points and I realized that from pytest-cov's perspective fixing the order wouldn't really help because pytest loads all the plugins first and only later it starts calling hooks. IOW I don't have a hook to initialize coverage tracer before other plugins get imported.

Well, nothing short of doing it at import time which is way too tricky (inspecting the call stack? ugh ...).

[nicoddemus]

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

[Colin-b]

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

Hi @nicoddemus

I tried that but it does not solve the issue.

The command launched by travis is "pytest --cov", the configuration loaded is the exact one you advise.

The "pytest11" entrypoint is the root of the module leading to no coverage at all.

What's confusing here is that between your comment and the documentation, the content of the file is not the same. Also this page is a bit unclear, what specific action is suppose to start pytest-cov engine manually?

Do you have any explanation, documentation, tips to share as to what are the exact options to use in order to make sure that pytest-cov can be use to check coverage of a pytest plugin (containing a "pytest11" entrypoint) ?

Thanks again

[gaborbernat]

I've solved this by using coverage natively instead through pytest-cov, see https://github.com/pytest-dev/pytest-print/blob/eb776107d83a94e90edeb7381d679251ae45bcf0/tox.ini#L27-L33 👍

[tmax22]

2023 and still no way to do so? i'm trying to write plugin to a pytest-plugin. and the only way to do so is to make sure that my plugin is loaded before the other plugin

[RonnyPfannschmidt]

So far nobody came up with a good solution in the upstream project pluggy

It's been on my radar before I had kids, now not so much