Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have a way to control order of plugin loading #935

Open
ionelmc opened this issue Aug 11, 2015 · 51 comments
Open

Have a way to control order of plugin loading #935

ionelmc opened this issue Aug 11, 2015 · 51 comments
Labels
type: enhancement new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature

Comments

@ionelmc
Copy link
Member

ionelmc commented Aug 11, 2015

It appears that the order of pkg_resources.iter_entry_points is pretty arbitrary.

I would like pytest to order that so I can say pytest-cov must load before pytest-benchmark. Currently I cannot use pytest-cov to measure pytest-benchmark due to the ordering issue.

@RonnyPfannschmidt RonnyPfannschmidt added type: enhancement new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature extplugin labels Aug 12, 2015
@RonnyPfannschmidt
Copy link
Member

needs consideration in pluggy i think

we might want to add a own setuptools metadata writer to pluggy, to have more informative plugin specs

@ionelmc
Copy link
Member Author

ionelmc commented Aug 12, 2015

We could just sort the entrypoints before calling them, no?

Would there by any problem if I decide to rename my entrypoint?

@RonnyPfannschmidt
Copy link
Member

in my oppinion odering by name or lexical order is flawed when dependencies come into play

@ionelmc
Copy link
Member Author

ionelmc commented Aug 12, 2015

Just for reference, my original issue was with the imports, I wanted pytest-cov to be imported before pytest-benchmark. I have found a workaround since (albeit a bit contrived), just to measure coverage ...

@RonnyPfannschmidt
Copy link
Member

early plugin import is planned since a while, just never found a god way to implement

@RonnyPfannschmidt RonnyPfannschmidt added this to the 3.0 milestone Sep 13, 2015
@The-Compiler
Copy link
Member

I can see how this would be useful for pytest-cov, but it shouldn't block 3.0. Changing the milestone to 3.1 for now.

@The-Compiler The-Compiler modified the milestones: 3.1.0, 3.0 Aug 5, 2016
@dedsm
Copy link

dedsm commented Sep 6, 2016

I'm also a victim of this, I'm using pytest_env and pytest_django, my django settings read some variables from environment (that I want to set with pytest_env), but in some computers pytest_django loads before pytest_env, so tests run without the environment.

@nicoddemus
Copy link
Member

Back to the topic: the idea is to just somehow "mark" a plugin so it tries to be loaded as early as possible? How one would mark such a plugin?

@dedsm
Copy link

dedsm commented Sep 6, 2016

I think the easiest way is making py.test load first the plugins passed with the -p command line option before the automatic loading using pkg_resources

@nicoddemus
Copy link
Member

nicoddemus commented Sep 6, 2016

@dedsm thanks for the suggestion. I think it is a valid idea, but that does not cover (heh) pytest-cov though... it is a plugin which by definition should be loaded as early as possible, independently if users have passed it into the command line or not.

I'm wondering if people have something in mind already.

@ionelmc
Copy link
Member Author

ionelmc commented Sep 6, 2016

One idea is to introduce another alternative entrypoint name that supports ordering. The object (a module?) that a plugin exports though that entrypoint should have some sort of priority property/attribute. Old style plugins would have default priority at 100 and ordering is done in descending order (plugins with high priority loads first). How about that?

@ionelmc
Copy link
Member Author

ionelmc commented Sep 6, 2016

Actually scratch that, accessing the priority attribute would imply importing module and causing undue issues. Perhaps an entrypoint just for the priority number? Hmmmm

@nicoddemus
Copy link
Member

Would we need to have priority numbers, or something like tryfirst (where we just pile every hook which is declared as tryfirst and hope that's enough)? Priority numbers would mean some coordination between plugin authors...

Not that I have a better idea, mind you. 😜

@The-Compiler
Copy link
Member

The first thing which came to mind are pytest11_tryfirst and pytest11_trylast entrypoints 😆

As for that vs. a priority value, I've actually talked to Holger about that one for plugin hooks, and we agreed it's the much nicer (if less powerful) mechanism, because it requires no (hard to impossible) coordination between authors. As a counter-example, nose seems to use priority values, and apparently that didn't end up being a good idea 😉

@dedsm
Copy link

dedsm commented Sep 7, 2016

I like the tryfirst trylast idea, I think that would solve most of the cases.

for the more obscure ones, how about having a way to alter the order via the config file? a simple list that the developer can alter with a placeholder for the "rest"? something like

[pytest]
plugin_order =
plugin_a
plugin_z
all

that coupled with debug information about the order of loading would be enough for the rest of the cases.

@RonnyPfannschmidt
Copy link
Member

i propose the concept of a pluginspec

instead of having just random modules without order, each plugin should have a spec object telling its dependencies, and the order for consideration for required/optional dependencies

tryfirst/last is imho pretty much a horrific mess

@dedsm
Copy link

dedsm commented Sep 7, 2016

the problem is none of these cases are because of lack of dependencies, the django plugin should never depend on the env plugin, same as in the case of the coverage plugin

@RonnyPfannschmidt
Copy link
Member

so what will you do if both plugins become tryfirst?

tryfirst/last do not at all solve the problem, they just bolt a non-solution on top

what would help is a way to specifiy dependencies in plugins and out of them
if your pytest,ini tells the pluginmanager, oh, in my case django needs env, you specify a topology

also we need a topology anyway - for example py.tests internal plugins can never be loaded via setuptools, and tryfirst/last couldnt fix that

@dedsm
Copy link

dedsm commented Sep 7, 2016

that's why I agree on having a really simple ordering builtin, but ultimately the control should reside in the developer because as far as I can see this problem is not common and should be treated individually

@RonnyPfannschmidt
Copy link
Member

for me the problem is very common and currently solved by not using setuptools, but relying purely on ordered pytest-plugins specification

@dedsm
Copy link

dedsm commented Sep 7, 2016

I agree on setuptools not being a solution, but how and who should decide the topology of plugins that don't have anything to do with each other? it's naiveness thinking that every plugin will specify that relationship with all the rest of the plugins, it's not maintainable

@RonnyPfannschmidt
Copy link
Member

there is need for 2 mechanisms

a) plugin authors spelling dependencies they do know
b) plugin users spelling dependencies plugin authors cant know

@The-Compiler
Copy link
Member

So how would pytest-cov declare that it should be loaded as early as possible?

@RonnyPfannschmidt
Copy link
Member

i propose a priority value for an "absolute order" and before/after listing for putting topology on top of that

also a pytest.ini entry for adding extra topology "hints"

@mattbennett
Copy link

I am struggling with this. The workaround suggested in #935 (comment) does not work if you're trying to order plugins that register themselves using setuptool entry points.

My conftest.py contains:

pytest_plugins = [
    'third_party_plugin_1',
    'third_party_plugin_2',
]

After my conftest registers the plugins, I get ValueError: Plugin already registered when the setuptools entry point subsequently runs.

Using pytest==3.7.2

@virogenesis
Copy link

virogenesis commented Jan 7, 2019

How about a dependency order setting in a plugin making it load after a certain plugin in case it's defined.

That would solve the problem I am having.

depends_on = [...]
Making it load after the ones defined in the list.

@RonnyPfannschmidt
Copy link
Member

@vbarbaresi thats basically the starting point of the topology i was talking about before

@nickwilliams-eventbrite
Copy link

I would just like to throw in another user story here. We have a project using pytest-cov and pytest-django. On one machine, pytest-cov loads first, and all of our __init__.pys and settings files are marked as covered, resulting in coverage of about 90%. On another machine, pytest-django loads first, and all of our __init__.pys and settings files are marked as uncovered, resulting in a coverage of only 74%. This is a HUGE difference and makes it hard to set fail-if-coverage-under thresholds and be able to rely on them.

I really do like this suggestion I saw above:

[pytest]
plugin_order =
    plugin_a
    plugin_z
    all

That seems the most elegant to me, IMO.

@nickwilliams-eventbrite
Copy link

nickwilliams-eventbrite commented Feb 1, 2019

Related (and perhaps I should report an actual bug on this one):

If I look at the help with pytest --help, it says this:

  -p name               early-load given plugin (multi-allowed). To avoid
                        loading of plugins, use the `no:` prefix, e.g.
                        `no:doctest`.

Using that and a suggestion above, hoping I could early-load the coverage plugin, I tried that out:

$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 ...........

Using -p no:pytest_cov results in the same behavior. So It appears that -p name does the exact same thing as -p no:name, which is contrary to the documentation. You can't early-load plugins.

@RonnyPfannschmidt
Copy link
Member

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

@nickwilliams-eventbrite
Copy link

nickwilliams-eventbrite commented Feb 2, 2019

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

That's incredible confusing, and also not documented. Also, if that's the case, I can't figure out what I'm supposed to use to make pytest-cov load early. Things I've tried:

# disables plugin
$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 --cov=.......

# does not recognize it
$ pytest -p pytest-cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'pytest-cov'

# does not recognize it
$ pytest -p cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'cov'

# finally no error, but does not load it early
$ pytest -p coverage
================= test session starts =================
platform linux -- Python 3.7.1, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
Django settings: xxxx.settings.dev_local (from environment variable)
rootdir: /srv/xxxx, inifile: setup.cfg
plugins: django-3.4.6, cov-2.6.1 .......
...
xxxx/settings/__init__.py                           36     36      0      0     0.0%   1-135
xxxx/settings/dev_local.py                          33     33      6      0     0.0%   1-125
xxxx/settings/jenkins_test.py                       24     24      4      0     0.0%   1-131
xxxx/settings/production_docker.py                  23     23      9      0     0.0%   1-125
...

# gives an error about its already being loaded
$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 11, in <module>
    sys.exit(main())
...
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140434823318496': <_pytest.config.PytestPluginManager object at 0x7fb987c20be0>, 'pytestconfig': <_pytest.config.Config object at 0x7fb98498c5c0> ......

Is -p just broken?

@RonnyPfannschmidt
Copy link
Member

its simply not in sync with the setuptools plugin loading mechanism which registers plugins at a different name than their import name while -p works in terms of import names

so in a sense - -p is broken

@nickwilliams-eventbrite
Copy link

Want me to file a new bug about it?

@ionelmc
Copy link
Member Author

ionelmc commented Feb 24, 2019

Ok so I've tried patching iter_entry_points and I realized that from pytest-cov's perspective fixing the order wouldn't really help because pytest loads all the plugins first and only later it starts calling hooks. IOW I don't have a hook to initialize coverage tracer before other plugins get imported.

Well, nothing short of doing it at import time which is way too tricky (inspecting the call stack? ugh ...).

ionelmc added a commit to pytest-dev/pytest-cov that referenced this issue Feb 25, 2019
… plugins. For now this change doesn't really solve anything, see: pytest-dev/pytest#935 (comment)
@nicoddemus
Copy link
Member

nicoddemus commented Mar 31, 2019

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

@Colin-b
Copy link

Colin-b commented Feb 10, 2020

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

Hi @nicoddemus

I tried that but it does not solve the issue.

The command launched by travis is "pytest --cov", the configuration loaded is the exact one you advise.

The "pytest11" entrypoint is the root of the module leading to no coverage at all.

What's confusing here is that between your comment and the documentation, the content of the file is not the same. Also this page is a bit unclear, what specific action is suppose to start pytest-cov engine manually?

Do you have any explanation, documentation, tips to share as to what are the exact options to use in order to make sure that pytest-cov can be use to check coverage of a pytest plugin (containing a "pytest11" entrypoint) ?

Thanks again

@gaborbernat
Copy link
Contributor

I've solved this by using coverage natively instead through pytest-cov, see https://github.com/pytest-dev/pytest-print/blob/eb776107d83a94e90edeb7381d679251ae45bcf0/tox.ini#L27-L33 👍

tony added a commit to tmux-python/libtmux that referenced this issue Sep 10, 2022
Moving from conftest.py to pytest_plugin.py (which comes via a
setuptools entry point) requires a workaround to prevent loss
of coverage from real libtmux tests relying on our pytest plugin.

See also:
- https://pytest-cov.readthedocs.io/en/latest/plugins.html
- pytest-dev/pytest#935 (comment)
tony added a commit to vcs-python/libvcs that referenced this issue Sep 11, 2022
Moving from conftest.py to pytest_plugin.py (which comes via a
setuptools entry point) requires a workaround to prevent loss
of coverage from real libtmux tests relying on our pytest plugin.

See also:
- https://pytest-cov.readthedocs.io/en/latest/plugins.html
- pytest-dev/pytest#935 (comment)
tony added a commit to vcs-python/libvcs that referenced this issue Sep 11, 2022
Moving from conftest.py to pytest_plugin.py (which comes via a
setuptools entry point) requires a workaround to prevent loss
of coverage from real libtmux tests relying on our pytest plugin.

See also:
- https://pytest-cov.readthedocs.io/en/latest/plugins.html
- pytest-dev/pytest#935 (comment)
@tmax22
Copy link

tmax22 commented Oct 30, 2023

2023 and still no way to do so? i'm trying to write plugin to a pytest-plugin. and the only way to do so is to make sure that my plugin is loaded before the other plugin

@RonnyPfannschmidt
Copy link
Member

So far nobody came up with a good solution in the upstream project pluggy

It's been on my radar before I had kids, now not so much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature
Projects
None yet
Development

No branches or pull requests