New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have a way to control order of plugin loading #935

Open
ionelmc opened this Issue Aug 11, 2015 · 37 comments

Comments

Projects
None yet
10 participants
@ionelmc
Member

ionelmc commented Aug 11, 2015

It appears that the order of pkg_resources.iter_entry_points is pretty arbitrary.

I would like pytest to order that so I can say pytest-cov must load before pytest-benchmark. Currently I cannot use pytest-cov to measure pytest-benchmark due to the ordering issue.

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 12, 2015

needs consideration in pluggy i think

we might want to add a own setuptools metadata writer to pluggy, to have more informative plugin specs

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

We could just sort the entrypoints before calling them, no?

Would there by any problem if I decide to rename my entrypoint?

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 12, 2015

in my oppinion odering by name or lexical order is flawed when dependencies come into play

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

Just for reference, my original issue was with the imports, I wanted pytest-cov to be imported before pytest-benchmark. I have found a workaround since (albeit a bit contrived), just to measure coverage ...

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Aug 12, 2015

early plugin import is planned since a while, just never found a god way to implement

@RonnyPfannschmidt RonnyPfannschmidt added this to the 3.0 milestone Sep 13, 2015

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Aug 5, 2016

I can see how this would be useful for pytest-cov, but it shouldn't block 3.0. Changing the milestone to 3.1 for now.

@The-Compiler The-Compiler modified the milestones: 3.1.0, 3.0 Aug 5, 2016

@dedsm

This comment has been minimized.

dedsm commented Sep 6, 2016

I'm also a victim of this, I'm using pytest_env and pytest_django, my django settings read some variables from environment (that I want to set with pytest_env), but in some computers pytest_django loads before pytest_env, so tests run without the environment.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Sep 6, 2016

Back to the topic: the idea is to just somehow "mark" a plugin so it tries to be loaded as early as possible? How one would mark such a plugin?

@dedsm

This comment has been minimized.

dedsm commented Sep 6, 2016

I think the easiest way is making py.test load first the plugins passed with the -p command line option before the automatic loading using pkg_resources

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Sep 6, 2016

@dedsm thanks for the suggestion. I think it is a valid idea, but that does not cover (heh) pytest-cov though... it is a plugin which by definition should be loaded as early as possible, independently if users have passed it into the command line or not.

I'm wondering if people have something in mind already.

@ionelmc

This comment has been minimized.

Member

ionelmc commented Sep 6, 2016

One idea is to introduce another alternative entrypoint name that supports ordering. The object (a module?) that a plugin exports though that entrypoint should have some sort of priority property/attribute. Old style plugins would have default priority at 100 and ordering is done in descending order (plugins with high priority loads first). How about that?

@ionelmc

This comment has been minimized.

Member

ionelmc commented Sep 6, 2016

Actually scratch that, accessing the priority attribute would imply importing module and causing undue issues. Perhaps an entrypoint just for the priority number? Hmmmm

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Sep 6, 2016

Would we need to have priority numbers, or something like tryfirst (where we just pile every hook which is declared as tryfirst and hope that's enough)? Priority numbers would mean some coordination between plugin authors...

Not that I have a better idea, mind you. 😜

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Sep 7, 2016

The first thing which came to mind are pytest11_tryfirst and pytest11_trylast entrypoints 😆

As for that vs. a priority value, I've actually talked to Holger about that one for plugin hooks, and we agreed it's the much nicer (if less powerful) mechanism, because it requires no (hard to impossible) coordination between authors. As a counter-example, nose seems to use priority values, and apparently that didn't end up being a good idea 😉

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

I like the tryfirst trylast idea, I think that would solve most of the cases.

for the more obscure ones, how about having a way to alter the order via the config file? a simple list that the developer can alter with a placeholder for the "rest"? something like

[pytest]
plugin_order =
plugin_a
plugin_z
all

that coupled with debug information about the order of loading would be enough for the rest of the cases.

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

i propose the concept of a pluginspec

instead of having just random modules without order, each plugin should have a spec object telling its dependencies, and the order for consideration for required/optional dependencies

tryfirst/last is imho pretty much a horrific mess

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

the problem is none of these cases are because of lack of dependencies, the django plugin should never depend on the env plugin, same as in the case of the coverage plugin

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

so what will you do if both plugins become tryfirst?

tryfirst/last do not at all solve the problem, they just bolt a non-solution on top

what would help is a way to specifiy dependencies in plugins and out of them
if your pytest,ini tells the pluginmanager, oh, in my case django needs env, you specify a topology

also we need a topology anyway - for example py.tests internal plugins can never be loaded via setuptools, and tryfirst/last couldnt fix that

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

that's why I agree on having a really simple ordering builtin, but ultimately the control should reside in the developer because as far as I can see this problem is not common and should be treated individually

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

for me the problem is very common and currently solved by not using setuptools, but relying purely on ordered pytest-plugins specification

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

I agree on setuptools not being a solution, but how and who should decide the topology of plugins that don't have anything to do with each other? it's naiveness thinking that every plugin will specify that relationship with all the rest of the plugins, it's not maintainable

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

there is need for 2 mechanisms

a) plugin authors spelling dependencies they do know
b) plugin users spelling dependencies plugin authors cant know

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Sep 7, 2016

So how would pytest-cov declare that it should be loaded as early as possible?

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

i propose a priority value for an "absolute order" and before/after listing for putting topology on top of that

also a pytest.ini entry for adding extra topology "hints"

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

and how about just ditching automatic discovery and make the developer set the order of the plugins all the time? like django middlewares and several other packages do

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Sep 7, 2016

that makes adding plugins to a current env very hard
for example xdist would need to be auto-added

@dedsm

This comment has been minimized.

dedsm commented Sep 7, 2016

then just load everything automatically and just make clear that weird things can happen and if you get into the problem, just alter the loading order in the configuration? I don't think there's an actual need to have extra things in the plugin code and/or hooks, making it optional in the configuration would be really simple to implement IMHO, but then I'm not a core contributor or anything so just my 2 cents

@EmilStenstrom

This comment has been minimized.

EmilStenstrom commented Oct 20, 2016

I just want to echo @dedsm's bug report that this undefined ordering makes tests fail for our team on some computers but not others. The only difference between them is the order at which plugins are loaded. It seems totally arbitary, and since we have no way to control it we can't use some of the plugins at all.

@fletom

This comment has been minimized.

fletom commented Feb 20, 2017

+1

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Feb 21, 2017

@EmilStenstrom a workaround i currently use is - have one plugin, and that declares all other plugins in correct order - setup-tools cant be relied on

@shreyashah

This comment has been minimized.

shreyashah commented Apr 4, 2017

@RonnyPfannschmidt : I am trying to achieve the same thing of ordering the plugins. Trying to understand your solution better of "have one plugin, and that declares all other plugins in correct order". How do we do this?
I mean, where do I declare this in my master plugin?

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Apr 5, 2017

have a myplugin.py and it will declare

pytest_plugins = [
 'plugin1',
 'plugin2',
 ...,
]
@robnagler

This comment has been minimized.

robnagler commented Oct 6, 2017

Running into this issue with xdist/forked. Is there a way to code a plugin to force a dependency? A simple import doesn't do it.

robnagler added a commit to radiasoft/download that referenced this issue Oct 6, 2017

See pytest-dev/pytest#935: plugin order is random
Well, doesn't seem to be truly random in that the
order the plugins are installed seems to matter

robnagler added a commit to radiasoft/sirepo that referenced this issue Oct 6, 2017

See pytest-dev/pytest#935: plugin order is random
Well, doesn't seem to be truly random in that the
order the plugins are installed seems to matter
@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Oct 6, 2017

@robnagler nope, and the code you wrote to force order is fundamentally broken

without more context i cant help as the codebase you work with is hard to navigate without prior knowledge

@robnagler

This comment has been minimized.

robnagler commented Oct 6, 2017

@RonnyPfannschmidt You are right. The code doesn't work.

This comment #935 (comment) suggests we could write a plugin to load other plugins. We could do that, do you have a public example?

The PyKern plugin does not have to be an entry point. It could be explicitly registered by a config. I don't want to start hacking around until I'm sure that path would work. Again, an example would be appreciated.

I'm not sure it matters, but the goal of the plugin is to set some policies:

  • Every test file has the option of having a "work" and a "data" directory, e.g. foo_test.py could have foo_work and foo_data.
  • foo_work needs to be cleared before the first test in the file runs. This allows you to debug previous runs easily.
  • foo_work and foo_data may contain python files so ignore them during collection.
  • Finally, the plugin tracks the current test file. This is useful for managing state across the test file's cases automatically.
  • And, we always run --boxed/--forked for consistency. We run Python-wrapped Fortran, C, C++, codes, and they often fail in mysterious ways. Just easier to always run in separate processes.

robnagler added a commit to radiasoft/pykern that referenced this issue Oct 6, 2017

Another try at radiasoft/sirepo#905
One of the pytest authors responded:
pytest-dev/pytest#935 (comment)
There's no way to force dependencies for pytest plugins.

robnagler added a commit to radiasoft/sirepo that referenced this issue Oct 6, 2017

Another try at #905
One of the pytest authors responded:
pytest-dev/pytest#935 (comment)
There's no way to force dependencies for pytest plugins.
@robnagler

This comment has been minimized.

robnagler commented Oct 6, 2017

@RonnyPfannschmidt never mind. I fixed it using conftest.py. Order seems to be guaranteed that way.

@mattbennett

This comment has been minimized.

mattbennett commented Aug 23, 2018

I am struggling with this. The workaround suggested in #935 (comment) does not work if you're trying to order plugins that register themselves using setuptool entry points.

My conftest.py contains:

pytest_plugins = [
    'third_party_plugin_1',
    'third_party_plugin_2',
]

After my conftest registers the plugins, I get ValueError: Plugin already registered when the setuptools entry point subsequently runs.

Using pytest==3.7.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment