Join GitHub today
Wrong fixture/conftest being used #2836
I discovered a weird issue when running int tests on our Jenkins node at work, which I cannot reproduce inside a dev/vagrant VM using the same versions. The behaviour described below is only on the Jenkins node, but I don't really know where to go next in working out why it's happening. The original errors were in a way more complex setup, but I've managed to reduce this to a fairly minimal example. I doubt anyone else will be able to repro this in isolation, but i'm hoping someone can help me with where to look next to work out what's going on.
Minimal example file structure:
So we have 2 packages, each containing a conftest defining a fixture
What we see is that when
As said at the start, it works fine in a local development (vagrant) VM. Both cases (broken jenkins and working dev VM) are running Ubuntu 14.04 and python 3.5, and I've also created a new virtual env that only contains/installs py.test 3.2.3 to rule out other packages affecting this.
Are there any known bugs/issues in pytest that can cause this sort of behaviour? The original issue was found in a complex nest of parameterised fixtures defined at different levels, and I had originally assumed the bug was in that nest, but distilling it down to this simple example rules all the complexity out. It seems like pytest is simply using the wrong fixture.
Any help or guidance anyone can provide getting to the bottom of this would be very much appreciated!
Thanks both. The minimal example given above fails on Jenkins but not elsewhere. I added
So that suggests it does recognise/collect both conftest modules, yet it's still using the fixture from the
I'll also do a bit more digging into the env vars (Jenkins provides a lot of them, but a very quick skim doesn't show anything obviously askew, and all our other (quite a few of them) python/pytest jobs are working fine fortunately.
So I've been doing a bit more digging, adding a bunch of debugging into pytest to see what's going on. I've found a couple of odd things, but I don't know if this is expected behaviour - it feels like it isn't. The crux of the error/odd behaviour seems to be is that at the point that the aa_foo test is collected.
However, when the
It seemed weird to me that the second test function was being built with both fixtures being passed to it. Unless I misunderstand the scoping/visiblity rules of fixtures, in the setup we have, the contests of
Renamed packages to
The key difference is the contents of the
E.g. in the case where the packages share this common root name, for some reason the fixtures from the first package become valid for use in the second package. Is this intended behaviour? It certainly seems to be odd. For example, it means that the following setup works, but I would expect it to fail:
I would expect
So I think the fact it works locally and not on Jenkins is just symptom. The real issue appears to be is that pytest allows test functions to use fixtures they shouldn't be able to see at all.
In the meantime I'm going to keep digging into how/why both fixtures get passed to the second function when the names share the short common root, in the hope it gets clearer if this is expected or not.
I think I've found the cause of my woes, but I'll need someone to confirm if this is expected behaviour or a bug. I suspect (hope) it's the latter!
The problem seems to come down to this method: https://github.com/pytest-dev/pytest/blob/master/_pytest/fixtures.py#L1133
The way it decides if the node can see the fixture is a simple
This means that in the case we have a test node like
I'll make a minimal example repo to show the behaviour.
Minimal example shown here: https://github.com/tom-dalton-fanduel/pytest-bug-demo/tree/master