DM-12454: Be explicit about which python pytest-xdist uses.

[jdswinbank]

When starting a Python interpreter, it will add its own site-packages
directory to sys.path if it's not already there.

Normally, this works fine. However, when the same interpreter can be accessed
through two different paths, it can cause both of those paths to be added to
sys.path.

For example, in our standard, newinstall.sh provided, environment, we might
refer to the Python interpreter either as:

/ssd/lsstsw/stack3_20171021/python/miniconda3-4.3.21/bin/python

or as:

/ssd/lsstsw/stack3_20171021/python/current/bin/python

If we include the former's site packages directory corresponding to the former
on our $PYTHONPATH and then run it, everything is fine. Howver, if we instead
invoke the latter, it will insert a new site-packages directory,
corresponding to that provided by Miniconda, to our sys.path. It doesn't add
it at the end of the list of paths, but rather inserts it at some (arbitrary?)
point in the middle.

This then causes confusion when there are packages installed by Miniconda
which shadow packages provided by EUPS: even if the latter are set up, they
can be overriden by this new entry on sys.path. Confusion ensues.

Here, we explicitly normalize the path used by pytest-xdist to invoke Python
so that we avoid this problem.

[timj]

This seems annoyingly unclear but I guess we have to live with it. Is this only caused by the soft link? Is the real problem that pytest-xdist is installed in the miniconda tree as well? Is this a bug in xdist itself?

[timj]

I think this deserves a comment.

[jdswinbank]

Is this only caused by the soft link?

It's certainly most directly caused by the soft link — I think any time you are invoking multiple copies of the same Python interpreter which you'd like to have the same environment but which you're calling by different names, you're asking for trouble.

Is the real problem that pytest-xdist is installed in the miniconda tree as well?

That's the immediate issue that brought this to our attention. However, it's a general problem: any package that gets installed into the Conda environment might leak through and override the EUPS-provided equivalent.

Is this a bug in xdist itself?

I don't think that correct behaviour is adequately well specified to say. I don't have spare cycles to spend thinking about it further.