Difference in sys.path order, depending on how run

[nedbat]

In the coverage.py test suite, I have tests that check various environmental values to see that they are they same for coverage run myprog.py as they are for python myprog.py. One of the values is sys.path. Recently, these tests started failing because a .egg file I installed with easy_install would appear in the middle of sys.path for coverage, but at the end for python.

I'm guessing this has something to do with #674, but I don't understand why.

The actual diff of sys.path for the two environments is:

      "path": [
          "/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/test_cover_52005401",
          "/Users/ned/coverage/trunk/tests/modules",
          "/Users/ned/coverage/trunk/tests/zipmods.zip",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python27.zip",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/plat-darwin",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/plat-mac",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/plat-mac/lib-scriptpackages",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/lib-tk",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/lib-old",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/lib-dynload",
          "/usr/local/pythonz/pythons/CPython-2.7.11/lib/python2.7",
          "/usr/local/pythonz/pythons/CPython-2.7.11/lib/python2.7/plat-darwin",
          "/usr/local/pythonz/pythons/CPython-2.7.11/lib/python2.7/lib-tk",
          "/usr/local/pythonz/pythons/CPython-2.7.11/lib/python2.7/plat-mac",
          "/usr/local/pythonz/pythons/CPython-2.7.11/lib/python2.7/plat-mac/lib-scriptpackages",
-         "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/site-packages/covtestegg1-0.0.0-py2.7.egg",
          "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/site-packages",
          "/Users/ned/coverage/trunk",
+         "/Users/ned/coverage/trunk/.tox/py27/lib/python2.7/site-packages/covtestegg1-0.0.0-py2.7.egg",
      ]

Is this expected? Is there something I can do to get the identical behavior in the two environments?

[jaraco]

Hi Ned. I am slightly surprised by the behavior too, and I think you're right that #674 is implicated. To confirm, just set SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite, and if the tests pass again, we'll know for sure that the change is implicated.

Assuming that's the case, here's what I think is happening:

• tox creates the environment using an older version of setuptools, with SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite. This causes the easy-install.pth file to have the rewrite technique at the bottom, causing eggs to appear earlier in the stack.
• tox installs the covtestegg1, so it appears before site-packages.
• tests are run
• setuptools is upgraded, setting SETUPTOOLS_SYS_PATH_TECHNIQUE=raw, and either that upgrade or the installation of another package causes eggs to appear after site-packages.
• tests are run without coverage, but sys.path is different now.

I have a few questions - is setuptools being upgraded as part of the process? If you ran the coverage tests again at the end, I would expect sys.path to be the same as the second run; is it?

It occurs to me that a more proper workaround, assuming you have Setuptools >= 12 installed from the beginning, is that if you were to set SETUPTOOLS_SYS_PATH_TECHINQUE=raw, it would give forward compatibility and should give consistent results.

I hope that helps. Let me know what you find.

[nedbat]

@jaraco running the failing test with SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite makes the test pass.

In this case, the test runs python and coverage directly, one after the other within one test, so I'm not sure there's an opportunity for things to change between them. It seems like something is different about running a program with "python" and running it with a setuptools-created command like "coverage".

Here's the test: https://github.com/nedbat/coveragepy/blob/master/tests/test_process.py#L432-L438

    def test_coverage_run_is_like_python(self):
        with open(TRY_EXECFILE) as f:
            self.make_file("run_me.py", f.read())
        out_cov = self.run_command("coverage run run_me.py")
        out_py = self.run_command("python run_me.py")
        self.assertMultiLineEqual(out_cov, out_py)
        self.assert_execfile_output(out_cov)

To run one test:

COVERAGE_NO_PYTRACER=1 tox -e py27 -- tests/test_process.py:ProcessTest.test_coverage_run_is_like_python

I can try other experiments if you direct me :)

[jaraco]

Oh, my. I wouldn't have expected it, but the import pkg_resources is implicated.

$ python -m venv ~/.envs/issue686
$ SETUPTOOLS_SYS_PATH_TECHINQUE=raw ~/.envs/issue686/bin/python -m easy_install --version                                                               
setuptools 20.10.1 from /Users/jaraco/.envs/issue686/lib/python3.5/site-packages (Python 3.5)
$ SETUPTOOLS_SYS_PATH_TECHINQUE=raw ~/.envs/issue686/bin/python -m easy_install -q requests
$ SETUPTOOLS_SYS_PATH_TECHINQUE=raw ~/.envs/issue686/bin/python -c "import pkg_resources, sys, pprint; pprint.pprint(sys.path)" > with-pkg_resources.txt
$ SETUPTOOLS_SYS_PATH_TECHINQUE=raw ~/.envs/issue686/bin/python -c "import sys, pprint; pprint.pprint(sys.path)" > without-pkg_resources.txt            
$ diff with-pkg_resources.txt without-pkg_resources.txt 
6,7c6,7
<  '/Users/jaraco/.envs/issue686/lib/python3.5/site-packages/requests-2.10.0-py3.5.egg',
<  '/Users/jaraco/.envs/issue686/lib/python3.5/site-packages']
---
>  '/Users/jaraco/.envs/issue686/lib/python3.5/site-packages',
>  '/Users/jaraco/.envs/issue686/lib/python3.5/site-packages/requests-2.10.0-py3.5.egg']

@minrk: Can you investigate and see if you can identify a way for pkg_resources to honor the ordering implied by easy-install.pth per your pull request?

[minrk]

@jaraco I've found the culprit, but I'm not 100% sure what the desired behavior is.

The triggering sequence:

1. requests.egg is already on sys.path, so it is in the initial working set, immediately after its site-packages, as .pth files are supposed to behave.
2. _initialize_master_working_set activates all distributions already on sys.path
3. activate triggers insert_on, which defines its behavior as "Insert self.location in path before its nearest parent directory"

It seems to me that activate shouldn't be touching sys.path if the dist is already on sys.path, at least in this case, but I don't know all of the various cases dist.activate() should cover.

Simply removing the .activate() call from the startup sequence gives the behavior I want/expect, but I'm guessing that's not the right answer.

The high-level problem seems to be that

1. .activate() triggers a forceful, unconditional modification of sys.path.
2. any eggs on sys.path get .activated() on import of pkg_resources.

For my usage, 1. seems to be the problem - I never want pkg_resources to touch sys.path under any circumstances.

I also don't fully understand why pkg_resources is modifying sys.path, because easy-install.pth has already done so (or not, with SETUPTOOLS_SYS_PATH_TECHINQUE=raw).

I assume there are some exceptional cases where pkg_resources is actually meant to modify sys.path. Can you enumerate those, so we can limit the condition when sys.path is modified to less than always?

[nedbat]

I can confirm that setuptools 25.1.1 (with #698 in it) fixes my original problem. Thanks :)