FileNotFoundError on older Pythons with flake8 3.2

[jaraco]

Recently and inexplicably, tests have started failing on Python 3.2 and earlier in Travis-CI, such as seen in this run. Mainly one test is failing:

_____________________ TestUserInstallTest.test_local_index _____________________

self = <setuptools.tests.test_easy_install.TestUserInstallTest object at 0x7f7d68cbab10>

foo_package = '/tmp/pytest-of-travis/pytest-0/test_local_index0'

install_target = '/tmp/pytest-of-travis/pytest-0/test_local_index0'

    def test_local_index(self, foo_package, install_target):

        """

            The local index must be used when easy_install locates installed

            packages.

            """

        dist = Distribution()

        dist.script_name = 'setup.py'

        cmd = ei.easy_install(dist)

        cmd.install_dir = install_target

        cmd.args = ['foo']

        cmd.ensure_finalized()

        cmd.local_index.scan([foo_package])

>       res = cmd.easy_install('foo')

/home/travis/build/pypa/setuptools/setuptools/tests/test_easy_install.py:289: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

./setuptools/command/easy_install.py:633: in easy_install

    self.install_site_py()

./setuptools/command/easy_install.py:1276: in install_site_py

    source = resource_string("setuptools", "site-patch.py")

./pkg_resources/__init__.py:1214: in resource_string

    self, resource_name

./pkg_resources/__init__.py:1456: in get_resource_string

    return self._get(self._fn(self.module_path, resource_name))

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pkg_resources.DefaultProvider object at 0x7f7d68be0490>

path = './setuptools/site-patch.py'

    def _get(self, path):

>       with open(path, 'rb') as stream:

E       FileNotFoundError: [Errno 2] No such file or directory: './setuptools/site-patch.py'

./pkg_resources/__init__.py:1576: FileNotFoundError

I have no idea why it's failing.

[jaraco]

I'm able to replicate this failure on my local workstation.

[jaraco]

I can't tell when this issue started. It looks like there was a clean build 5 days ago, and the first failure happened about an hour ago, so something apparently changed in Travis-CI and on my machine in that period. I haven't run brew update this week. I've tried downgrading pip and tox and setuptools, but to no avail.

[jaraco]

I can replicate the failure surgically running this command:

python -m tox -e py27 -- -k 'test_local_index and test_easy_install'

Even if I go back to the code where the last clean build happened, I get the error, so it's definitely something environmental triggering the failure. I noticed that pytest was recently updated to 3.0.4, so I tried downgrading pytest to 3.0.3, but that did not help.

[jaraco]

I grabbed the requirements specifically from the list of requirements emitted in the last clean build (thanks pytest for reporting that information), and the test passes. Now I should be able to bisect those changes and find the offending package.

[jaraco]

Installing flake8==3.2.1 and pyflakes==1.3.0 causes the failure whereas flake8==3.0.4 and pyflakes==1.2.3 does not.

[jaraco]

It seems that with these later versions of flake8 and pyflakes installed, the setuptools module is getting picked up in the current directory with a __file__ attribute that's relative to the current working directory. Here I've replicated the issue without setuptools source:

$ tree
.
├── setuptools
│   └── __init__.py
├── test_issue852.py
└── tox.ini

1 directory, 3 files
$ cat test_issue852.py 
import setuptools

def test_path():
    assert setuptools.__file__.startswith('/')
$ cat tox.ini
[tox]
skipsdist = true

[testenv]
deps =
  pyflakes == 1.3.0
  flake8 == 3.2.0
  pytest-flake8
  pytest
commands =
  python -m pytest
$ python -m tox -e py27       
py27 installed: configparser==3.5.0,enum34==1.1.6,flake8==3.2.0,mccabe==0.5.2,py==1.4.31,pycodestyle==2.2.0,pyflakes==1.3.0,pytest==3.0.4,pytest-flake8==0.8.1
py27 runtests: PYTHONHASHSEED='888178008'
py27 runtests: commands[0] | python -m pytest
====================================== test session starts =======================================
platform darwin -- Python 2.7.12, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: /Users/jaraco/issue-852, inifile: 
plugins: flake8-0.8.1
collected 1 items 

test_issue852.py F

============================================ FAILURES ============================================
___________________________________________ test_path ____________________________________________

    def test_path():
>       assert setuptools.__file__.startswith('/')
E       assert False
E        +  where False = <built-in method startswith of str object at 0x11048f9b0>('/')
E        +    where <built-in method startswith of str object at 0x11048f9b0> = 'setuptools/__init__.py'.startswith
E        +      where 'setuptools/__init__.py' = setuptools.__file__

test_issue852.py:4: AssertionError
==================================== 1 failed in 0.05 seconds ====================================
ERROR: InvocationError: '/Users/jaraco/issue-852/.tox/py27/bin/python -m pytest'
____________________________________________ summary _____________________________________________
ERROR:   py27: commands failed

[jaraco]

And I've confirmed, running that same test with pyflakes 1.2.3 and flake8 3.0.4, the test passes.

[jaraco]

Here are the differences in pyflakes and the differences in flake8.

[jaraco]

One noticeable difference in flake8 is it now imports setuptools in flake8/main/debug.py. Indeed, if I comment out that import, the problem goes away.

[jaraco]

Running pdb, I can see the stack when that import is invoked. It's early in the pytest setup:

$ python -m tox -e py27
py27 installed: configparser==3.5.0,enum34==1.1.6,flake8==3.2.1,mccabe==0.5.2,py==1.4.31,pycodestyle==2.2.0,pyflakes==1.3.0,pytest==3.0.4,pytest-flake8==0.8.1
py27 runtests: PYTHONHASHSEED='2843664332'
py27 runtests: commands[0] | python -m pytest
> /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/flake8/main/debug.py(8)<module>()
-> import setuptools
(Pdb) w
  /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py(174)_run_module_as_main()
-> "__main__", fname, loader, pkg_name)
  /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py(72)_run_code()
-> exec code in run_globals
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/pytest.py(17)<module>()
-> raise SystemExit(pytest.main())
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/config.py(47)main()
-> config = _prepareconfig(args, plugins)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/config.py(132)_prepareconfig()
-> pluginmanager=pluginmanager, args=args)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(745)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(339)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(334)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(613)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(248)_wrapped_call()
-> call_outcome = _CallOutcome(func)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(265)__init__()
-> self.result = func()
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(614)execute()
-> res = hook_impl.function(*args)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/config.py(882)pytest_cmdline_parse()
-> self.parse(args)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/config.py(1038)parse()
-> self._preparse(args, addopts=addopts)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/config.py(1001)_preparse()
-> self.pluginmanager.load_setuptools_entrypoints(entrypoint_name)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(510)load_setuptools_entrypoints()
-> plugin = ep.load()
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/pkg_resources/__init__.py(2291)load()
-> return self.resolve()
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/pkg_resources/__init__.py(2297)resolve()
-> module = __import__(self.module_name, fromlist=['__name__'], level=0)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/_pytest/assertion/rewrite.py(212)load_module()
-> py.builtin.exec_(co, mod.__dict__)
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/py/_builtin.py(221)exec_()
-> exec2(obj, globals, locals)
  <string>(7)exec2()
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/pytest_flake8.py(6)<module>()
-> from flake8.main import application
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/flake8/main/application.py(14)<module>()
-> from flake8.main import options
  /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/flake8/main/options.py(3)<module>()
-> from flake8.main import debug
> /Users/jaraco/issue-852/.tox/py27/lib/python2.7/site-packages/flake8/main/debug.py(8)<module>()
-> import setuptools

[jaraco]

In the context of the test, sys.path looks like: ['/Users/jaraco/issue-852', '', '/Users/jaraco/issue-852/.tox/py27/lib/python27.zip', .... So whatever is adding that absolute path before the '' path is what's allowing the tests to pass when setuptools wasn't imported during the plugin setup phase, but that initial value is not present at the time the plugins are initialized.

[jaraco]

I notice that the issue does not occur on Python 3.3+ because import setuptools creates a setuptools.__file__ that's absolute, even when setuptools is resolved from a sys.path entry of ''.

$ python3 -c "import setuptools; print(setuptools.__file__)"
/Users/jaraco/issue-852/setuptools/__init__.py
$ python2 -c "import setuptools; print(setuptools.__file__)"
setuptools/__init__.pyc

[jaraco]

This latest commit fixed most of the failures, but something is still failing on pypy3. Aah. It's support for Python 3.2 dropped from pip. Separate issue.