setup_requires fails with ValueError when namespace package installed by pip

Originally reported by: jaraco (Bitbucket: jaraco, GitHub: jaraco)

---

I discovered this today. I have a project whose dependencies rely on jaraco.apt and jaraco.timing. In my environment, I have installed another project in the same namespace, jaraco.clipboard using pip. When running the tests for that project, which loads the install_requires and test_requires dependencies via pytest-runner, the loading of the dependencies fails with this traceback:

Traceback (most recent call last):
  File "setup.py", line 50, in <module>
    setuptools.setup(**setup_params)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/jaraco/Dropbox/code/yg/gryphon.cases/.eggs/pytest_runner-2.6.2-py3.5.egg/ptr.py", line 65, in run
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 827, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1072, in best_match
    return self.obtain(req, installer)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1084, in obtain
    return installer(requirement)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
    return cmd.easy_install(req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 640, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 670, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 853, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1081, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1067, in run_setup
    run_setup(setup_script, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 246, in run_setup
    raise
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 243, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 273, in run
    return func()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 242, in runner
    _execfile(setup_script, ns)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-ponrpmyj/jaraco.apt-1.0/setup.py", line 28, in <module>
    'requests',
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/dist.py", line 269, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 827, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1072, in best_match
    return self.obtain(req, installer)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1084, in obtain
    return installer(requirement)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
    return cmd.easy_install(req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 640, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 670, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 853, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1081, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1067, in run_setup
    run_setup(setup_script, args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 233, in run_setup
    with setup_context(setup_dir):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/sandbox.py", line 194, in setup_context
    __import__('setuptools')
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/__init__.py", line 11, in <module>
    from setuptools.extern.six.moves import filterfalse, map
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
    from pkg_resources.extern import VendorImporter
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2912, in <module>
    @_call_aside
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2898, in _call_aside
    f(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2937, in _initialize_master_working_set
    add_activation_listener(lambda dist: dist.activate())
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 956, in subscribe
    callback(dist)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2937, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2488, in activate
    declare_namespace(pkg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2070, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2021, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2039, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2037, in position_in_sys_path
    return sys_path.index(_normalize_cached(os.sep.join(parts)))
ValueError: '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-fqumduuk/jaraco.timing-1.2.2' is not in list

In another run, it failed with a similar traceback, but complained about jaraco.apt not being in the list. Re-running the tests subsequently will succeed, so it seems the issue is affected by the sandbox installer but also by the order in which dependencies are added.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/513

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I need to do more work to investigate, but I wanted to capture my findings. Also, this ticket may be related to #490.

[robwhess]

I just wanted to check in on this issue. I think I'm seeing the same thing. Our team has several packages that are namespace packages. I have some of them listed out in install_requires for one particular package (also a namespace package). When I run python setup.py nosetests on that package, I get an ImportError in the tests each time I try to import something from one of the namespace packages in install_requires. I can see those packages installed as eggs in the .eggs/ directory.

I've had problems using setuptools with namespace packages in the past, too. For example, even using python setup.py install with multiple namespace packages doesn't seem to work, and it's become my common practice to instead run something like python setup.py sdist && pip install dist/*.

Anyway, let me know if there's anything I can do to help debug this problem. I'll be happy to help where I can.

[robwhess]

Woops. @jaraco, I didn't notice you weren't actually explicitly on this thread, so I just wanted to @ mention you.

[jaraco]

Hi @robwhess: I do get notified on every setuptools message, so no need to mention me explicitly, but never hurts if you're unsure.

I'm pretty sure this issue is related to #250 and pypa/pip#3. I'm not familiar with what python setup.py nosetests does, but I suspect it's similar to what setup.py test or setup.py pytest (from pytest-runner) do: installs packages using easy_install to ./.eggs and link them into the environment before running tests.

I've had problems using setuptools with namespace packages in the past, too. For example, even using python setup.py install with multiple namespace packages doesn't seem to work, and it's become my common practice to instead run something like python setup.py sdist && pip install dist/*.

This is definitely linked to pypa/pip#3. A few notes:

• When you invoke python setup.py install, the behavior is not obvious. If the setup.py imports setuptools, the package will be installed as an egg using easy_install, which creates a multi-homed package, something which pip doesn't support well, and which I've attempted to document here.
• If setup.py does not import setuptools but instead invokes distutils, the package will be installed in a flat form, similar to how pip installs them, but until Python 3.3 and PEP 420 namespace support, that namespace package can only exist in one location.
• When you invoke python setup.py sdist && pip install dist/*, that's effectively pip install ., and effectively forces the flat, non-egg installation even if setuptools is present in the setup.py.

Only in the first case will eggs installed in ./.eggs be able to co-exist with packages installed via pip or distutils (when a namespace is shared).