Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with namespace packages, like zc.buildout, installed by pip when other packages from the namespace are installed by buildout #410

Open
jimfulton opened this issue Aug 26, 2017 · 20 comments

Comments

@jimfulton
Copy link
Member

@jimfulton jimfulton commented Aug 26, 2017

I've tried to avoid bootstrapping by installing buildout with pip and then using that to run or bootstrap buildout. If I use buildout to install additional packages from the zc namespace, they end up being unimportable. This is a big problem. :)

Amongst the reasons:

  • See pypa/pip#4695.

  • Namespace packages installed in eggs have __init__.py files, but namespace packages installed by pip don't, in an effort to conform to pep420. According to pep420 means that the namespaces installed by pip should be excluded.

@k4ml

This comment has been minimized.

Copy link

@k4ml k4ml commented Sep 22, 2017

Never run into this. We also use pip to install buildout. This is our bootstrap.py:-

import os
import sys
import subprocess
import functools

run = functools.partial(subprocess.call, shell=True)
run('virtualenv -p python3 .env')
run('.env/bin/pip install --upgrade setuptools')
run('.env/bin/pip install --upgrade pip')
run('.env/bin/pip install zc.buildout')

if not os.path.exists('buildout.cfg'):
    TPL = """
[buildout]
extends = buildout/development.cfg
"""

    f = open('buildout.cfg', 'w')
    f.write(TPL)

run('.env/bin/buildout bootstrap')

We never use any other zc. packages, so I try adding zc.catalog into our buildout.cfg and manage to install and import it fine:-

>>> import zc.catalog
>>> zc.catalog
<module 'zc.catalog' from '/home/kamal/git/gostack/eggs/zc.catalog-2.0.1-py3.4.egg/zc/catalog/__init__.py'>
>>> import zc.buildout
>>> zc.buildout
<module 'zc.buildout' from '/home/kamal/git/gostack/.env/lib/python3.4/site-packages/zc/buildout/__init__.py'>
@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Jan 3, 2018

I was experiencing this before, tried to reproduce today but was unsuccessful. Even tried to play around with the zc.buildout/setuptools version too see if there was a specific version that was causing the issue - but I couldn't find that either.

Fixed by magic? @jimfulton are you still getting the issue?

@jimfulton

This comment has been minimized.

Copy link
Member Author

@jimfulton jimfulton commented Jan 3, 2018

I have lots of weird issues with pip and namespace packages, independent of buildout. Personally, I avoid mixing buildout and pip for dealing with package installation. I think whether or not this bites you depends on many factors (Python version? Pip version?).

@tseaver

This comment has been minimized.

Copy link
Contributor

@tseaver tseaver commented Jan 3, 2018

I think whether or not this bites you depends on many factors (Python version? Pip version?).

Or setuptools version: setuptools and pip have never worked and played well together when it comes to namespace packages. The symptom reappears from time to time when one of them is touched by a maintainer who doesn't use them.

@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Jan 3, 2018

Ok, I hit the bug again today :( here is how to reproduce...

buildout.cfg:

[buildout]
parts = zc.catalog

[zc.catalog]
recipe = zc.recipe.egg
eggs = zc.catalog
interpreter = py

console:

$ python3.6 -m venv py
etc...

$ py/bin/pip install -U setuptools zc.buildout
Collecting setuptools
  Using cached setuptools-38.2.5-py2.py3-none-any.whl
Collecting zc.buildout
  Using cached zc.buildout-2.10.0-py2.py3-none-any.whl
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.2.5 zc.buildout-2.10.0

$ py/bin/buildout
etc...

$ bin/py
>>> import zc.catalog
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'zc.catalog'
@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Jan 3, 2018

Using just easy_install has problems...

$ py/bin/easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Downloading https://pypi.python.org/packages/e9/c3/5986db56819bd88e1a250cad2a97249211686b1b7b5d95f9ab64d403a2cb/setuptools-38.2.5.zip#md5=abfd02fba07b381c3a9682a32d765cc6
Best match: setuptools 38.2.5
etc...

$ py/bin/easy_install -U zc.buildout
Traceback (most recent call last):
  File "py/bin/easy_install", line 11, in <module>
    load_entry_point('setuptools==38.2.5', 'console_scripts', 'easy_install')()
  File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/tmp/bo/py/lib/python3.6/site-packages/setuptools-38.2.5-py3.6.egg/setuptools/command/easy_install.py", line 47, in <module>
  File "/tmp/bo/py/lib/python3.6/site-packages/setuptools-38.2.5-py3.6.egg/setuptools/sandbox.py", line 15, in <module>
ModuleNotFoundError: No module named 'pkg_resources.py31compat'

But if you use a combination of pip and easy_install everything works...

$ py/bin/pip install -U setuptools
etc ...

$ py/bin/easy_install -U zc.buildout
Searching for zc.buildout
Reading https://pypi.python.org/simple/zc.buildout/
Downloading https://pypi.python.org/packages/6f/e0/6d49af55c9ef2893429e668d88cd54b955f804869e703ac241c0ea5b49c7/zc.buildout-2.10.0-py2.py3-none-any.whl#md5=c6c6bfe595d83be2e29c1c7b17df0268
Best match: zc.buildout 2.10.0
etc...

$ py/bin/buildout
Updating zc.catalog.

$ bin/py
>>> import zc.catalog
@reinout

This comment has been minimized.

Copy link
Contributor

@reinout reinout commented Jan 4, 2018

This could be related to #434.

  • Either #434's problem is partially a namespace package problem.
  • Or #410 (this issue) is partially a python 3.5/3.6 problem.

In both cases something isn't found.

@arterrey, can you post what's inside your bin/py script? Especially the sys.path[:] = .... lines?

@mgedmin

This comment has been minimized.

Copy link
Member

@mgedmin mgedmin commented Jan 23, 2018

FWIW I'm seeing this problem on Python 2.7.

My Makefile does

virtualenv -p python2.7 python
python/bin/pip install -U setuptools
python/bin/pip install -U wheel
python/bin/pip install -U zc.buildout
python/bin/buildout bootstrap
bin/buildout

My buildout.cfg uses zc.zope3recipes to generate a bin/foo-instance script, which looks like this

#!/home/mg/src/ivija/python/bin/python2.7

import sys
sys.path[0:0] = [
  '/home/mg/.cache/buildout/eggs/zdaemon-4.1.0-py2.7.egg',
  '/home/mg/src/ivija/python/lib/python2.7/site-packages',
  '/home/mg/.cache/buildout/eggs/ZConfig-2.8.0-py2.7.egg',
  '/home/mg/.cache/buildout/eggs/zc.zope3recipes-0.18.0-py2.7.egg',
  ]

import zc.zope3recipes.ctl

if __name__ == '__main__':
    ...

And, when executed, it fails with an ImportError

Traceback (most recent call last):
  File "bin/ivija-instance", line 11, in <module>
    import zc.zope3recipes.ctl
ImportError: No module named zope3recipes.ctl

Note that buildout itself can import recipes from the zc.* namespace, but buildout-generated scripts fail.

The versions of various components I have are:

$ python/bin/pip list
pip (9.0.1)
setuptools (38.4.0)
wheel (0.30.0)
zc.buildout (2.11.0)
@gyst

This comment has been minimized.

Copy link
Contributor

@gyst gyst commented Feb 8, 2018

arterrey's workaround of updating zc.buildout via easy_install before running buildout itself works for me, in a py36 venv.

@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Feb 12, 2018

@reinout: Soz for the very much delayed response, the contents of my bin/py script are...

#!/tmp/bug/py/bin/python3.6

import sys

sys.path[0:0] = [
  '/tmp/bug/eggs/ZConfig-3.2.0-py3.6.egg',
  '/tmp/bug/eggs/zc.lockfile-1.2.1-py3.6.egg',
  '/tmp/bug/eggs/zodbpickle-1.0-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/transaction-2.1.2-py3.6.egg',
  '/tmp/bug/eggs/ZODB-5.3.0-py3.6.egg',
  '/tmp/bug/eggs/python_gettext-3.0-py3.6.egg',
  '/tmp/bug/eggs/zope.deprecation-4.3.0-py3.6.egg',
  '/tmp/bug/eggs/zope.annotation-4.6.0-py3.6.egg',
  '/tmp/bug/eggs/zope.dottedname-4.2-py3.6.egg',
  '/tmp/bug/eggs/zope.filerepresentation-4.2.0-py3.6.egg',
  '/tmp/bug/eggs/zope.size-4.2.0-py3.6.egg',
  '/tmp/bug/eggs/zope.traversing-4.2.0-py3.6.egg',
  '/tmp/bug/eggs/zope.keyreference-4.1.0-py3.6.egg',
  '/tmp/bug/eggs/zope.lifecycleevent-4.2.0-py3.6.egg',
  '/tmp/bug/eggs/six-1.11.0-py3.6.egg',
  '/tmp/bug/eggs/zope.browser-2.2.0-py3.6.egg',
  '/tmp/bug/eggs/zope.configuration-4.1.0-py3.6.egg',
  '/tmp/bug/eggs/zope.contenttype-4.3.0-py3.6.egg',
  '/tmp/bug/eggs/zope.exceptions-4.2.0-py3.6.egg',
  '/tmp/bug/eggs/zope.i18n-4.3.1-py3.6.egg',
  '/tmp/bug/eggs/zope.event-4.3.0-py3.6.egg',
  '/tmp/bug/eggs/zope.location-4.1.0-py3.6.egg',
  '/tmp/bug/eggs/zope.proxy-4.3.0-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/BTrees-4.4.1-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/persistent-4.2.4.2-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/pytz-2018.3-py3.6.egg',
  '/tmp/bug/eggs/zope.catalog-4.2.1-py3.6.egg',
  '/tmp/bug/eggs/zope.component-4.4.1-py3.6.egg',
  '/tmp/bug/eggs/zope.container-4.2.1-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/zope.i18nmessageid-4.1.0-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/zope.index-4.3.0-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/zope.interface-4.4.3-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/zope.intid-4.3.0-py3.6.egg',
  '/tmp/bug/eggs/zope.publisher-4.3.2-py3.6.egg',
  '/tmp/bug/eggs/zope.schema-4.5.0-py3.6.egg',
  '/tmp/bug/eggs/zope.security-4.2.2-py3.6-linux-x86_64.egg',
  '/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg',
  '/tmp/bug/py/lib/python3.6/site-packages',
  ]


_interactive = True
if len(sys.argv) > 1:
    _options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:')
    _interactive = False
    for (_opt, _val) in _options:
        if _opt == '-i':
            _interactive = True
        elif _opt == '-c':
            exec(_val)
        elif _opt == '-m':
            sys.argv[1:] = _args
            _args = []
            __import__("runpy").run_module(
                 _val, {}, "__main__", alter_sys=True)

    if _args:
        sys.argv[:] = _args
        __file__ = _args[0]
        del _options, _args
        with open(__file__, 'U') as __file__f:
            exec(compile(__file__f.read(), __file__, "exec"))

if _interactive:
    del _interactive
    __import__("code").interact(banner="", local=globals())
@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Feb 12, 2018

Here is perhaps an explanation the difference between the two installs of zc.buidout with pip and easy_install...

Pip installs zc.buildout with the following structure:

$ cd py/lib/python3.6/site-packages/
$ ls
easy_install.py  pip-9.0.1.dist-info  __pycache__  setuptools-38.5.1.dist-info  zc.buildout-2.11.0.dist-info
pip              pkg_resources        setuptools   zc                           zc.buildout-2.11.0-py2.7-nspkg.pth
$ cd zc
$ ls
buildout
$ cd buildout/
$ ls
allowhosts.txt              configparser.py      download.py        __init__.py     runsetup.txt        testrecipes.py
bootstrap_cl_settings.test  configparser.test    download.txt       pep425tags.py   setup.txt           tests.py
bootstrap.txt               debugging.txt        easy_install.py    __pycache__     testing_bugfix.txt  update.txt
buildout.py                 dependencylinks.txt  easy_install.txt   repeatable.txt  testing.py          windows.txt
buildout.txt                downloadcache.txt    extends-cache.txt  rmtree.py       testing.txt

Whilst easy_install installs zc.buildout as:

$ cd py/lib/python3.6/site-packages/
$ ls
easy-install.pth  pip                  pkg_resources  setuptools                   zc.buildout-2.11.0-py3.6.egg
easy_install.py   pip-9.0.1.dist-info  __pycache__    setuptools-38.5.1.dist-info
$ cd zc.buildout-2.11.0-py3.6.egg/
$ ls
EGG-INFO  zc  zc.buildout-2.11.0-py2.7-nspkg.pth

Pip appears to collapse all the name-space packages together into the one tree whilst easy_install keeps them separated in there respective egg packages. That may be an ok thing to todo considering we now have implicit name spaces from PEP 420

Now if I remove the __init__.py out of the zc.catalog egg and also make it an implicit name space things work again...

$ rm eggs/zc.catalog-2.0.1-py3.6.egg/zc/__init__.py 
$ bin/py
>>> import zc.catalog
>>> print(zc.catalog.__path__)
['/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc/catalog']

Looks like implicit and explicit name spaces conflict, it could perhaps be a python bug or a problem with pkg_resources

@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Feb 12, 2018

Some further debugging...

After pdb debugging, it looks like the implicit zc namespace package is setup right from the beginning of the execution of the bin/py script before the sys.path is updated.

(pdb) sys.modules['zc']
<module 'zc' (namespace)>
(pdb) sys.modules['zc'].__path__
_NamespacePath(['/tmp/bug/py/lib/python3.6/site-packages/zc'])

After updating the sys.path you can reload the module and you get the pkg_resources version of the name space packages (I think before implicit name space packages pgk_resources emulated the namespaces etc.) by using importlib.reload(). After which our buildout installed name space packages work.

(Pdb) import importlib
(Pdb) importlib.reload(sys.modules['zc'])
<module 'zc' from '/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc/__init__.py'>
(Pdb) sys.modules['zc'].__path__
['/tmp/bug/eggs/zc.lockfile-1.2.1-py3.6.egg/zc', '/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc']
(Pdb) import zc.catalog
(Pdb) 

But now we can't iimport zc.buildout :( ...

(Pdb) import zc.buildout
*** ModuleNotFoundError: No module named 'zc.buildout'

Looks like having implicit name space packages prevents the pkg_resources emulated name space packages from even being initialized.

@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented Feb 13, 2018

Ok, so some possible solutions...

  • Solution 1: get setuptools to modify installed eggs striping the __init__.py from namespace modules for newer versions of python.

  • Solution 2: Modifiy the the shabang at the bigining of generated scripts so that it starts Python with the -S option "don't imply 'import site' on initialization" this seems to squash the behaviour of loading the implicit name space modules (I'm not sure what other side effects this has) and secondly get pkg_resources name space emulation to support implicit name spaces.

I don't think both of those solutions are compatible with each other - I think I prefer the first solution, it seems the simplest. Any other thoughts?

@delijati

This comment has been minimized.

Copy link

@delijati delijati commented May 2, 2018

Seams to be fixed with setuptools 39.1.0 and pip 10.0.1 on this example buildout. I used python 3.6

@arterrey

This comment has been minimized.

Copy link

@arterrey arterrey commented May 3, 2018

Still broke for me...

console:

$ python3.6 -m venv py
...
$ py/bin/pip install -U pip
...
$ py/bin/pip install -U setuptools zc.buildout
...
$ py/bin/buildout
...
$ bin/py
>>> import zc.catalog
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'zc.catalog'
@jimfulton

This comment has been minimized.

Copy link
Member Author

@jimfulton jimfulton commented May 3, 2018

@arterrey what versions of pip and setuptools are you using?

@delijati

This comment has been minimized.

Copy link

@delijati delijati commented May 3, 2018

@arterrey i used

virtualenv --version
15.2.0

So it seams it has something to do with the virtualenv in python3.6

@mauritsvanrees

This comment has been minimized.

Copy link
Contributor

@mauritsvanrees mauritsvanrees commented May 14, 2018

On Mac, when I try what @arterrey does,, with his original small buildout.cfg I get the same problem, a ModuleNotFoundError on zc.catalog.

I get setuptools 39.0.1 and zc.buildout 2.11.3.

It doesn't matter whether I try python3.6 -m venv py or use virtualenv 13.1.2 or latest 15.2.0.

@mgedmin

This comment has been minimized.

Copy link
Member

@mgedmin mgedmin commented Nov 7, 2018

So far the only workaround for this that works for me on Python 2.7 is to use easy_install zc.buildout instead of pip install zc.buildout.

@jimfulton

This comment has been minimized.

Copy link
Member Author

@jimfulton jimfulton commented Nov 7, 2018

I think it's past time to remove buildout from the zc namespace. I suppose if someone started with a "clean" virtualenv and installed buildout (not zc.buildout), then everything would just work. We'd probably still need a zc.buildout package as a facade for things that imported things from zc.buildout.

(We should move zc.recipe.egg in the same fashion, just to be tidy.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.