New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-level package namespaces break with Setuptools 38 #1214

Closed
georgevreilly opened this Issue Nov 27, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@georgevreilly

georgevreilly commented Nov 27, 2017

Multi-level packages imported via setup.py develop work in Setuptools 37, but fail in Setuptools 38.x due to missing __init__.py in the upper levels of the namespace.

For example, after installing protobuf, a simple import google.protobuf.message should work but doesn't. It can be hacked around by touch protobuf-3.5.0.post1-py2.7.egg/google/__init__.py in site-packages.

Setuptools 37.0

~/tmp/s38.py:

from setuptools import setup

setup(
    name="s38",
    version="0.0.1",
    install_requires=['protobuf', 'zope.deprecation', 'pastedeploy']
)
$ virtualenv --no-setuptools -p python2.7 setuptools37
Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /Users/georgevreilly/setuptools37/bin/python2.7
Also creating executable in /Users/georgevreilly/setuptools37/bin/python
Installing pip, wheel...done.

$ source ./setuptools37/bin/activate

(setuptools37) $ pip install 'setuptools==37.0'
Collecting setuptools==37.0
  Using cached setuptools-37.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-37.0.0

(setuptools37) $ python -c 'import setuptools; print(setuptools.__version__)'
37.0.0

(setuptools37) $ python ~/tmp/s38.py develop
    running develop
    running egg_info
    writing requirements to s38.egg-info/requires.txt
    writing s38.egg-info/PKG-INFO
    writing top-level names to s38.egg-info/top_level.txt
    writing dependency_links to s38.egg-info/dependency_links.txt
    reading manifest file 's38.egg-info/SOURCES.txt'
    writing manifest file 's38.egg-info/SOURCES.txt'
    running build_ext
    Creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/s38.egg-link (link to .)
    Adding s38 0.0.1 to easy-install.pth file

    Installed /Users/georgevreilly
    Processing dependencies for s38==0.0.1
    Searching for pastedeploy
    Reading https://pypi.python.org/simple/pastedeploy/
    Downloading https://pypi.python.org/packages/0f/90/8e20cdae206c543ea10793cbf4136eb9a8b3f417e04e40a29d72d9922cbd/PasteDeploy-1.5.2.tar.gz#md5=352b7205c78c8de4987578d19431af3b
    Best match: PasteDeploy 1.5.2
    Processing PasteDeploy-1.5.2.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-TZTFFm/PasteDeploy-1.5.2/setup.cfg
    Running PasteDeploy-1.5.2/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-TZTFFm/PasteDeploy-1.5.2/egg-dist-tmp-a2YwV8
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Extracting PasteDeploy-1.5.2-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding PasteDeploy 1.5.2 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Searching for zope.deprecation
    Reading https://pypi.python.org/simple/zope.deprecation/
    Downloading https://pypi.python.org/packages/a1/18/2dc5e6bfe64fdc3b79411b67464c55bb0b43b127051a20f7f492ab767758/zope.deprecation-4.3.0.tar.gz#md5=2166b2cb7e0e96a21104e6f8f9b696bb
    Best match: zope.deprecation 4.3.0
    Processing zope.deprecation-4.3.0.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-hWzALQ/zope.deprecation-4.3.0/setup.cfg
    Running zope.deprecation-4.3.0/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-hWzALQ/zope.deprecation-4.3.0/egg-dist-tmp-hANPOl
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Extracting zope.deprecation-4.3.0-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding zope.deprecation 4.3.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Searching for protobuf
    Reading https://pypi.python.org/simple/protobuf/
    Downloading https://pypi.python.org/packages/e9/96/61a46b3cb4b25e4eb1b685266a98463f4a1887b7d4bf8142986c443215d2/protobuf-3.5.0.post1-py2.7.egg#md5=17755f6139c1d85aa75d1c4d77d04cf5
    Best match: protobuf 3.5.0.post1
    Processing protobuf-3.5.0.post1-py2.7.egg
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Extracting protobuf-3.5.0.post1-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding protobuf 3.5.0.post1 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Searching for six>=1.9
    Reading https://pypi.python.org/simple/six/
    Downloading https://pypi.python.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz#md5=d12789f9baf7e9fb2524c0c64f1773f8
    Best match: six 1.11.0
    Processing six-1.11.0.tar.gz
    Writing /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-84SWpD/six-1.11.0/setup.cfg
    Running six-1.11.0/setup.py -q bdist_egg --dist-dir /var/folders/2t/tx6brjtj21vfs98xcmfq1jvr0000gn/T/easy_install-84SWpD/six-1.11.0/egg-dist-tmp-Kh1NXs
    no previously-included directories found matching 'documentation/_build'
    zip_safe flag not set; analyzing archive contents...
    six: module references __path__
    creating /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Extracting six-1.11.0-py2.7.egg to /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Adding six 1.11.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools37/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Searching for setuptools==37.0.0
    Best match: setuptools 37.0.0
    Adding setuptools 37.0.0 to easy-install.pth file
    Installing easy_install script to /Users/georgevreilly/setuptools37/bin
    Installing easy_install-3.6 script to /Users/georgevreilly/setuptools37/bin

    Using /Users/georgevreilly/setuptools37/lib/python2.7/site-packages
    Finished processing dependencies for s38==0.0.1

(setuptools37) $ lssitepackages
    PasteDeploy-1.5.2-py2.7.egg      pkg_resources                    six-1.11.0-py2.7.egg
    easy-install.pth                 protobuf-3.5.0.post1-py2.7.egg   wheel
    easy_install.py                  s38.egg-link                     wheel-0.30.0.dist-info
    easy_install.pyc                 setuptools                       zope.deprecation-4.3.0-py2.7.egg
    pip                              setuptools-37.0.0.dist-info
    pip-9.0.1.dist-info              setuptools.pth

(setuptools37) $ python -c 'from paste.deploy.loadwsgi import AttrDict' && echo yes
yes

(setuptools37) $ python -c 'import google.protobuf.message' && echo yes
yes

(setuptools37) $ python -c 'from zope.deprecation import deprecated' && echo yes
yes

Setuptools 38.2.1

$ virtualenv -p python2.7 setuptools38b
    Running virtualenv with interpreter /usr/local/bin/python2.7
    New python executable in /Users/georgevreilly/setuptools38b/bin/python2.7
    Also creating executable in /Users/georgevreilly/setuptools38b/bin/python
    Installing setuptools, pip, wheel...done.

$ source setuptools38b/bin/activate

(setuptools38b) $ python -c 'import setuptools; print(setuptools.__version__)'
38.2.1

(setuptools38b) $ python ~/tmp/s38.py develop
    running develop
    running egg_info
    creating s38.egg-info
    writing requirements to s38.egg-info/requires.txt
    writing s38.egg-info/PKG-INFO
    writing top-level names to s38.egg-info/top_level.txt
    writing dependency_links to s38.egg-info/dependency_links.txt
    writing manifest file 's38.egg-info/SOURCES.txt'
    reading manifest file 's38.egg-info/SOURCES.txt'
    writing manifest file 's38.egg-info/SOURCES.txt'
    running build_ext
    Creating /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/s38.egg-link (link to .)
    Adding s38 0.0.1 to easy-install.pth file

    Installed /Users/georgevreilly
    Processing dependencies for s38==0.0.1
    Searching for pastedeploy
    Reading https://pypi.python.org/simple/pastedeploy/
    Downloading https://pypi.python.org/packages/31/28/51201a54aeecbd02eff767d17050b302f6fd98fdfecb4e3f4c9301ba6ef8/PasteDeploy-1.5.2-py2.py3-none-any.whl#md5=1134e25af0d52190abd9f8d7c6867f9e
    Best match: PasteDeploy 1.5.2
    Processing PasteDeploy-1.5.2-py2.py3-none-any.whl
    Installing PasteDeploy-1.5.2-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/EGG-INFO/requires.txt
    Adding PasteDeploy 1.5.2 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg
    Searching for zope.deprecation
    Reading https://pypi.python.org/simple/zope.deprecation/
    Downloading https://pypi.python.org/packages/ee/33/625098914ec59b3006adf2cdf44a721e9671f4836af9eeb8cbe14e485954/zope.deprecation-4.3.0-py2.py3-none-any.whl#md5=fb3b24ef645dcfaff8ab3028da7209f2
    Best match: zope.deprecation 4.3.0
    Processing zope.deprecation-4.3.0-py2.py3-none-any.whl
    Installing zope.deprecation-4.3.0-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg/EGG-INFO/requires.txt
    Adding zope.deprecation 4.3.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/zope.deprecation-4.3.0-py2.7.egg
    Searching for protobuf
    Reading https://pypi.python.org/simple/protobuf/
    Downloading https://pypi.python.org/packages/74/34/52e4fcc023f36ba8c408765032e6b9052eff115b01a17e3d2af48fac3a97/protobuf-3.5.0.post1-py2.py3-none-any.whl#md5=74c4b42404b6f55348c280dc511b95c1
    Best match: protobuf 3.5.0.post1
    Processing protobuf-3.5.0.post1-py2.py3-none-any.whl
    Installing protobuf-3.5.0.post1-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    writing requirements to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg/EGG-INFO/requires.txt
    Adding protobuf 3.5.0.post1 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/protobuf-3.5.0.post1-py2.7.egg
    Searching for six>=1.9
    Reading https://pypi.python.org/simple/six/
    Downloading https://pypi.python.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl#md5=866ab722be6bdfed6830f3179af65468
    Best match: six 1.11.0
    Processing six-1.11.0-py2.py3-none-any.whl
    Installing six-1.11.0-py2.py3-none-any.whl to /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    Adding six 1.11.0 to easy-install.pth file

    Installed /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages/six-1.11.0-py2.7.egg
    Searching for setuptools==38.2.1
    Best match: setuptools 38.2.1
    Adding setuptools 38.2.1 to easy-install.pth file
    Installing easy_install script to /Users/georgevreilly/setuptools38b/bin
    Installing easy_install-3.6 script to /Users/georgevreilly/setuptools38b/bin

    Using /Users/georgevreilly/setuptools38b/lib/python2.7/site-packages
    Finished processing dependencies for s38==0.0.1

(setuptools38b) $ lssitepackages
    PasteDeploy-1.5.2-py2.7.egg      pkg_resources                    six-1.11.0-py2.7.egg
    easy-install.pth                 protobuf-3.5.0.post1-py2.7.egg   wheel
    easy_install.py                  s38.egg-link                     wheel-0.30.0.dist-info
    easy_install.pyc                 setuptools                       zope.deprecation-4.3.0-py2.7.egg
    pip                              setuptools-38.2.1.dist-info
    pip-9.0.1.dist-info              setuptools.pth

(setuptools38b) $ python -c 'from paste.deploy.loadwsgi import AttrDict' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named paste.deploy.loadwsgi

(setuptools38b) $ python -c 'import google.protobuf.message' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named google.protobuf.message

(setuptools38b) $ python -c 'from zope.deprecation import deprecated' && echo yes
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named zope.deprecation

Diffing the Trees

(setuptools38b) $ diff -u <(tree setuptools37/lib/python2.7/site-packages) <(tree setuptools38b/lib/python2.7/site-packages)
--- /dev/fd/63	2017-11-26 19:06:51.000000000 -0800
+++ /dev/fd/62	2017-11-26 19:06:51.000000000 -0800
@@ -1,28 +1,23 @@
-setuptools37/lib/python2.7/site-packages
+setuptools38b/lib/python2.7/site-packages
 ├── PasteDeploy-1.5.2-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
+│   │   ├── RECORD
+│   │   ├── WHEEL
 │   │   ├── entry_points.txt
 │   │   ├── namespace_packages.txt
-│   │   ├── not-zip-safe
+│   │   ├── pydist.json
 │   │   ├── requires.txt
 │   │   └── top_level.txt
+│   ├── PasteDeploy-1.5.2-py2.6-nspkg.pth
 │   └── paste
-│       ├── __init__.py
-│       ├── __init__.pyc
 │       └── deploy
 │           ├── __init__.py
-│           ├── __init__.pyc
 │           ├── compat.py
-│           ├── compat.pyc
 │           ├── config.py
-│           ├── config.pyc
 │           ├── converters.py
-│           ├── converters.pyc
 │           ├── loadwsgi.py
-│           ├── loadwsgi.pyc
 │           ├── paster_templates
 │           │   └── paste_deploy
 │           │       ├── +package+
@@ -31,9 +26,7 @@
 │           │       └── docs
 │           │           └── devel_config.ini_tmpl
 │           ├── paster_templates.py
-│           ├── paster_templates.pyc
-│           ├── util.py
-│           └── util.pyc
+│           └── util.py
 ├── easy-install.pth
 ├── easy_install.py
 ├── easy_install.pyc
@@ -607,210 +600,116 @@
 │   └── py31compat.pyc
 ├── protobuf-3.5.0.post1-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
+│   │   ├── RECORD
+│   │   ├── WHEEL
+│   │   ├── metadata.json
 │   │   ├── namespace_packages.txt
-│   │   ├── not-zip-safe
 │   │   ├── requires.txt
 │   │   └── top_level.txt
-│   └── google
-│       ├── __init__.py
-│       ├── __init__.pyc
-│       └── protobuf
-│           ├── __init__.py
-│           ├── __init__.pyc
-│           ├── any_pb2.py
-│           ├── any_pb2.pyc
-│           ├── any_test_pb2.py
-│           ├── any_test_pb2.pyc
-│           ├── api_pb2.py
-│           ├── api_pb2.pyc
-│           ├── compiler
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── plugin_pb2.py
-│           │   └── plugin_pb2.pyc
-│           ├── descriptor.py
-│           ├── descriptor.pyc
-│           ├── descriptor_database.py
-│           ├── descriptor_database.pyc
-│           ├── descriptor_pb2.py
-│           ├── descriptor_pb2.pyc
-│           ├── descriptor_pool.py
-│           ├── descriptor_pool.pyc
-│           ├── duration_pb2.py
-│           ├── duration_pb2.pyc
-│           ├── empty_pb2.py
-│           ├── empty_pb2.pyc
-│           ├── field_mask_pb2.py
-│           ├── field_mask_pb2.pyc
-│           ├── internal
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── _parameterized.py
-│           │   ├── _parameterized.pyc
-│           │   ├── any_test_pb2.py
-│           │   ├── any_test_pb2.pyc
-│           │   ├── api_implementation.py
-│           │   ├── api_implementation.pyc
-│           │   ├── containers.py
-│           │   ├── containers.pyc
-│           │   ├── decoder.py
-│           │   ├── decoder.pyc
-│           │   ├── descriptor_database_test.py
-│           │   ├── descriptor_database_test.pyc
-│           │   ├── descriptor_pool_test.py
-│           │   ├── descriptor_pool_test.pyc
-│           │   ├── descriptor_pool_test1_pb2.py
-│           │   ├── descriptor_pool_test1_pb2.pyc
-│           │   ├── descriptor_pool_test2_pb2.py
-│           │   ├── descriptor_pool_test2_pb2.pyc
-│           │   ├── descriptor_test.py
-│           │   ├── descriptor_test.pyc
-│           │   ├── encoder.py
-│           │   ├── encoder.pyc
-│           │   ├── enum_type_wrapper.py
-│           │   ├── enum_type_wrapper.pyc
-│           │   ├── factory_test1_pb2.py
-│           │   ├── factory_test1_pb2.pyc
-│           │   ├── factory_test2_pb2.py
-│           │   ├── factory_test2_pb2.pyc
-│           │   ├── file_options_test_pb2.py
-│           │   ├── file_options_test_pb2.pyc
-│           │   ├── generator_test.py
-│           │   ├── generator_test.pyc
-│           │   ├── import_test_package
-│           │   │   ├── __init__.py
-│           │   │   ├── __init__.pyc
-│           │   │   ├── inner_pb2.py
-│           │   │   ├── inner_pb2.pyc
-│           │   │   ├── outer_pb2.py
-│           │   │   └── outer_pb2.pyc
-│           │   ├── json_format_test.py
-│           │   ├── json_format_test.pyc
-│           │   ├── message_factory_test.py
-│           │   ├── message_factory_test.pyc
-│           │   ├── message_listener.py
-│           │   ├── message_listener.pyc
-│           │   ├── message_set_extensions_pb2.py
-│           │   ├── message_set_extensions_pb2.pyc
-│           │   ├── message_test.py
-│           │   ├── message_test.pyc
-│           │   ├── missing_enum_values_pb2.py
-│           │   ├── missing_enum_values_pb2.pyc
-│           │   ├── more_extensions_dynamic_pb2.py
-│           │   ├── more_extensions_dynamic_pb2.pyc
-│           │   ├── more_extensions_pb2.py
-│           │   ├── more_extensions_pb2.pyc
-│           │   ├── more_messages_pb2.py
-│           │   ├── more_messages_pb2.pyc
-│           │   ├── packed_field_test_pb2.py
-│           │   ├── packed_field_test_pb2.pyc
-│           │   ├── proto_builder_test.py
-│           │   ├── proto_builder_test.pyc
-│           │   ├── python_message.py
-│           │   ├── python_message.pyc
-│           │   ├── reflection_test.py
-│           │   ├── reflection_test.pyc
-│           │   ├── service_reflection_test.py
-│           │   ├── service_reflection_test.pyc
-│           │   ├── symbol_database_test.py
-│           │   ├── symbol_database_test.pyc
-│           │   ├── test_bad_identifiers_pb2.py
-│           │   ├── test_bad_identifiers_pb2.pyc
-│           │   ├── test_util.py
-│           │   ├── test_util.pyc
-│           │   ├── testing_refleaks.py
-│           │   ├── testing_refleaks.pyc
-│           │   ├── text_encoding_test.py
-│           │   ├── text_encoding_test.pyc
-│           │   ├── text_format_test.py
-│           │   ├── text_format_test.pyc
-│           │   ├── type_checkers.py
-│           │   ├── type_checkers.pyc
-│           │   ├── unknown_fields_test.py
-│           │   ├── unknown_fields_test.pyc
-│           │   ├── well_known_types.py
-│           │   ├── well_known_types.pyc
-│           │   ├── well_known_types_test.py
-│           │   ├── well_known_types_test.pyc
-│           │   ├── wire_format.py
-│           │   ├── wire_format.pyc
-│           │   ├── wire_format_test.py
-│           │   └── wire_format_test.pyc
-│           ├── json_format.py
-│           ├── json_format.pyc
-│           ├── map_proto2_unittest_pb2.py
-│           ├── map_proto2_unittest_pb2.pyc
-│           ├── map_unittest_pb2.py
-│           ├── map_unittest_pb2.pyc
-│           ├── message.py
-│           ├── message.pyc
-│           ├── message_factory.py
-│           ├── message_factory.pyc
-│           ├── proto_builder.py
-│           ├── proto_builder.pyc
-│           ├── pyext
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── cpp_message.py
-│           │   ├── cpp_message.pyc
-│           │   ├── python_pb2.py
-│           │   └── python_pb2.pyc
-│           ├── reflection.py
-│           ├── reflection.pyc
-│           ├── service.py
-│           ├── service.pyc
-│           ├── service_reflection.py
-│           ├── service_reflection.pyc
-│           ├── source_context_pb2.py
-│           ├── source_context_pb2.pyc
-│           ├── struct_pb2.py
-│           ├── struct_pb2.pyc
-│           ├── symbol_database.py
-│           ├── symbol_database.pyc
-│           ├── test_messages_proto2_pb2.py
-│           ├── test_messages_proto2_pb2.pyc
-│           ├── test_messages_proto3_pb2.py
-│           ├── test_messages_proto3_pb2.pyc
-│           ├── text_encoding.py
-│           ├── text_encoding.pyc
-│           ├── text_format.py
-│           ├── text_format.pyc
-│           ├── timestamp_pb2.py
-│           ├── timestamp_pb2.pyc
-│           ├── type_pb2.py
-│           ├── type_pb2.pyc
-│           ├── unittest_arena_pb2.py
-│           ├── unittest_arena_pb2.pyc
-│           ├── unittest_custom_options_pb2.py
-│           ├── unittest_custom_options_pb2.pyc
-│           ├── unittest_import_pb2.py
-│           ├── unittest_import_pb2.pyc
-│           ├── unittest_import_public_pb2.py
-│           ├── unittest_import_public_pb2.pyc
-│           ├── unittest_mset_pb2.py
-│           ├── unittest_mset_pb2.pyc
-│           ├── unittest_mset_wire_format_pb2.py
-│           ├── unittest_mset_wire_format_pb2.pyc
-│           ├── unittest_no_arena_import_pb2.py
-│           ├── unittest_no_arena_import_pb2.pyc
-│           ├── unittest_no_arena_pb2.py
-│           ├── unittest_no_arena_pb2.pyc
-│           ├── unittest_no_generic_services_pb2.py
-│           ├── unittest_no_generic_services_pb2.pyc
-│           ├── unittest_pb2.py
-│           ├── unittest_pb2.pyc
-│           ├── unittest_proto3_arena_pb2.py
-│           ├── unittest_proto3_arena_pb2.pyc
-│           ├── util
-│           │   ├── __init__.py
-│           │   ├── __init__.pyc
-│           │   ├── json_format_proto3_pb2.py
-│           │   └── json_format_proto3_pb2.pyc
-│           ├── wrappers_pb2.py
-│           └── wrappers_pb2.pyc
+│   ├── google
+│   │   └── protobuf
+│   │       ├── __init__.py
+│   │       ├── any_pb2.py
+│   │       ├── any_test_pb2.py
+│   │       ├── api_pb2.py
+│   │       ├── compiler
+│   │       │   ├── __init__.py
+│   │       │   └── plugin_pb2.py
+│   │       ├── descriptor.py
+│   │       ├── descriptor_database.py
+│   │       ├── descriptor_pb2.py
+│   │       ├── descriptor_pool.py
+│   │       ├── duration_pb2.py
+│   │       ├── empty_pb2.py
+│   │       ├── field_mask_pb2.py
+│   │       ├── internal
+│   │       │   ├── __init__.py
+│   │       │   ├── _parameterized.py
+│   │       │   ├── any_test_pb2.py
+│   │       │   ├── api_implementation.py
+│   │       │   ├── containers.py
+│   │       │   ├── decoder.py
+│   │       │   ├── descriptor_database_test.py
+│   │       │   ├── descriptor_pool_test.py
+│   │       │   ├── descriptor_pool_test1_pb2.py
+│   │       │   ├── descriptor_pool_test2_pb2.py
+│   │       │   ├── descriptor_test.py
+│   │       │   ├── encoder.py
+│   │       │   ├── enum_type_wrapper.py
+│   │       │   ├── factory_test1_pb2.py
+│   │       │   ├── factory_test2_pb2.py
+│   │       │   ├── file_options_test_pb2.py
+│   │       │   ├── generator_test.py
+│   │       │   ├── import_test_package
+│   │       │   │   ├── __init__.py
+│   │       │   │   ├── inner_pb2.py
+│   │       │   │   └── outer_pb2.py
+│   │       │   ├── json_format_test.py
+│   │       │   ├── message_factory_test.py
+│   │       │   ├── message_listener.py
+│   │       │   ├── message_set_extensions_pb2.py
+│   │       │   ├── message_test.py
+│   │       │   ├── missing_enum_values_pb2.py
+│   │       │   ├── more_extensions_dynamic_pb2.py
+│   │       │   ├── more_extensions_pb2.py
+│   │       │   ├── more_messages_pb2.py
+│   │       │   ├── packed_field_test_pb2.py
+│   │       │   ├── proto_builder_test.py
+│   │       │   ├── python_message.py
+│   │       │   ├── reflection_test.py
+│   │       │   ├── service_reflection_test.py
+│   │       │   ├── symbol_database_test.py
+│   │       │   ├── test_bad_identifiers_pb2.py
+│   │       │   ├── test_util.py
+│   │       │   ├── testing_refleaks.py
+│   │       │   ├── text_encoding_test.py
+│   │       │   ├── text_format_test.py
+│   │       │   ├── type_checkers.py
+│   │       │   ├── unknown_fields_test.py
+│   │       │   ├── well_known_types.py
+│   │       │   ├── well_known_types_test.py
+│   │       │   ├── wire_format.py
+│   │       │   └── wire_format_test.py
+│   │       ├── json_format.py
+│   │       ├── map_proto2_unittest_pb2.py
+│   │       ├── map_unittest_pb2.py
+│   │       ├── message.py
+│   │       ├── message_factory.py
+│   │       ├── proto_builder.py
+│   │       ├── pyext
+│   │       │   ├── __init__.py
+│   │       │   ├── cpp_message.py
+│   │       │   └── python_pb2.py
+│   │       ├── reflection.py
+│   │       ├── service.py
+│   │       ├── service_reflection.py
+│   │       ├── source_context_pb2.py
+│   │       ├── struct_pb2.py
+│   │       ├── symbol_database.py
+│   │       ├── test_messages_proto2_pb2.py
+│   │       ├── test_messages_proto3_pb2.py
+│   │       ├── text_encoding.py
+│   │       ├── text_format.py
+│   │       ├── timestamp_pb2.py
+│   │       ├── type_pb2.py
+│   │       ├── unittest_arena_pb2.py
+│   │       ├── unittest_custom_options_pb2.py
+│   │       ├── unittest_import_pb2.py
+│   │       ├── unittest_import_public_pb2.py
+│   │       ├── unittest_mset_pb2.py
+│   │       ├── unittest_mset_wire_format_pb2.py
+│   │       ├── unittest_no_arena_import_pb2.py
+│   │       ├── unittest_no_arena_pb2.py
+│   │       ├── unittest_no_generic_services_pb2.py
+│   │       ├── unittest_pb2.py
+│   │       ├── unittest_proto3_arena_pb2.py
+│   │       ├── util
+│   │       │   ├── __init__.py
+│   │       │   └── json_format_proto3_pb2.py
+│   │       └── wrappers_pb2.py
+│   └── protobuf-3.5.0.post1-py2.7-nspkg.pth
 ├── s38.egg-link
 ├── setuptools
 │   ├── __init__.py
@@ -887,6 +786,8 @@
 │   ├── extern
 │   │   ├── __init__.py
 │   │   └── __init__.pyc
+│   ├── glibc.py
+│   ├── glibc.pyc
 │   ├── glob.py
 │   ├── glob.pyc
 │   ├── gui-32.exe
@@ -904,6 +805,8 @@
 │   ├── namespaces.pyc
 │   ├── package_index.py
 │   ├── package_index.pyc
+│   ├── pep425tags.py
+│   ├── pep425tags.pyc
 │   ├── py27compat.py
 │   ├── py27compat.pyc
 │   ├── py31compat.py
@@ -924,9 +827,11 @@
 │   ├── unicode_utils.pyc
 │   ├── version.py
 │   ├── version.pyc
+│   ├── wheel.py
+│   ├── wheel.pyc
 │   ├── windows_support.py
 │   └── windows_support.pyc
-├── setuptools-37.0.0.dist-info
+├── setuptools-38.2.1.dist-info
 │   ├── DESCRIPTION.rst
 │   ├── INSTALLER
 │   ├── LICENSE.txt
@@ -941,13 +846,13 @@
 ├── setuptools.pth
 ├── six-1.11.0-py2.7.egg
 │   ├── EGG-INFO
+│   │   ├── DESCRIPTION.rst
 │   │   ├── PKG-INFO
-│   │   ├── SOURCES.txt
-│   │   ├── dependency_links.txt
-│   │   ├── not-zip-safe
+│   │   ├── RECORD
+│   │   ├── WHEEL
+│   │   ├── metadata.json
 │   │   └── top_level.txt
-│   ├── six.py
-│   └── six.pyc
+│   └── six.py
 ├── wheel
 │   ├── __init__.py
 │   ├── __init__.pyc
@@ -999,24 +904,20 @@
 │   └── top_level.txt
 └── zope.deprecation-4.3.0-py2.7.egg
     ├── EGG-INFO
+    │   ├── DESCRIPTION.rst
     │   ├── PKG-INFO
-    │   ├── SOURCES.txt
-    │   ├── dependency_links.txt
+    │   ├── RECORD
+    │   ├── WHEEL
+    │   ├── metadata.json
     │   ├── namespace_packages.txt
-    │   ├── not-zip-safe
     │   ├── requires.txt
     │   └── top_level.txt
-    └── zope
-        ├── __init__.py
-        ├── __init__.pyc
-        └── deprecation
-            ├── __init__.py
-            ├── __init__.pyc
-            ├── deprecation.py
-            ├── deprecation.pyc
-            ├── fixture.py
-            ├── fixture.pyc
-            ├── tests.py
-            └── tests.pyc
+    ├── zope
+    │   └── deprecation
+    │       ├── __init__.py
+    │       ├── deprecation.py
+    │       ├── fixture.py
+    │       └── tests.py
+    └── zope.deprecation-4.3.0-py3.6-nspkg.pth

-69 directories, 950 files
+69 directories, 851 files
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Nov 27, 2017

Proposed fix here: #1215

@reinout

This comment has been minimized.

Contributor

reinout commented Dec 1, 2017

It doesn't work yet for me. I'm trying to get buildout to work with the new wheel support in buildout/buildout#428, but the tests don't run because the namespace package zope.testrunner isn't importable

https://pypi.python.org/packages/df/a6/899d21e90e2fccec31481ed5c137c4f967c59ad6f827546d7f60b74a8455/zope.testrunner-4.7.0-py2.py3-none-any.whl

In the fix, on line da1c78f#diff-17b52a268b98d01dd98fa746689f9383R135, you look for namespace-packages.txt in the egg-info directory. But zope.testrunner's wheel has that file in its dist-info directory:

screen shot 2017-12-01 at 10 40 51

I don't know if the wheel is incorrect or if dist-info should be checked "in addition to" or "instead of" egg-info.

If I add an __init__.py in the zope/ directory of the unpacked wheel, zope.testrunner is importable.

@leorochael

This comment has been minimized.

Contributor

leorochael commented Dec 1, 2017

I suggest we reopen this issue until @reinout's comment is addressed. I've made a concrete suggestion for a fix here.

@leorochael

This comment has been minimized.

Contributor

leorochael commented Dec 1, 2017

I suggest we reopen this issue until @reinout's comment is addressed. I've made a concrete suggestion for a fix here.

@reinout, Can you check if the fix I proposed above solves the namespace package issue on zope.testrunner?

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Dec 1, 2017

@reinout: I can't reproduce the issue, can you give more detailed steps? Here is what I used:

> python -m venv venv
> ./venv/bin/pip install -q -U setuptools
> ./venv/bin/easy_install -q zope.testrunner
> ./venv/bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3
six==1.11.0
zope.exceptions==4.2.0
zope.interface==4.4.3
zope.testrunner==4.8.1
>  ./venv/bin/python -c 'import zope'
> ls venv/lib/python3.6/site-packages/zope.testrunner-4.8.1-py3.6.egg/zope/__init__.py
venv/lib/python3.6/site-packages/zope.testrunner-4.8.1-py3.6.egg/zope/__init__.py
@reinout

This comment has been minimized.

Contributor

reinout commented Dec 1, 2017

The link of @leorochael is faulty, this is the right one: #1215 (comment)

@benoit-pierre I "tested" it through buildout's tests. I'll see if I can make a small example (though yours looks good).

@leorochael

This comment has been minimized.

Contributor

leorochael commented Dec 1, 2017

@reinout: I can't reproduce the issue, can you give more detailed steps?

@benoit-pierre, here is a reproduction:

$ virtualenv setuptools_1214 --python=python2
$ cd setuptools_1214
$ bin/pip install --upgrade pip setuptools
$ bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3

Here we make use of an explicit (though not often used) use-case of setuptools, a plugin directory that is populated with eggs that are dynamically installed by the aplication:

$ mkdir plugins scripts
$ bin/easy_install -qm -s scripts -d plugins zope.testrunner
$ ls plugins/
six-1.11.0-py2.7.egg
zope.exceptions-4.2.0-py2.7.egg
zope.interface-4.4.3-py2.7-linux-x86_64.egg
zope.testrunner-4.8.1-py2.7.egg

And here we make use of the plugins directory:

$ bin/python
Python 2.7.6 (default, Nov 23 2017, 15:49:48) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, glob
>>> sys.path[:0] = glob.glob('plugins/*.egg')
>>> import zope.testrunner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.testrunner
>>> import zope.interface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.interface
>>> import zope.exceptions
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zope.exceptions
>>> import six
>>> print six.__file__
plugins/six-1.11.0-py2.7.egg/six.pyc

All of the packages above should have been imported. Not only six.

@reinout

This comment has been minimized.

Contributor

reinout commented Dec 1, 2017

That's a different problem from mine. In my case, zope.interface and zope.exceptions can be imported just fine (being build from a *.tar.gz).

zope.testrunner is a .whl, not a .egg. The problem seems to be that buildout is asking setuptools to extract it like an egg instead of converting the wheel to an egg.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Dec 1, 2017

I can't reproduce:

> python2 -m virtualenv venv
New python executable in venv/bin/python2
Also creating executable in venv/bin/python
Installing setuptools, pip, wheel...done.
> ./venv/bin/pip install -q -U setuptools
> ./venv/bin/pip freeze --all
pip==9.0.1
setuptools==38.2.3
wheel==0.30.0
> mkdir plugins
> PYTHONPATH=plugins ./venv/bin/easy_install -qm -s scripts -d plugins zope.testrunner
> ls plugins
six-1.11.0-py2.7.egg/             zope.interface-4.4.3-py2.7-linux-x86_64.egg/
zope.exceptions-4.2.0-py2.7.egg/  zope.testrunner-4.8.1-py2.7.egg/
> ./venv/bin/python
Python 2.7.14 (default, Sep 20 2017, 01:25:59)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import glob, sys
>>> sys.path[:0] = glob.glob('plugins/*.egg')
>>> import zope.testrunner
>>> zope.__file__
'[...]/plugins/zope.interface-4.4.3-py2.7-linux-x86_64.egg/zope/__init__.py'
>>>
@reinout

This comment has been minimized.

Contributor

reinout commented Dec 1, 2017

I think we can close this again. Setuptools handles it just fine.
I was just calling the wrong function. buildout/buildout@4904f96 is what I should have done.

@leorochael

This comment has been minimized.

Contributor

leorochael commented Dec 1, 2017

It's still closed ;-)

And I can't reproduce it either. don't know what I did wrong the first time on my reproduction. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment