New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMI zope errors related to PYTHON_INSTALL_LAYOUT #2823

Open
douglarek opened this Issue Apr 13, 2016 · 27 comments

Comments

Projects
None yet
@douglarek

douglarek commented Apr 13, 2016


       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
./letsencrypt-auto certonly --standalone --email www@icocoa.me -d icocoa.me -d www.icocoa.me --debug
Checking for new version...
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Requesting root privileges to run letsencrypt...
   sudo /home/ec2-user/.local/share/letsencrypt/bin/letsencrypt certonly --standalone --email www@icocoa.me -d icocoa.me -d www.icocoa.me --debug
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module>
    from letsencrypt.main import main
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/dist-packages/letsencrypt/main.py", line 11, in <module>
    import zope.component
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/dist-packages/zope/component/__init__.py", line 16, in <module>
    from zope.interface import Interface
ImportError: No module named interface
/home/ec2-user/.local/share/letsencrypt/bin/pip install zope.interface interface
Collecting zope.interface
Collecting interface
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from zope.interface)
Collecting zope.schema (from interface)
Requirement already satisfied (use --upgrade to upgrade): zope.event in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from zope.schema->interface)
Installing collected packages: zope.interface, zope.schema, interface
Successfully installed interface zope.interface zope.schema-4.4.2
You are using pip version 8.0.3, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/home/ec2-user/.local/share/letsencrypt/bin/python
Python 2.7.10 (default, Dec  8 2015, 18:25:23)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from zope.interface import Interface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named interface

master branch 9d7519c

@ricog

This comment has been minimized.

Show comment
Hide comment
@ricog

ricog Apr 14, 2016

I had similar errors, but was able to get it working with these commands (as root): #1680 (comment)

ricog commented Apr 14, 2016

I had similar errors, but was able to get it working with these commands (as root): #1680 (comment)

@robertsky

This comment has been minimized.

Show comment
Hide comment
@robertsky

robertsky Apr 18, 2016

Having the same issue. The commands that @ricog suggested didn't work. There is a similar report made earlier, #2804.

robertsky commented Apr 18, 2016

Having the same issue. The commands that @ricog suggested didn't work. There is a similar report made earlier, #2804.

@keemyb

This comment has been minimized.

Show comment
Hide comment
@keemyb

keemyb Apr 20, 2016

export PYTHON_INSTALL_LAYOUT=amzon as root was what got this working for me, as stated here #1680 (comment)

keemyb commented Apr 20, 2016

export PYTHON_INSTALL_LAYOUT=amzon as root was what got this working for me, as stated here #1680 (comment)

@douglarek

This comment has been minimized.

Show comment
Hide comment
@douglarek

douglarek Apr 27, 2016

The commands that @keemyb suggested didn't work.

douglarek commented Apr 27, 2016

The commands that @keemyb suggested didn't work.

@keemyb

This comment has been minimized.

Show comment
Hide comment

keemyb commented Apr 27, 2016

@douglarek what about this comment?

@douglarek

This comment has been minimized.

Show comment
Hide comment

douglarek commented Apr 27, 2016

@keemyb #1680 (comment) not work too

@mmylokostenko

This comment has been minimized.

Show comment
Hide comment
@mmylokostenko

mmylokostenko May 24, 2016

unset PYTHON_INSTALL_LAYOUT as root was what got this working for me. i found solution here http://stackoverflow.com/questions/30983243/django-installed-in-dist-packages

mmylokostenko commented May 24, 2016

unset PYTHON_INSTALL_LAYOUT as root was what got this working for me. i found solution here http://stackoverflow.com/questions/30983243/django-installed-in-dist-packages

@douglarek

This comment has been minimized.

Show comment
Hide comment
@douglarek

douglarek May 24, 2016

@MadMaxUa thanks, unset PYTHON_INSTALL_LAYOUT works!

douglarek commented May 24, 2016

@MadMaxUa thanks, unset PYTHON_INSTALL_LAYOUT works!

@douglarek douglarek closed this May 28, 2016

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Jul 27, 2016

I'm still seeing this problem on AWS EC2 (using git ffc1874), but this issue and #2872 both have been closed.

unset PYTHON_INSTALL_LAYOUT helped, but I wouldn't have found that out without trying to file a but about this :)

kornelski commented Jul 27, 2016

I'm still seeing this problem on AWS EC2 (using git ffc1874), but this issue and #2872 both have been closed.

unset PYTHON_INSTALL_LAYOUT helped, but I wouldn't have found that out without trying to file a but about this :)

@pde pde changed the title from AMI install errors to AMI zope errors related to PYTHON_INSTALL_LAYOUT Aug 1, 2016

@pde pde reopened this Aug 1, 2016

@pde

This comment has been minimized.

Show comment
Hide comment
@pde

pde Aug 1, 2016

Member

Possibly we should mitigate this by printing a warning if PYTHON_INSTALL_LAYOUT is set? Or by catching the ImportError and printing a suggestion in that case?

Member

pde commented Aug 1, 2016

Possibly we should mitigate this by printing a warning if PYTHON_INSTALL_LAYOUT is set? Or by catching the ImportError and printing a suggestion in that case?

@pombredanne

This comment has been minimized.

Show comment
Hide comment
@pombredanne

pombredanne Sep 5, 2016

IMHO this PYTHON_INSTALL_LAYOUT is a rather bad undocumented quirk made by AMZN patching without documenting the standard Python distutils. Someone should ask them to clear up their acts or document this. This is very arcane and is only on these Amazon Linux images.

pombredanne commented Sep 5, 2016

IMHO this PYTHON_INSTALL_LAYOUT is a rather bad undocumented quirk made by AMZN patching without documenting the standard Python distutils. Someone should ask them to clear up their acts or document this. This is very arcane and is only on these Amazon Linux images.

@robertclemens

This comment has been minimized.

Show comment
Hide comment
@robertclemens

robertclemens Feb 2, 2017

Can someone update me on this?
unset PYTHON_INSTALL_LAYOUT did not help. Maybe I'm missing something?

robertclemens commented Feb 2, 2017

Can someone update me on this?
unset PYTHON_INSTALL_LAYOUT did not help. Maybe I'm missing something?

@swoodford

This comment has been minimized.

Show comment
Hide comment
@swoodford

swoodford Feb 9, 2017

I was able to work around my issue by starting over with the certbot installation and renewal process following current instructions on the website from the beginning. I think the problem arose somewhere after updates to the underlying tools and since it had been so long since installing initially.
Setting or unsetting PYTHON_INSTALL_LAYOUT did not help in my case.

swoodford commented Feb 9, 2017

I was able to work around my issue by starting over with the certbot installation and renewal process following current instructions on the website from the beginning. I think the problem arose somewhere after updates to the underlying tools and since it had been so long since installing initially.
Setting or unsetting PYTHON_INSTALL_LAYOUT did not help in my case.

@mattwoods11

This comment has been minimized.

Show comment
Hide comment
@mattwoods11

mattwoods11 Apr 5, 2017

I had the same issue from what appears to be a bad install of certbot. I fixed it by running the following:

unset PYTHON_INSTALL_LAYOUT
/root/.local/share/letsencrypt/bin/pip install --upgrade certbot

mattwoods11 commented Apr 5, 2017

I had the same issue from what appears to be a bad install of certbot. I fixed it by running the following:

unset PYTHON_INSTALL_LAYOUT
/root/.local/share/letsencrypt/bin/pip install --upgrade certbot

@sakurai-youhei

This comment has been minimized.

Show comment
Hide comment
@sakurai-youhei

sakurai-youhei Jul 5, 2017

FYI: I needed 2 steps.

rm -rf /root/.local/share/letsencrypt/
PYTHON_INSTALL_LAYOUT= certbot-auto certonly --debug ....

certbot-auto --version
certbot 0.15.0

uname -a
Linux ip-172-31-22-179 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

set | grep PYTHON
PYTHON_INSTALL_LAYOUT=amzn

sakurai-youhei commented Jul 5, 2017

FYI: I needed 2 steps.

rm -rf /root/.local/share/letsencrypt/
PYTHON_INSTALL_LAYOUT= certbot-auto certonly --debug ....

certbot-auto --version
certbot 0.15.0

uname -a
Linux ip-172-31-22-179 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

set | grep PYTHON
PYTHON_INSTALL_LAYOUT=amzn

@ChristopherSchultz

This comment has been minimized.

Show comment
Hide comment
@ChristopherSchultz

ChristopherSchultz Jul 5, 2017

@sakurai-youhei This seemed to work for me. Time to add --no-self-upgrade to my crontab. :)

ChristopherSchultz commented Jul 5, 2017

@sakurai-youhei This seemed to work for me. Time to add --no-self-upgrade to my crontab. :)

@chenrui333

This comment has been minimized.

Show comment
Hide comment
@chenrui333

chenrui333 Sep 19, 2017

In my case, I took the advice from here:
rm -rf /opt/eff.org/certbot/

Then the script ./certbot-auto --version --debug works for me now (I also did the pip and virtualenv upgrades):

# ./certbot-auto --version --debug
Bootstrapping dependencies for Amazon... (you can skip this with --no-bootstrap)
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                | 2.1 kB  00:00:00
amzn-updates                                                                                                                                             | 2.3 kB  00:00:00
93 packages excluded due to repository priority protections
Package gcc-4.8.3-3.20.amzn1.noarch already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.1k-15.99.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1k-15.99.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.28.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-12.0.7-1.13.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-pip-6.1.1-1.23.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Installing Python packages...
Installation succeeded.
certbot 0.18.1

chenrui333 commented Sep 19, 2017

In my case, I took the advice from here:
rm -rf /opt/eff.org/certbot/

Then the script ./certbot-auto --version --debug works for me now (I also did the pip and virtualenv upgrades):

# ./certbot-auto --version --debug
Bootstrapping dependencies for Amazon... (you can skip this with --no-bootstrap)
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                | 2.1 kB  00:00:00
amzn-updates                                                                                                                                             | 2.3 kB  00:00:00
93 packages excluded due to repository priority protections
Package gcc-4.8.3-3.20.amzn1.noarch already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.1k-15.99.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1k-15.99.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.28.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-12.0.7-1.13.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-pip-6.1.1-1.23.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Installing Python packages...
Installation succeeded.
certbot 0.18.1
@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 10, 2017

Contributor

I'll start researching this issue tomorrow. I'll keep this thread updated with findings.

Contributor

TheNavigat commented Nov 10, 2017

I'll start researching this issue tomorrow. I'll keep this thread updated with findings.

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 11, 2017

Contributor

Using Amazon Linux official docker image, certbot-auto worked perfectly after installing all dependencies. However, PYTHON_INSTALL_LAYOUT was not set by default. Setting PYTHON_INSTALL_LAYOUT to amzn and running certbot-auto failed as mentioned above.

Still researching why this happens.

Contributor

TheNavigat commented Nov 11, 2017

Using Amazon Linux official docker image, certbot-auto worked perfectly after installing all dependencies. However, PYTHON_INSTALL_LAYOUT was not set by default. Setting PYTHON_INSTALL_LAYOUT to amzn and running certbot-auto failed as mentioned above.

Still researching why this happens.

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 13, 2017

Contributor

I set up 2 different virtual environments where in one environment I've installed the zope.interface package with PYTHON_INSTALL_LAYOUT unset, and for the second one, I've installed the package with PYTHON_INSTALL_LAYOUT set to amzn.

Using pip to install the zope.interface package using the dependency requirements defined in the certbot-auto script prints the following:

Collecting zope.component==4.2.2 (from -r /root/dependency-requirements.txt (line 1))
  Downloading zope.component-4.2.2.tar.gz (546kB)
    100% |################################| 552kB 251kB/s 
Collecting zope.event==4.1.0 (from -r /root/dependency-requirements.txt (line 3))
  Downloading zope.event-4.1.0.tar.gz (476kB)
    100% |################################| 481kB 207kB/s 
Collecting zope.interface==4.1.3 (from -r /root/dependency-requirements.txt (line 5))
  Downloading zope.interface-4.1.3.tar.gz (141kB)
    100% |################################| 143kB 210kB/s 
Requirement already satisfied: setuptools in ./testset/lib/python2.7/site-packages (from zope.component==4.2.2->-r /root/dependency-requirements.txt (line 1))
Installing collected packages: zope.interface, zope.event, zope.component
  Running setup.py install for zope.interface ... done
  Running setup.py install for zope.event ... done
  Running setup.py install for zope.component ... done
Successfully installed zope.component-4.2.2 zope.event-4.1.0 zope.interface-4.1.3

The extra zope packages are used as negative feedback.

Upon running Python on the faulty virtual environment and running the following code:

import pip
for i in pip.get_installed_distributions(local_only=True):
    print(i)

Only the following is printed:

wheel 0.30.0
setuptools 36.7.2
pip 9.0.1
zope.event 4.1.0
zope.component 4.2.2

It should be noted that using the Python version inside the virtual environment where zope.interface was installed with PYTHON_INSTALL_LAYOUT not set, the following output was printed:

zope.interface 4.1.3
zope.event 4.1.0
zope.component 4.2.2
wheel 0.30.0
setuptools 36.7.2
pip 9.0.1

Output of pip freeze:

bash-4.2# testnoset/bin/pip freeze
zope.component==4.2.2
zope.event==4.1.0
zope.interface==4.1.3

bash-4.2# testset/bin/pip freeze
zope.component==4.2.2
zope.event==4.1.0

tl;dr: For some reason when PYTHON_INSTALL_LAYOUT is set to amzn, pip says that zope.interface is installed correctly. However, it's never actually installed.

Still investigating further.

Contributor

TheNavigat commented Nov 13, 2017

I set up 2 different virtual environments where in one environment I've installed the zope.interface package with PYTHON_INSTALL_LAYOUT unset, and for the second one, I've installed the package with PYTHON_INSTALL_LAYOUT set to amzn.

Using pip to install the zope.interface package using the dependency requirements defined in the certbot-auto script prints the following:

Collecting zope.component==4.2.2 (from -r /root/dependency-requirements.txt (line 1))
  Downloading zope.component-4.2.2.tar.gz (546kB)
    100% |################################| 552kB 251kB/s 
Collecting zope.event==4.1.0 (from -r /root/dependency-requirements.txt (line 3))
  Downloading zope.event-4.1.0.tar.gz (476kB)
    100% |################################| 481kB 207kB/s 
Collecting zope.interface==4.1.3 (from -r /root/dependency-requirements.txt (line 5))
  Downloading zope.interface-4.1.3.tar.gz (141kB)
    100% |################################| 143kB 210kB/s 
Requirement already satisfied: setuptools in ./testset/lib/python2.7/site-packages (from zope.component==4.2.2->-r /root/dependency-requirements.txt (line 1))
Installing collected packages: zope.interface, zope.event, zope.component
  Running setup.py install for zope.interface ... done
  Running setup.py install for zope.event ... done
  Running setup.py install for zope.component ... done
Successfully installed zope.component-4.2.2 zope.event-4.1.0 zope.interface-4.1.3

The extra zope packages are used as negative feedback.

Upon running Python on the faulty virtual environment and running the following code:

import pip
for i in pip.get_installed_distributions(local_only=True):
    print(i)

Only the following is printed:

wheel 0.30.0
setuptools 36.7.2
pip 9.0.1
zope.event 4.1.0
zope.component 4.2.2

It should be noted that using the Python version inside the virtual environment where zope.interface was installed with PYTHON_INSTALL_LAYOUT not set, the following output was printed:

zope.interface 4.1.3
zope.event 4.1.0
zope.component 4.2.2
wheel 0.30.0
setuptools 36.7.2
pip 9.0.1

Output of pip freeze:

bash-4.2# testnoset/bin/pip freeze
zope.component==4.2.2
zope.event==4.1.0
zope.interface==4.1.3

bash-4.2# testset/bin/pip freeze
zope.component==4.2.2
zope.event==4.1.0

tl;dr: For some reason when PYTHON_INSTALL_LAYOUT is set to amzn, pip says that zope.interface is installed correctly. However, it's never actually installed.

Still investigating further.

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 13, 2017

Contributor

I ran pip in verbose mode to be able to observe what exactly happens when I try to install zope.interface. The output when PYTHON_INSTALL_LAYOUT was set to amzn was different from when it wasn't set. Here are the tailing ends of the logs, where differences can be observed.

With PYTHON_INSTALL_LAYOUT set to amzn:

    copying zope/interface/tests/test_verify.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/advisory_testing.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/m2.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/test_adapter.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    running install_egg_info
    removing '/root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7.egg-info' (and everything under it)
    Copying src/zope.interface.egg-info to /root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7.egg-info
    Installing /root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7-nspkg.pth
    running install_scripts
    writing list of installed files to '/tmp/pip-xKPwss-record/install-record.txt'
    Running setup.py install for zope.interface: finished with status 'done'
  Removing source in /tmp/pip-build-hPGjIs/zope.interface
Successfully installed zope.interface-4.1.3
Cleaning up...

With PYTHON_INSTALL_LAYOUT not set:

    copying zope/interface/tests/test_verify.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/advisory_testing.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/m2.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/test_adapter.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/interfaces.py to interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/advice.py to advice.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/declarations.py to declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/registry.py to registry.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/exceptions.py to exceptions.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/verify.py to verify.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/_compat.py to _compat.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/ro.py to ro.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/document.py to document.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/adapter.py to adapter.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/_flatten.py to _flatten.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/interface.py to interface.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/interfaces.py to interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/idatetime.py to idatetime.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/sequence.py to sequence.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/mapping.py to mapping.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/test_import_interfaces.py to test_import_interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/test_idatetime.py to test_idatetime.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/basemapping.py to basemapping.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_element.py to test_element.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_interface.py to test_interface.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/idummy.py to idummy.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_ro.py to test_ro.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_odd_declarations.py to test_odd_declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/dummy.py to dummy.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/m1.py to m1.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_sorting.py to test_sorting.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/ifoo.py to ifoo.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_advice.py to test_advice.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/ifoo_other.py to ifoo_other.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_document.py to test_document.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_registry.py to test_registry.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_exceptions.py to test_exceptions.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_interfaces.py to test_interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_declarations.py to test_declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/odd.py to odd.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_verify.py to test_verify.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/advisory_testing.py to advisory_testing.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/m2.py to m2.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_adapter.py to test_adapter.pyc
    running install_egg_info
    Copying src/zope.interface.egg-info to /root/testnoset/lib64/python2.7/site-packages/zope.interface-4.1.3-py2.7.egg-info
    Installing /root/testnoset/lib64/python2.7/site-packages/zope.interface-4.1.3-py2.7-nspkg.pth
    running install_scripts
    writing list of installed files to '/tmp/pip-8Dakj3-record/install-record.txt'
    Running setup.py install for zope.interface: finished with status 'done'
  Removing source in /tmp/pip-build-V6AHmP/zope.interface
Successfully installed zope.interface-4.1.3
Cleaning up...
Contributor

TheNavigat commented Nov 13, 2017

I ran pip in verbose mode to be able to observe what exactly happens when I try to install zope.interface. The output when PYTHON_INSTALL_LAYOUT was set to amzn was different from when it wasn't set. Here are the tailing ends of the logs, where differences can be observed.

With PYTHON_INSTALL_LAYOUT set to amzn:

    copying zope/interface/tests/test_verify.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/advisory_testing.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/m2.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    copying zope/interface/tests/test_adapter.py -> /root/testset/lib64/python2.7/dist-packages/zope/interface/tests
    running install_egg_info
    removing '/root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7.egg-info' (and everything under it)
    Copying src/zope.interface.egg-info to /root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7.egg-info
    Installing /root/testset/lib64/python2.7/dist-packages/zope.interface-4.1.3-py2.7-nspkg.pth
    running install_scripts
    writing list of installed files to '/tmp/pip-xKPwss-record/install-record.txt'
    Running setup.py install for zope.interface: finished with status 'done'
  Removing source in /tmp/pip-build-hPGjIs/zope.interface
Successfully installed zope.interface-4.1.3
Cleaning up...

With PYTHON_INSTALL_LAYOUT not set:

    copying zope/interface/tests/test_verify.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/advisory_testing.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/m2.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    copying zope/interface/tests/test_adapter.py -> /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/interfaces.py to interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/advice.py to advice.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/declarations.py to declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/registry.py to registry.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/exceptions.py to exceptions.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/verify.py to verify.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/_compat.py to _compat.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/ro.py to ro.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/document.py to document.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/adapter.py to adapter.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/_flatten.py to _flatten.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/interface.py to interface.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/interfaces.py to interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/idatetime.py to idatetime.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/sequence.py to sequence.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/mapping.py to mapping.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/test_import_interfaces.py to test_import_interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/test_idatetime.py to test_idatetime.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/common/tests/basemapping.py to basemapping.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_element.py to test_element.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_interface.py to test_interface.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/idummy.py to idummy.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_ro.py to test_ro.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_odd_declarations.py to test_odd_declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/dummy.py to dummy.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/__init__.py to __init__.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/m1.py to m1.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_sorting.py to test_sorting.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/ifoo.py to ifoo.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_advice.py to test_advice.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/ifoo_other.py to ifoo_other.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_document.py to test_document.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_registry.py to test_registry.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_exceptions.py to test_exceptions.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_interfaces.py to test_interfaces.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_declarations.py to test_declarations.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/odd.py to odd.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_verify.py to test_verify.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/advisory_testing.py to advisory_testing.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/m2.py to m2.pyc
    byte-compiling /root/testnoset/lib64/python2.7/site-packages/zope/interface/tests/test_adapter.py to test_adapter.pyc
    running install_egg_info
    Copying src/zope.interface.egg-info to /root/testnoset/lib64/python2.7/site-packages/zope.interface-4.1.3-py2.7.egg-info
    Installing /root/testnoset/lib64/python2.7/site-packages/zope.interface-4.1.3-py2.7-nspkg.pth
    running install_scripts
    writing list of installed files to '/tmp/pip-8Dakj3-record/install-record.txt'
    Running setup.py install for zope.interface: finished with status 'done'
  Removing source in /tmp/pip-build-V6AHmP/zope.interface
Successfully installed zope.interface-4.1.3
Cleaning up...
@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 16, 2017

Contributor

The behaviour persists in newer versions of the zope.interface package. However, no differences in pip installation verbose logs are observed then.

Contributor

TheNavigat commented Nov 16, 2017

The behaviour persists in newer versions of the zope.interface package. However, no differences in pip installation verbose logs are observed then.

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 16, 2017

Contributor

The directory structure of both installations differ.

Here's the structure of the correct installation (with PYTHON_INSTALL_LAYOUT not set): https://gist.github.com/TheNavigat/865685c729e3428c027cacf8a02be6b2

Here's the directory structure with PYTHON_INSTALL_LAYOUT=amzn: https://gist.github.com/TheNavigat/7485706f7d84bb1e2a132c3b0f24f6f9

Diff: https://gist.github.com/TheNavigat/d0e1dd065495e043d8af52b69f8be45a

Contributor

TheNavigat commented Nov 16, 2017

The directory structure of both installations differ.

Here's the structure of the correct installation (with PYTHON_INSTALL_LAYOUT not set): https://gist.github.com/TheNavigat/865685c729e3428c027cacf8a02be6b2

Here's the directory structure with PYTHON_INSTALL_LAYOUT=amzn: https://gist.github.com/TheNavigat/7485706f7d84bb1e2a132c3b0f24f6f9

Diff: https://gist.github.com/TheNavigat/d0e1dd065495e043d8af52b69f8be45a

@bradsolves

This comment has been minimized.

Show comment
Hide comment
@bradsolves

bradsolves Nov 18, 2017

@TheNavigat Thank you for picking up this issue and taking it forward. Following the commands suggested by @chenrui333 (among others), I am receiving the following:

Requesting to rerun ./certbot-auto with root privileges...
Bootstrapping dependencies for Amazon... (you can skip this with --no-bootstrap)
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
Package gcc-4.8.5-1.22.amzn1.noarch already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.2k-8.105.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.2k-8.105.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.28.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-15.1.0-1.14.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-pip-9.0.1-1.24.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Traceback (most recent call last):
File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 7, in
from certbot.main import main
File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/certbot/main.py", line 9, in
import zope.component
File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/zope/component/init.py", line 16, in
from zope.interface import Interface
ImportError: No module named interface

not sure if this helps, but I wanted to add to the log. I am happy to try other commands and provide the output as well.

Thank you in advance.

UPDATE: about two minutes after posting, I found this post: #5100 (comment) and issued the following commands:

$ unset PYTHON_INSTALL_LAYOUT
$ sudo /opt/eff.org/certbot/venv/local/bin/pip install --upgrade certbot
$ /opt/eff.org/certbot/venv/local/bin/certbot --help

and the errors have cleared allowing me to successfully renew 😄 🎉 🌮 😄

bradsolves commented Nov 18, 2017

@TheNavigat Thank you for picking up this issue and taking it forward. Following the commands suggested by @chenrui333 (among others), I am receiving the following:

Requesting to rerun ./certbot-auto with root privileges...
Bootstrapping dependencies for Amazon... (you can skip this with --no-bootstrap)
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
Package gcc-4.8.5-1.22.amzn1.noarch already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.2k-8.105.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.2k-8.105.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.28.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-15.1.0-1.14.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.121.amzn1.x86_64 already installed and latest version
Package python27-pip-9.0.1-1.24.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Traceback (most recent call last):
File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 7, in
from certbot.main import main
File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/certbot/main.py", line 9, in
import zope.component
File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/zope/component/init.py", line 16, in
from zope.interface import Interface
ImportError: No module named interface

not sure if this helps, but I wanted to add to the log. I am happy to try other commands and provide the output as well.

Thank you in advance.

UPDATE: about two minutes after posting, I found this post: #5100 (comment) and issued the following commands:

$ unset PYTHON_INSTALL_LAYOUT
$ sudo /opt/eff.org/certbot/venv/local/bin/pip install --upgrade certbot
$ /opt/eff.org/certbot/venv/local/bin/certbot --help

and the errors have cleared allowing me to successfully renew 😄 🎉 🌮 😄

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Nov 19, 2017

Contributor

@bradsolves Unfortunately this doesn't add much. Unsetting PYTHON_INSTALL_LAYOUT fixes the problem, but what I'm investigating is why this happens. The way pip's behaviour changes because of the variable isn't clear. If the solution is to indeed unset the variable, we need to understand why, and whether it breaks anything else.

Contributor

TheNavigat commented Nov 19, 2017

@bradsolves Unfortunately this doesn't add much. Unsetting PYTHON_INSTALL_LAYOUT fixes the problem, but what I'm investigating is why this happens. The way pip's behaviour changes because of the variable isn't clear. If the solution is to indeed unset the variable, we need to understand why, and whether it breaks anything else.

@tseaver

This comment has been minimized.

Show comment
Hide comment
@tseaver

tseaver Dec 3, 2017

I don't know from Amazon Linux, but am a maintainer for zope.interface. Is it possible that configuring the amzn layout sets up Python such that an older / incompatible version of zope.interface is already installed?

One way to test this hypothesis would be to forcibly upgrade it, e.g. $ pip install -U zope.interface.

tseaver commented Dec 3, 2017

I don't know from Amazon Linux, but am a maintainer for zope.interface. Is it possible that configuring the amzn layout sets up Python such that an older / incompatible version of zope.interface is already installed?

One way to test this hypothesis would be to forcibly upgrade it, e.g. $ pip install -U zope.interface.

@TheNavigat

This comment has been minimized.

Show comment
Hide comment
@TheNavigat

TheNavigat Dec 3, 2017

Contributor

@tseaver I used this file, https://github.com/certbot/certbot/blob/master/letsencrypt-auto-source/pieces/dependency-requirements.txt, to install zope.interface. The file lists the specific version and hash. It resulted in the error above when PYTHON_INSTALL_LAYOUT was set to amzn. When the exact same package (pip install -r dependency-requirements.txt - I modifed the file to only include the zope.interface package) was installed with PYTHON_INSTALL_LAYOUT unset, it worked.

I don't think the version is the problem.

Contributor

TheNavigat commented Dec 3, 2017

@tseaver I used this file, https://github.com/certbot/certbot/blob/master/letsencrypt-auto-source/pieces/dependency-requirements.txt, to install zope.interface. The file lists the specific version and hash. It resulted in the error above when PYTHON_INSTALL_LAYOUT was set to amzn. When the exact same package (pip install -r dependency-requirements.txt - I modifed the file to only include the zope.interface package) was installed with PYTHON_INSTALL_LAYOUT unset, it worked.

I don't think the version is the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment