Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install docker-compose==1.25 fails due to lack of python headers for building subprocess32 #7030

Closed
kostrzewa9ld opened this issue Nov 19, 2019 · 7 comments
Labels

Comments

@kostrzewa9ld
Copy link

@kostrzewa9ld kostrzewa9ld commented Nov 19, 2019

Description of the issue

When running pip install docker-compose==1.25 either in virtual env or in centos:7 docker container pip tries to compile subprocess32 and requires python headers.

Context information (for bug reports)

Output of docker-compose version

1.25

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

(paste here)

Steps to reproduce the issue

  1. virtualenv /tmp/foo
  2. source /tmp/foo/bin/activate
  3. pip install docker-compose==1.25

Observed result

Installation fails due to lack of python dev files.

Expected result

Installation succeeds and python dev files are not required.

Stacktrace / full error message

Running setup.py install for subprocess32 ... error
    ERROR: Command errored out with exit status 1:
     command: /tmp/foo/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-YMsrMi/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/foo/include/site/python2.7/subprocess32
         cwd: /tmp/pip-install-OtSipE/subprocess32/
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying subprocess32.py -> build/lib.linux-x86_64-2.7
    running build_ext
    running build_configure
     _posixsubprocess_config.h is already up to date.
    building '_posixsubprocess32' extension
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-PPrPZj/python2.7-2.7.15=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
    _posixsubprocess.c:16:10: fatal error: Python.h: No such file or directory
     #include "Python.h"
              ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /tmp/foo/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-YMsrMi/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/foo/include/site/python2.7/subprocess32 Check the logs for full command output.

Additional information

OS version / distribution, docker-compose install method, etc.
CentOS 7 docker, Ubuntu 18.04.3 LTS

@plemelin

This comment has been minimized.

Copy link

@plemelin plemelin commented Nov 19, 2019

I face the issue mentionned by @kostrzewa9ld and installed python-devel with yum. This allowed the installation of docker-compose to proceed. However, docker-compose will now fail with:

Traceback (most recent call last):
  File \"/usr/bin/docker-compose\", line 7, in <module>
    from compose.cli.main import main
  File \"/usr/lib/python2.7/site-packages/compose/cli/main.py\", line 23, in <module>
    from ..bundle import get_image_digests
  File \"/usr/lib/python2.7/site-packages/compose/bundle.py\", line 12, in <module>
    from .config.serialize import denormalize_config
  File \"/usr/lib/python2.7/site-packages/compose/config/__init__.py\", line 6, in <module>
    from .config import ConfigurationError
  File \"/usr/lib/python2.7/site-packages/compose/config/config.py\", line 50, in <module>
    from .validation import match_named_volumes
  File \"/usr/lib/python2.7/site-packages/compose/config/validation.py\", line 12, in <module>
    from jsonschema import Draft4Validator
  File \"/usr/lib/python2.7/site-packages/jsonschema/__init__.py\", line 33, in <module>
    import importlib_metadata as metadata
  File \"/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py\", line 9, in <module>
    import zipp
  File \"/usr/lib/python2.7/site-packages/zipp.py\", line 12, in <module>
    import more_itertools
  File \"/usr/lib/python2.7/site-packages/more_itertools/__init__.py\", line 1, in <module>
    from more_itertools.more import *  # noqa
  File \"/usr/lib/python2.7/site-packages/more_itertools/more.py\", line 340
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax"
@ulyssessouza

This comment has been minimized.

Copy link
Collaborator

@ulyssessouza ulyssessouza commented Nov 19, 2019

@plemelin Could you please provide more info? That would be great to open another bug report answering the template questions.

@plemelin

This comment has been minimized.

Copy link

@plemelin plemelin commented Nov 19, 2019

@ulyssessouza

As I was writing how to reproduce the issue to you, it gave me the opportunity to rubber duck... The issue I'm facing (and many people will be facing at this point) is that the default version of pip installed by python-pip is version 8 (current version is 19...). Here is the bug report that woke me up: erikrose/more-itertools#272

Here how to reproduce it easily (running this from my arch linux + docker machine):

sudo docker pull centos:centos7
sudo docker run --rm -it centos:centos7 /bin/bash

yum update -y
yum install -y epel-release
yum install -y python-pip

pip install docker-compose
# BAM. Fails with needing gcc

yum install -y gcc
pip install docker-compose
# BAM. Fails with needing Python.h

yum install -y python-devel
pip install docker-compose

docker-compose --version
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 7, in <module>
    from compose.cli.main import main
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 23, in <module>
    from ..bundle import get_image_digests
  File "/usr/lib/python2.7/site-packages/compose/bundle.py", line 12, in <module>
    from .config.serialize import denormalize_config
  File "/usr/lib/python2.7/site-packages/compose/config/__init__.py", line 6, in <module>
    from .config import ConfigurationError
  File "/usr/lib/python2.7/site-packages/compose/config/config.py", line 50, in <module>
    from .validation import match_named_volumes
  File "/usr/lib/python2.7/site-packages/compose/config/validation.py", line 12, in <module>
    from jsonschema import Draft4Validator
  File "/usr/lib/python2.7/site-packages/jsonschema/__init__.py", line 33, in <module>
    import importlib_metadata as metadata
  File "/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py", line 9, in <module>
    import zipp
  File "/usr/lib/python2.7/site-packages/zipp.py", line 12, in <module>
    import more_itertools
  File "/usr/lib/python2.7/site-packages/more_itertools/__init__.py", line 1, in <module>
    from more_itertools.more import *  # noqa
  File "/usr/lib/python2.7/site-packages/more_itertools/more.py", line 340
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

Now, this lead me here: erikrose/more-itertools#272
Which then let me fix the installation like so:

sudo docker pull centos:centos7
sudo docker run --rm -it centos:centos7 /bin/bash

yum update -y
yum install -y epel-release
yum install -y python-pip python-devel gcc
pip install --upgrade pip

pip install docker-compose
docker-compose --version
docker-compose version 1.25.0, build b42d419
@ulyssessouza

This comment has been minimized.

Copy link
Collaborator

@ulyssessouza ulyssessouza commented Nov 19, 2019

@kostrzewa9ld Note that python-dev is mentioned in https://docs.docker.com/compose/install/ as dependency when installing via pip. Also note that it's an alternative installation.

@kostrzewa9ld

This comment has been minimized.

Copy link
Author

@kostrzewa9ld kostrzewa9ld commented Nov 19, 2019

@ulyssessouza
Just for the record as I guess it will be resolved by update of documentation, but as far as I can tell python-dev is only mentioned as a required dependency for alpine

For alpine, the following dependency packages are needed: py-pip, python-dev, libffi-dev, openssl-dev, gcc, libc-dev, and make.

What's more it only started to be required for non-alpine distros since 1.25, which is not even mentioned in this document:

The instructions below outline installation of the current stable release (v1.24.1) of Compose.

@jamshid

This comment has been minimized.

Copy link

@jamshid jamshid commented Nov 22, 2019

I guess docker doesn't care because they're dropping python 2 support?
#6890

Or is Compose going away with Swarm?

FWIW pip3 seems to work fine:

yum install -y python3-pip
pip3 install docker-compose 
@nandogameiro

This comment has been minimized.

Copy link

@nandogameiro nandogameiro commented Nov 27, 2019

This command solved the issue for me
sudo pip install 'more-itertools>=4.0.0,<=5.0.0' --force-reinstall

On this comment below user says more-itertools version >6.0 dropped support to python 2, so that's why need to downgrade to version 5.0.0

erikrose/more-itertools#272 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.