Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cant find Python.h inside a virtualenv #1739

Closed
tzulberti opened this issue Mar 19, 2020 · 8 comments
Closed

Cant find Python.h inside a virtualenv #1739

tzulberti opened this issue Mar 19, 2020 · 8 comments
Labels
bug

Comments

@tzulberti
Copy link
Contributor

@tzulberti tzulberti commented Mar 19, 2020

Issue

I try to install a package for python2 inside a virtualenv and the compilation fails mentioning that it can't find Python.h.

But, if I install the same package at the system level it can find that issue

I tested this with the lastest virutalenv release (20.0.12)

Environment

You can tests the build using this Dockerfile

FROM amazonlinux:1

RUN yum -y install python27 python27-devel python36 python36-devel python27-pip git gcc

RUN pip-3.6 install --upgrade virtualenv

RUN cd /root && virtualenv --python python2.7 /tmp/bar

RUN source /tmp/bar/bin/activate \
    && pip install cython numpy \
    && pip install cython-bbox

And the error when building that dockerfile is:

    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/bar/lib64/python2.7/site-packages/numpy/core/include -I/tmp/bar/include/python2.7 -c src/cython_bbox.c -o build/temp.linux-x86_64-2.7/src/cython_bbox.o -Wno-cpp
    src/cython_bbox.c:26:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

70 setup logging to NOTSET [DEBUG report:43]
71 created app data folder /root/.local/share/virtualenv [DEBUG app_data:52]
78 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:44]
79 proposed PythonInfo(spec=CPython3.6.10.final.0-64, exe=/usr/bin/python3.6, platform=linux, version='3.6.10 (default, Feb 10 2020, 19:55:14) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ascii-ANSI_X3.4-1968) [INFO builtin:50]
79 discover PATH[0]=/usr/local/sbin [DEBUG builtin:84]
79 filesystem is case-sensitive [DEBUG info:28]
80 discover PATH[1]=/usr/local/bin [DEBUG builtin:84]
80 discover PATH[2]=/usr/sbin [DEBUG builtin:84]
81 discover PATH[3]=/usr/bin [DEBUG builtin:84]
81 Attempting to acquire lock 139953546058720 on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
81 Lock 139953546058720 acquired on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
81 get interpreter info via cmd: /usr/bin/python2.7 /usr/local/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:108]
124 write PythonInfo to /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:94]
125 Attempting to release lock 139953546058720 on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
125 Lock 139953546058720 released on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
125 proposed PathPythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ANSI_X3.4-1968-None) [INFO builtin:50]
125 accepted PathPythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ANSI_X3.4-1968-None) [DEBUG builtin:52]
204 create virtual environment via CPython2Posix(dest=/tmp/bar, clear=False, global=False) [INFO session:52]
204 no include folders as can't find include marker /usr/local/include/python2.7/Python.h [DEBUG cpython2:53]
205 create folder /tmp/bar/bin [DEBUG _sync:21]
205 create folder /tmp/bar/lib/python2.7/site-packages [DEBUG _sync:21]
205 create folder /tmp/bar/lib64/python2.7/site-packages [DEBUG _sync:21]
206 write /tmp/bar/pyvenv.cfg [DEBUG pyenv_cfg:34]
206 	home = /usr [DEBUG pyenv_cfg:38]
206 	implementation = CPython [DEBUG pyenv_cfg:38]
206 	version_info = 2.7.16.final.0 [DEBUG pyenv_cfg:38]
206 	virtualenv = 20.0.12 [DEBUG pyenv_cfg:38]
206 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
206 	base-prefix = /usr [DEBUG pyenv_cfg:38]
206 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
206 	base-executable = /usr/bin/python2.7 [DEBUG pyenv_cfg:38]
206 copy /usr/bin/python2.7 to /tmp/bar/bin/python [DEBUG _sync:48]
207 symlink /usr/lib64/python2.7/os.py to /tmp/bar/lib64/python2.7/os.py [DEBUG _sync:40]
207 symlink /usr/lib64/python2.7/os.pyc to /tmp/bar/lib64/python2.7/os.pyc [DEBUG _sync:40]
207 symlink directory /usr/lib64/python2.7/lib-dynload to /tmp/bar/lib64/python2.7/lib-dynload [DEBUG _sync:40]
208 create /tmp/bar/lib/python2.7/site-packages/_virtualenv.py [DEBUG api:78]
208 create virtualenv import hook file /tmp/bar/lib/python2.7/site-packages/_virtualenv.pth [DEBUG api:81]
208 ============================== target debug ============================== [DEBUG session:54]
209 debug via /tmp/bar/bin/python /usr/local/lib/python3.6/site-packages/virtualenv/create/debug.py [DEBUG creator:187]
208 {
  "sys": {
    "executable": "/tmp/bar/bin/python",
    "_base_executable": null,
    "prefix": "/tmp/bar",
    "base_prefix": "/usr",
    "real_prefix": "/usr",
    "exec_prefix": "/tmp/bar",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python27.zip",
      "/usr/lib64/python2.7",
      "/usr/lib64/python2.7/plat-linux2",
      "/usr/lib64/python2.7/lib-tk",
      "/usr/lib64/python2.7/lib-old",
      "/usr/lib64/python2.7/lib-dynload",
      "/tmp/bar/lib64/python2.7/site-packages",
      "/tmp/bar/lib/python2.7/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._VirtualenvImporter'>"
    ],
    "fs_encoding": "ANSI_X3.4-1968",
    "io_encoding": null
  },
  "version": "2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]",
  "os": "<module 'os' from '/usr/lib64/python2.7/os.pyc'>",
  "site": "<module 'site' from '/usr/lib64/python2.7/site.pyc'>",
  "datetime": "<module 'datetime' from '/usr/lib64/python2.7/lib-dynload/datetime.so'>",
  "math": "<module 'math' from '/usr/lib64/python2.7/lib-dynload/math.so'>",
  "json": "<module 'json' from '/usr/lib64/python2.7/json/__init__.pyc'>"
} [DEBUG session:55]
233 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1) [INFO session:59]
234 Attempting to acquire lock 139953453730056 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [DEBUG filelock:270]
234 Lock 139953453730056 acquired on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [INFO filelock:274]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/pip-20.0.2-py2.py3-none-any.whl [DEBUG acquire:51]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/setuptools-44.0.0-py2.py3-none-any.whl [DEBUG acquire:51]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/wheel-0.34.2-py2.py3-none-any.whl [DEBUG acquire:51]
238 install wheel from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 install setuptools from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any of wheel-0.34.2-py2.py3-none-any.whl [DEBUG base:57]
239 install pip from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any of setuptools-44.0.0-py2.py3-none-any.whl [DEBUG base:57]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any of pip-20.0.2-py2.py3-none-any.whl [DEBUG base:57]
262 changing mode of /tmp/tmph18mndm1/wheel-2.7 to 755 [INFO util:566]
263 changing mode of /tmp/tmph18mndm1/wheel to 755 [INFO util:566]
263 changing mode of /tmp/tmph18mndm1/wheel2 to 755 [INFO util:566]
269 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /tmp/bar/lib/python2.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:48]
274 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /tmp/bar/lib/python2.7/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:48]
275 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /tmp/bar/lib/python2.7/site-packages/wheel [DEBUG _sync:48]
288 changing mode of /tmp/bar/bin/wheel-2.7 to 755 [INFO util:566]
289 changing mode of /tmp/bar/bin/wheel to 755 [INFO util:566]
289 changing mode of /tmp/bar/bin/wheel2 to 755 [INFO util:566]
289 generated console scripts wheel2 wheel-2.7 wheel [DEBUG base:53]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install-2.7 to 755 [INFO util:566]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install to 755 [INFO util:566]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install2 to 755 [INFO util:566]
324 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.virtualenv to /tmp/bar/lib/python2.7/site-packages/setuptools-44.0.0.virtualenv [DEBUG _sync:48]
325 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /tmp/bar/lib/python2.7/site-packages/setuptools [DEBUG _sync:48]
348 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /tmp/bar/lib/python2.7/site-packages/pkg_resources [DEBUG _sync:48]
352 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /tmp/bar/lib/python2.7/site-packages/easy_install.py [DEBUG _sync:48]
353 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /tmp/bar/lib/python2.7/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:48]
355 changing mode of /tmp/bar/bin/easy_install-2.7 to 755 [INFO util:566]
356 changing mode of /tmp/bar/bin/easy_install to 755 [INFO util:566]
356 changing mode of /tmp/bar/bin/easy_install2 to 755 [INFO util:566]
356 generated console scripts easy_install2 easy_install easy_install-2.7 [DEBUG base:53]
389 changing mode of /tmp/tmp1447pgjv/pip to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip2 to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip-2.7 to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip2.7 to 755 [INFO util:566]
415 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.virtualenv to /tmp/bar/lib/python2.7/site-packages/pip-20.0.2.virtualenv [DEBUG _sync:48]
416 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /tmp/bar/lib/python2.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:48]
417 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /tmp/bar/lib/python2.7/site-packages/pip [DEBUG _sync:48]
470 changing mode of /tmp/bar/bin/pip to 755 [INFO util:566]
470 changing mode of /tmp/bar/bin/pip2 to 755 [INFO util:566]
470 changing mode of /tmp/bar/bin/pip-2.7 to 755 [INFO util:566]
471 changing mode of /tmp/bar/bin/pip2.7 to 755 [INFO util:566]
471 generated console scripts pip pip2 pip2.7 pip-2.7 [DEBUG base:53]
471 Attempting to release lock 139953453730056 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [DEBUG filelock:315]
471 Lock 139953453730056 released on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [INFO filelock:318]
471 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:65]
473 write /tmp/bar/pyvenv.cfg [DEBUG pyenv_cfg:34]
473 	home = /usr [DEBUG pyenv_cfg:38]
473 	implementation = CPython [DEBUG pyenv_cfg:38]
473 	version_info = 2.7.16.final.0 [DEBUG pyenv_cfg:38]
473 	virtualenv = 20.0.12 [DEBUG pyenv_cfg:38]
473 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
473 	base-prefix = /usr [DEBUG pyenv_cfg:38]
473 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
473 	base-executable = /usr/bin/python2.7 [DEBUG pyenv_cfg:38]
473 created virtual environment CPython2.7.16.final.0-64 in 404ms
  creator CPython2Posix(dest=/tmp/bar, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

Thank you for the amazing work

@tzulberti tzulberti added the bug label Mar 19, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Mar 19, 2020

Here's the relevant error:

204 no include folders as can't find include marker /usr/local/include/python2.7/Python.h [DEBUG cpython2:53]

Where are the include headers stored? Would be helpful to also see /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json that specifies the include header location for /usr/bin/python2.7

@tzulberti

This comment has been minimized.

Copy link
Contributor Author

@tzulberti tzulberti commented Mar 19, 2020

The headers are stored at:

$ locate Python.h
/usr/include/python2.7/Python.h
/usr/include/python3.6m/Python.h

and the output of the JSON file is:

{
  "st_mtime": 1581360923.0,
  "path": "/usr/bin/python2.7",
  "content": {
    "base_exec_prefix": null,
    "system_stdlib": "/usr/lib/python2.7",
    "has_venv": false,
    "prefix": "/usr",
    "stdout_encoding": null,
    "executable": "/usr/bin/python2.7",
    "implementation": "CPython",
    "exec_prefix": "/usr",
    "platform": "linux2",
    "version": "2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]",
    "sysconfig_paths": {
      "platstdlib": "{platbase}/lib64/python{py_version_short}",
      "platlib": "{platbase}/local/lib64/python{py_version_short}/site-packages",
      "purelib": "{base}/local/lib/python{py_version_short}/site-packages",
      "stdlib": "{base}/lib/python{py_version_short}",
      "scripts": "{base}/local/bin",
      "include": "{base}/local/include/python{py_version_short}",
      "data": "{base}/local"
    },
    "base_prefix": null,
    "system_stdlib_platform": "/usr/lib64/python2.7",
    "_creators": null,
    "file_system_encoding": "ANSI_X3.4-1968",
    "version_info": {
      "major": 2,
      "minor": 7,
      "micro": 16,
      "releaselevel": "final",
      "serial": 0
    },
    "sysconfig_vars": {
      "base": "/usr",
      "platbase": "/usr",
      "PYTHONFRAMEWORK": "",
      "py_version_short": "2.7"
    },
    "path": [
      "/usr/local/lib/python3.6/site-packages/virtualenv/discovery",
      "/usr/lib/python27.zip",
      "/usr/lib64/python2.7",
      "/usr/lib64/python2.7/plat-linux2",
      "/usr/lib64/python2.7/lib-tk",
      "/usr/lib64/python2.7/lib-old",
      "/usr/lib64/python2.7/lib-dynload",
      "/usr/local/lib64/python2.7/site-packages",
      "/usr/local/lib/python2.7/site-packages",
      "/usr/lib64/python2.7/site-packages",
      "/usr/lib/python2.7/site-packages",
      "/usr/lib64/python2.7/dist-packages",
      "/usr/lib/python2.7/dist-packages"
    ],
    "max_size": 9223372036854775807,
    "real_prefix": null,
    "distutils_install": {
      "purelib": "lib/python2.7/site-packages",
      "headers": "include/python2.7/UNKNOWN",
      "platlib": "lib64/python2.7/site-packages",
      "data": "",
      "scripts": "bin"
    },
    "architecture": 64,
    "original_executable": "/usr/bin/python2.7",
    "os": "posix",
    "system_executable": "/usr/bin/python2.7"
  }
}
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Mar 19, 2020

Then that python is just broken 😢 The interpreter tells us that it should be inside

    "sysconfig_paths": {
      "include": "{base}/local/include/python{py_version_short}",
    },
    "sysconfig_vars": {
      "base": "/usr",
      "py_version_short": "2.7"
    },

Which resolves to /usr/local/include/python2.7 🤔 Why install them then under /usr/include/python2.7? Who packaged this Python 🤔

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Mar 19, 2020

We can fall back to the distuils path to resolve this but I'd recommend opening an issue for whoever packages that Python 👍 because they're doing something wrong here 👍

@tzulberti

This comment has been minimized.

Copy link
Contributor Author

@tzulberti tzulberti commented Mar 19, 2020

This source of this RPM package can be found at:

http://packages.us-east-1.amazonaws.com/2018.03/updates/eb308d7d16f1/x86_64/Packages/python27-devel-2.7.16-1.131.amzn1.x86_64.rpm

On Centos8 the Python.h is located where you expected:

[root@5ed8915be4f6 /]# locate Python.h
/usr/include/python3.6m/Python.h

And an original Centos 6, the files are also as expected:

[root@141d31fe5621 /]# locate Python.h
/usr/include/python2.6/Python.h

But I wasn't able to find Python27 package for Centos 6 or Centos8

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Mar 19, 2020

@tzulberti where you expected is actually where the following things says:

"sysconfig_paths": {
      "include": "{base}/local/include/python{py_version_short}",
    },

So this might not necessarily be /usr/include/pythonx.y. You can't tell where you expected without also posting the outputs of those values.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Mar 19, 2020

@tzulberti

This comment has been minimized.

Copy link
Contributor Author

@tzulberti tzulberti commented Mar 19, 2020

Thank you for solving the issue so fast.

I tested it using the Dockerfile in this ticket and it solved the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.