pip installs headers in virtualenv to a directory not included by default in build_ext

[segevfiner]

• Pip version: 9.0.1
• Python version: 2.7.13
• Operating system: Windows 10.0.15063 x64

Description:

When installing a package in a virtualenv, it's headers get installed to a different directory than plain distutils, which is not included by default in the build_ext include directories. Such headers won't be found by packages that depend on it unless the package takes special care to calculate the directory used by pip and adding it manually to it's Extensions in setup.py.

For example, on Windows:

• Plain distutils will install to: <prefix>/Include/<dist_name>
• setuptools in egg mode won't install them at all! easy_install (bdist_egg) doesn't install headers setuptools#1079 😛
• pip will install to: <prefix>/Include/site/python2.7/<dist_name>

What I've run:

# setup.py
from setuptools import setup

setup(
    name="spam",
    version="0.1.0",
    headers=["include/spam/spam.h"],
)

Then:

virtualenv venv
venv\Scripts\activate
pip install .

Workaround

import sys
import os
import sysconfig
from setuptools import setup, Extension

include_dirs = []

venv_include_dir = os.path.join(sys.prefix, 'include', 'site', 'python' + sysconfig.get_python_version())
if os.path.exists(venv_include_dir):
    include_dirs.append(venv_include_dir)

setup(
    name="ham",
    version="0.1.0",
    ext_modules=[
        Extension("ham", ["ham.c"], include_dirs=include_dirs),
    ],
)

[pradyunsg]

@pfmoore This is an compilation on Windows issue. I think you'd have more idea about this than me here.

[pfmoore]

The details of how Python sets and uses the sysconfig include directory are not something I know much about. You'd probably need one of the scientific Python people (who have plenty of battle scars from dealing with this stuff) to get a good answer here.

Some random thoughts:

1. pip install . is doing a source install, so it'll be setuptools that chooses where to install the headers, not pip.
2. For wheels, move_wheel_files puts the distribution-1.0.data/headers directory into the location defined by scheme["headers"]. This should come from sysconfig.
3. setuptools.Extension determines where to tell the compiler to look for headers, and it should be working that out from the scheme.

So, my suspicion is that any inconsistency is a result of the various components not agreeing on what scheme should be used, and/or the whole sysconfig scheme mechanism not being well defined for virtual environments (and the tools differing in how they choose to extend the mechanism to handle virtual environments).

So, in summary, the root of this is probably a limitation of the sysconfig mechanisms not covering virtual environments. That's a core Python issue ultimately. At the moment, there's an inconsistency in how the various tools extend the schemes to virtual environments, and we should probably make that more consistent (and standardise it maybe). Given that changing core Python is a long-term thing, agreeing a standard for tools to follow is a better short term approach (and the stdlib can catch up later).

But once again, I have little actual experience in this area - if the numeric guys say something different, they are right and I'm wrong 😄

[chrahunt]

Related to pypa/packaging-problems#84.

Is there anything directly actionable for pip here or would it be better to continue in a discuss.python.org topic or the issue above?

[chrahunt]

One strategy, as discussed in #7455 and here, is to inspect the installed packages/sys.path and determine the paths from that in your setup.py. I'll close this issue with the same conclusion. As mentioned over there, don't hesitate to post if you come up with something!