get_include returns incorrect directory for custom prefixes

[knedlsepp]

Issue description

Custom installations of python, which don't install python modules to the usual single global site-packages directory, but rather use a separate directory for every python-module (making use of PYTHONPATH to glue everything together) don't work with the current installation method.
This affects me, because I want to package pybind11 for NixOS, which uses this exact concept. Users of HPC installations, which use module load-type systems should also be affected.
This leads to the following error when trying to compile the python_example

fatal error: pybind11/pybind11.h: No such file or directory
 #include <pybind11/pybind11.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Reproducible example code:

While it is pretty specific to the toolchain I'm using, I still want to include the Code that produces my error.

let
  pinnedPkgs = (import <nixpkgs> {}).fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "eb7e0d42dbe44a452dcbf9afea979ac9591c2fd1";
    sha256 = "1nvz1mcmplb9m7bck9ga1qwz77y4b5sd0pfrb06vrda4p0qzc65v";
  };
  pkgs = import pinnedPkgs { };
  pybind11 = pkgs.python.pkgs.buildPythonPackage rec {
    pname = "pybind11";
    version = "2.2.3";
    src = pkgs.fetchurl {
      url = "https://github.com/pybind/pybind11/archive/v${version}.tar.gz";
      sha256 = "1sj0x4fwsbnwdai5sxpw1l1vh8m5hpbkfk3zanxcbcgs39jpnfrs";
    };
    buildInputs = with pkgs; [
      cmake
    ];
  };
in pkgs.python.pkgs.buildPythonPackage rec {
  name = "pybind11_python_example";
  src = pkgs.fetchFromGitHub {
    owner = "pybind";
    repo = "python_example";
    rev = "2ed5a68759cd6ff5d2e5992a91f08616ef457b5c";
    sha256 = "1lpslhag09gw6g891wxznpwmjx4giqmr3c2gjxzrcbh1lhilbcr2";
  };
  preBuild = ''
    export HOME=$(pwd); # Fixes docs trying to be installed to $HOME
  '';
  propagatedBuildInputs = [
    pybind11
  ];
}

Invoking nix-build outputs the following error:

these derivations will be built:
  /nix/store/ricnvdhvsp5anx1rwy0nmbkh8s0h1cxp-python2.7-pybind11_python_example.drv
building '/nix/store/ricnvdhvsp5anx1rwy0nmbkh8s0h1cxp-python2.7-pybind11_python_example.drv'...
unpacking sources
unpacking source archive /nix/store/raiqx4ix1501k6wxm25i0zyq7mhahy3c-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/test.py
patching sources
configuring
building
running bdist_wheel
running build
running build_ext
creating tmp
creating tmp/nix-build-python2.7-pybind11_python_example.drv-1
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/nix/store/6zmk1ankwbnydn447vnv8m3rqx8yjm6l-python-2.7.15/include/python2.7 -c /tmp/nix-build-python2.7-pybind11_python_example.drv-1/tmpJ6Pmow.cpp -o tmp/nix-build-python2.7-pybind11_python_example.drv-1/tmpJ6Pmow.o -std=c++14
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/nix/store/6zmk1ankwbnydn447vnv8m3rqx8yjm6l-python-2.7.15/include/python2.7 -c /tmp/nix-build-python2.7-pybind11_python_example.drv-1/tmpzDv7i5.cpp -o tmp/nix-build-python2.7-pybind11_python_example.drv-1/tmpzDv7i5.o -fvisibility=hidden
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
building 'python_example' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/nix/store/6zmk1ankwbnydn447vnv8m3rqx8yjm6l-python-2.7.15/include/python2.7 -I/tmp/nix-build-python2.7-pybind11_python_example.drv-1/source/.local/include/python2.7 -I/nix/store/6zmk1ankwbnydn447vnv8m3rqx8yjm6l-python-2.7.15/include/python2.7 -c src/main.cpp -o build/temp.linux-x86_64-2.7/src/main.o -DVERSION_INFO="0.0.1" -std=c++14 -fvisibility=hidden
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
src/main.cpp:1:10: fatal error: pybind11/pybind11.h: No such file or directory
 #include <pybind11/pybind11.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
builder for '/nix/store/ricnvdhvsp5anx1rwy0nmbkh8s0h1cxp-python2.7-pybind11_python_example.drv' failed with exit code 1
error: build of '/nix/store/ricnvdhvsp5anx1rwy0nmbkh8s0h1cxp-python2.7-pybind11_python_example.drv' failed

Diagnosis

The function get_include yields the location of the CPython headers:

$ python -c 'import pybind11; print(pybind11.get_include())'
/nix/store/nx3jw576gqw01iiijgsav39w2qa4cni2-python-2.7.14/include/python2.7
$ ls $(!!)
ls $(python -c 'import pybind11; print(pybind11.get_include())')
abstract.h         descrobject.h  longobject.h    pyctype.h       rangeobject.h
asdl.h             dictobject.h   marshal.h       py_curses.h     setobject.h
ast.h              dtoa.h         memoryobject.h  pydebug.h       sliceobject.h
bitset.h           enumobject.h   metagrammar.h   pyerrors.h      stringobject.h
boolobject.h       errcode.h      methodobject.h  pyexpat.h       structmember.h
bufferobject.h     eval.h         modsupport.h    pyfpe.h         structseq.h
bytearrayobject.h  fileobject.h   moduleobject.h  pygetopt.h      symtable.h
bytes_methods.h    floatobject.h  node.h          pymacconfig.h   sysmodule.h
bytesobject.h      frameobject.h  object.h        pymactoolbox.h  timefuncs.h
cellobject.h       funcobject.h   objimpl.h       pymath.h        token.h
ceval.h            genobject.h    opcode.h        pymem.h         traceback.h
classobject.h      graminit.h     osdefs.h        pyport.h        tupleobject.h
cobject.h          grammar.h      parsetok.h      pystate.h       ucnhash.h
codecs.h           import.h       patchlevel.h    pystrcmp.h      unicodeobject.h
code.h             intobject.h    pgen.h          pystrtod.h      warnings.h
compile.h          intrcheck.h    pgenheaders.h   Python-ast.h    weakrefobject.h
complexobject.h    iterobject.h   pyarena.h       Python.h
cStringIO.h        listobject.h   pycapsule.h     pythonrun.h
datetime.h         longintrepr.h  pyconfig.h      pythread.h

Whereas the headers of pybind11 are actually installed relative to the pybind11 python-module:
(Note that the prefixes differ: /nix/store/nx3jw... vs /nix/store/6y6s8...)

$ python -c 'import pybind11; print(pybind11.__file__)'
/nix/store/6y6s8ahfhlp7a0g7xq9x8lhpfcwqg00b-python2.7-pybind11-2.2.3/lib/python2.7/site-packages/pybind11/__init__.pyc
$ ls /nix/store/6y6s8ahfhlp7a0g7xq9x8lhpfcwqg00b-python2.7-pybind11-2.2.3/include/python2.7/pybind11/
attr.h         chrono.h   detail   eval.h        numpy.h      pybind11.h  stl.h
buffer_info.h  common.h   eigen.h  functional.h  operators.h  pytypes.h
cast.h         complex.h  embed.h  iostream.h    options.h    stl_bind.h

[dorier]

Users of HPC installations, which use module load-type systems should also be affected.

I have this issue on my HPC system, which uses spack (so environment modules under the cover). get_include() returns the path to Python, not the path to the pybind11 module.

[dorier]

Note that right now my fix to find out the path to pybind11's include directory from another Python package is the following:

import os
import pybind11

def get_pybind11_include():
    path = os.path.dirname(pybind11.__file__)
    return '/'.join(path.split('/')[0:-4] + ['include'])

Surely this trick could be used directly inside pybind11.get_include() to get the right path.

[henryiii]

I think this has been fixed in #2116 and 2.5.0, from what I can tell. I'll be working on something related soon, so please reopen if this is still a problem!

[blechta]

Cool, a simple MWE (used to fail before)

pip3 install --prefix /tmp/issue2116 .
cd /
PYTHON_VERSION=$(python3 -c 'import sysconfig; print(sysconfig.get_python_version(), end="")')
export PYTHONPATH="/tmp/issue2116/lib/python$PYTHON_VERSION/site-packages:$PYTHONPATH"
PYBIND11_INCLUDE_PATH=$(python3 -c 'import pybind11; print(pybind11.get_include(), end="")')
ls $PYBIND11_INCLUDE_PATH/pybind11

now working fine.