CMake cannot find Python headers for pybind11 project

[lanctot]

Hi all,

Wonderful project, thank you so much for providing this to the community! And thanks to everyone who responded on my thread yesterday; I'm now using g++ for Linux and clang++ on MacOS.

I seem to be running into a problem where my CMake project cannot find the Python development headers in the docker images, but I have never had this issue locally nor in our GitHub actions CI.

Specifically, I see:

-- Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES Development Development.Module Development.Embed) (found version "3.6.13")

when running CMake and then compilation errors that Python.h is not found.

My wheels file is here: https://github.com/lanctot/open_spiel/blob/master/.github/workflows/wheels.yml

The line we use to find the headers in CMake is here: https://github.com/lanctot/open_spiel/blob/6c096e4913e8ccc2126d6e0bb7b69a9ed552c91f/open_spiel/python/CMakeLists.txt#L1

Tagging @henryiii because I based my wheels file on his CMake example in the pybind11 repos.

[Czaki]

Maybe @henryiii could tell more, but I suggest checking CMake version installed inside docker. It may be too old. Also, you may try to install a modern version of CMake from pip. https://pypi.org/project/cmake/

[lanctot]

@Czaki thanks, great suggestion! I was hopeful because in the past older Ubuntu version needed a more recent CMake installed via pip. So I added the command to CIBW_BEFORE_BUILD, and verified the version is 3.18.3 which is late enough. But unfortunately the same problem occurs.

So my current guess is that the Python headers are in a non-standard location in the images, which would not be too surprising.

What I find strange is that @henryiii 's pybind11 cmake example does not require any additional path additions in his wheels.yml file. The CMake file is different from ours in that it uses a pybind11_add_module CMake definition from the pybind11 project (news to me!), here. So I wonder if that is doing something extra to find the right include directories within the Docker images.

[Czaki]

if find_package(Python3 COMPONENTS Interpreter Development) looks only for headers in standard location and not use python interpreter for it then it is the wrong solution. Using nonstandard localization of python is not so rare thing. Someone could use tools like anaconda or pyenv to install different python versions which often install python in the home directory.
Someone could need to use some system that does not provide the newest python version in the repository.

Sometimes user does not have administration rights and needs to install it in userspace.

The localization of Python interpreter used by cibuildwheel you could check here https://github.com/joerick/cibuildwheel/blob/master/cibuildwheel/resources/build-platforms.toml

[lanctot]

@Czaki I agree with everything you said. I expect the CMake find_package helper to use the interpreter (because you have to pass in the path to the interpreter for it to work), but it's not clear from the CMake documentation. I guess it's probably that something else.

Thanks for the link to that file specifying the paths. I am now printing the python interpreter that is being used when building the wheel, and it's /opt/_internal/cpython-3.6.13/bin/python, which I don't see in any of those paths.. but maybe a sym link to one of them? Does that path seem right to you? Do you expect the dev headers to be installed for that interpreter?

Also, is there an environment variable set by cibuildwheel which specifies the python root directory being uses that I can maybe send as a hint to my CMake build file?

[Czaki]

I do not think that this is a proper interpreter for build purpose.

I think that you could set such a variable inside your setup.py file.

[lanctot]

Ok, thanks. I'm obtaining it and passing it to CMake in the same way as @henryiii's CMake example (using sys.executable), as done here: https://github.com/pybind/cmake_example/blob/0baee7e073a9b3738052f543e6bed412aaa22750/setup.py#L46

[YannickJadoul]

@lanctot Can you try without requiring Development.Module? I.e. like this:

find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)

I believe the manylinux images don't provide the libraries by default, but just the headers.

I've ran into this before, but I'm not sure I can find the relevant issue again :-/

[YannickJadoul]

I've ran into this before, but I'm not sure I can find the relevant issue again :-/

OK, see here: https://pybind11.readthedocs.io/en/stable/compiling.html#findpython-mode

And this: pybind/pybind11#2689

(That was easier than expected! git blame is wonderful :-D )

[lanctot]

@lanctot Can you try without requiring Development.Module? I.e. like this:

find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)

Ok, very different output based on that change (but interesting.. but that doesn't work locally for some reason):

-- Found Python3: /private/tmp/cibw_bin/python (found version "3.6.8") found components: Interpreter Development.Module
   Python executable: /private/tmp/cibw_bin/python
   Python include dirs: /Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m

It is now failing in a different place (and does not seem related to cibuildwheel), so I will consider this fixed.. thanks @YannickJadoul !

BTW is that correct for the executable and include dirs combination? I'm surprised that the executable and include dirs don't share a common path prefix (maybe due to some symbolic linking under the hood?)

[henryiii]

I think that's pretty normal for virtual environments, the headers may come from the original location, and the executable will come from the venv dir.

[YannickJadoul]

-- Found Python3: /private/tmp/cibw_bin/python (found version "3.6.8") found components: Interpreter Development.Module
   Python executable: /private/tmp/cibw_bin/python
   Python include dirs: /Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m

Huh, so this is a path on macOS, and not in a manylinux Docker image? Shouldn't your macOS environment also have the Development.Module part?

[lanctot]

Oh I apologize, I am testing both ubuntu-20.04 (via manylinux2014) and macOS-10.15 concurrently and this last time the MacOS build failed first and cancelled the Ubuntu run.

I will try just the Ubuntu run later tonight to see what happens and report back. Thanks so much for the help!

[YannickJadoul]

Aha, OK, then all is fine, if the first issue was manylinux and the second was macOS. Thanks for clarifying! :-)

[lanctot]

Hi @YannickJadoul, just confirming that your suggestion worked and I was able to progress further on the ubuntu-20.04 (manylinux2014).

The build is now failing with different C++ compilation errors (both MacOS and Ubuntu), and they seem to be related to our code not being friendly to older/different compilers.. so I just have to go through and fix each instance.

Thanks again!

[YannickJadoul]

@lanctot Great to hear! Thanks for the small update :-)

[LecrisUT]

@YannickJadoul Is there any other fix than removing REQUIRED? A future version of cmake?

[henryiii]

I believe find_package(Python COMPONENTS Interpreter Development.Module REQUIRED) was the fix. It is absolutely not supported to look for the Development.Embed component (and thereby the Development meta component) on manylinux; that component is intentionally deleted to save space and stop a common mistake. I think there was a different issue related to C++ compilation that is not related to cibuildwheel or Python discovery.