Usage question on cmake FindPythonLibs in manylinux containers

[bsolomon1124]

^{Please let me know if this is better directed towards CMake itself.}

I am attempting to build manylinux2014 wheels for pytype, which uses CMake and its FindPythonLibs function. It is having trouble finding libpythonX.Y.so in certain cases.

Putting pytype aside, here is an MVCE:

Container shell:

docker container run -it --rm --workdir='/foo' \
    quay.io/pypa/manylinux2014_x86_64:latest

Example:

yum install -y cmake ninja-build python3-devel
{
  echo 'project(Foo)'
  echo 'cmake_minimum_required(VERSION 2.8)'
  echo 'find_package(PythonLibs)'
  echo 'if(NOT PYTHONLIBS_FOUND)'
  echo '  message(FATAL_ERROR "Could not find developer libraries")'
  echo 'endif()'
} > CMakeLists.txt
cmake -G Ninja -DPython_ADDITIONAL_VERSIONS=3.7

Output:

-- The C compiler identification is GNU 9.1.1
-- The CXX compiler identification is GNU 9.1.1
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Ninja
-- Check for working CXX compiler using: Ninja -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Could NOT find PythonLibs (missing:  PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) 
CMake Error at CMakeLists.txt:5 (message):
  Could not find developer libraries


-- Configuring incomplete, errors occurred!
See also "/foo/CMakeFiles/CMakeOutput.log".

Here the -DPython_ADDITIONAL_VERSIONS=3.7 comes from (I think) invoking the pytype bdist_wheel with /opt/python/cp37-cp37m/bin/python, for example.

Looking more closely, I see:

$ ls -1 /usr/lib64/libpython*
/usr/lib64/libpython2.7.so.1.0
/usr/lib64/libpython3.6m.so
/usr/lib64/libpython3.6m.so.1.0
/usr/lib64/libpython3.so

where the final 3 entries have been put there via the python3-devel install. I also see that the python3-devel install fetched python3-libs-3.6.8-13.el7.x86_64.rpm specifically.

So it looks like cmake is looking for a MAJOR.MINOR version, rather than just libpython3.so.
And indeed, if I run:

alias clean='rm -rvf CMakeCache.txt CMakeFiles'
clean
cmake -G Ninja -DPython_ADDITIONAL_VERSIONS=3.6

Then the cmake invocation succeeds:

-- The C compiler identification is GNU 9.1.1
-- The CXX compiler identification is GNU 9.1.1
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Ninja
-- Check for working CXX compiler using: Ninja -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PythonLibs: /usr/lib64/libpython3.6m.so (found version "3.6.8") 
-- Configuring done
-- Generating done
-- Build files have been written to: /foo

How can I allow FindPythonLibs to succeed here? Does cmake need the libpython3.so or a specific MAJOR.MINOR version? Would I need to install something like python37-devel? (I don't see it on the default yum repos so would probably need to get it from somewhere else.)

[bsolomon1124]

Edit: by setting PYTHON_LIBRARY='/usr/lib64/libpython3.so', it looks like the library gets found, but there's still an issue with the include dir:

PYTHON_LIBRARY='/usr/lib64/libpython3.so'
PYTHON_INCLUDE_DIR='/opt/_internal/cpython-3.7.7/include/python3.7m/'
clean
cmake -G Ninja -DPython_ADDITIONAL_VERSIONS=3.7

Produces:

-- Could NOT find PythonLibs (missing:  PYTHON_INCLUDE_DIRS) 
CMake Error at CMakeLists.txt:5 (message):
  Could not find developer libraries


-- Configuring incomplete, errors occurred!

[bsolomon1124]

Edit 2: it looks like using a more recent version of CMake 3.17.2 built from source makes this go away. The version that yum will pull is 2.8.x in the example above. I'll close this issue when I can confirm this.

[mayeut]

This seems like a duplicate of #484

it looks like using a more recent version of CMake 3.17.2 built from source makes this go away

You can pip install cmake to get the latest cmake c.f. https://pypi.org/project/cmake/

[bsolomon1124]

This seems like a duplicate of #484

Yes, absolutely. Thanks @mayeut .