cibuildwheel manylinux build issue

[TNonet]

Hi,

I am a big fan of carma!

I am transitioning some of my cython bindings that based off https://github.com/andrewcron/cy_armadillo to carma/pybind11 bindings. However, I can't seem to get it to build on AppVeyor using cibuildwheel.

It seems that carma's CMakeList is somewhat incompatible with manylinux which does not have python development libraries installed.

FIND_PACKAGE(Python3 COMPONENTS Interpreter Development NumPy REQUIRED)

This is resulting in the following errors.

Building cp38-manylinux_x86_64 wheel
CPython 3.8 manylinux x86_64
Setting up build environment...
    + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
    + which python
    + which pip
                                                              ✓ 0.12s
Building wheel...
    + rm -rf /tmp/cibuildwheel/built_wheel
    + mkdir -p /tmp/cibuildwheel/built_wheel
    + python -m pip wheel /project --wheel-dir=/tmp/cibuildwheel/built_wheel --no-deps
  error: subprocess-exited-with-error
  
  × Building wheel for l0learn (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [108 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/l0learn
      copying src/l0learn/models.py -> build/lib.linux-x86_64-cpython-38/l0learn
      copying src/l0learn/interface.py -> build/lib.linux-x86_64-cpython-38/l0learn
      copying src/l0learn/__init__.py -> build/lib.linux-x86_64-cpython-38/l0learn
      running egg_info
      writing src/l0learn.egg-info/PKG-INFO
      writing dependency_links to src/l0learn.egg-info/dependency_links.txt
      writing requirements to src/l0learn.egg-info/requires.txt
      writing top-level names to src/l0learn.egg-info/top_level.txt
      reading manifest file 'src/l0learn.egg-info/SOURCES.txt'
      writing manifest file 'src/l0learn.egg-info/SOURCES.txt'
      running build_ext
      -- The C compiler identification is GNU 10.2.1
      -- The CXX compiler identification is GNU 10.2.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- pybind11 v2.9.2
      -- Found PythonInterp: /opt/python/cp38-cp38/bin/python (found version "3.8.13")
      -- Found PythonLibs: /opt/_internal/cpython-3.8.13/lib
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- carma: enable arma no debug: OFF
      -- carma: enable extra debug: OFF
      -- carma: enable soft steal: OFF
      -- carma: enable hard steal: OFF
      -- carma: don't require owndata: OFF
      -- carma: don't require f-contiguous: OFF
      CMake Error at /tmp/pip-build-env-qup4s0j5/overlay/lib/python3.8/site-packages/cmake/data/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find Python3 (missing: Development NumPy Development.Module
        Development.Embed) (found version "3.8.13")
      Call Stack (most recent call first):
        /tmp/pip-build-env-qup4s0j5/overlay/lib/python3.8/site-packages/cmake/data/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
        /tmp/pip-build-env-qup4s0j5/overlay/lib/python3.8/site-packages/cmake/data/share/cmake-3.22/Modules/FindPython/Support.cmake:3[181](https://ci.appveyor.com/project/TNonet/l0learn/builds/43544744/job/vf1rpwwab3qkbtn3#L181) (find_package_handle_standard_args)
        /tmp/pip-build-env-qup4s0j5/overlay/lib/python3.8/site-packages/cmake/data/share/cmake-3.22/Modules/FindPython3.cmake:490 (include)
        external/carma/CMakeLists.txt:[187](https://ci.appveyor.com/project/TNonet/l0learn/builds/43544744/job/vf1rpwwab3qkbtn3#L187) (FIND_PACKAGE)

Do you have any advice on how I could work around this? I would also be happy to help with a patch/PR.

[RUrlus]

Hi @TNonet, thanks!

The latest stable should fix the issue. Please let me know if it solves the issue, I'll create a patch release if it resolved.

[TNonet]

Hi @RUrlus,

I am looking at it now! Thanks for such a fast response/resolve.

[TNonet]

Hi @RUrlus,

That worked, thanks!

I am now getting an undefined issue symbol issue on AppVeyor. I don't believe this issue is related to the original issue, but I would love some assistance. Before using carma, I would link armadillo, lapack, and blas with a C++ extension below.

Extension(...
              include_dirs=['.', np.get_include()],
              language="c++",
              libraries=["armadillo", "lapack", "blas"],
              extra_compile_args=["-std=c++11"],
              extra_link_args=["-std=c++11"],
              define_macros=macros,
              ),

However, now I am just including carma as a subdirectory in CMakeList files, and I believe carma/armadillo should "auto" find/link LAPACK and BLAS/OPENBLAS. However, I am getting the following issue when importing the successfully built package on AppVeyor, which I believe already has LAPACK, and BLAS/OPENBLAS installed on manylinux images.

cpython-37m-x86_64-linux-gnu.so: undefined symbol: dgemm_

Do you have any suggestions on how to fix this? It does successfully build and link on my MacOS builds.

[RUrlus]

Hi @TNonet,

carma links to the header only target of armadillo if none is provided. Hence, you need to link to BLAS/LAPACK yourself or use the CMake from Armadillo that is configured to link to BLAS/LAPACK.

If you haven't found it yet, I can highly recommend scikit-build. I've used it for other python packages with Pybind11 extensions, e.g. https://github.com/RUrlus/diptest

[TNonet]

Thanks for the direction! I will look into it!

[TNonet]

Hi @RUrlus,

I have been looking into scikit-build and diptest, and I think I am making good headway!

If you are willing to continue to help, I am running into two issues. I don't want to overstay my welcome, but I would be happy to help write up some beginner documentation or have two of my packages that hopefully will use carma as extended examples. I would like to give back for all your help!

First, when I build locally using pip install . most of the install process seems to run until the final install, which raises

CMake Error at cmake_install.cmake:56 (file):
    file cannot create directory: /usr/local/l0learn_core/lib.  Maybe need
    administrative privileges.

I am wondering if the last operation of my CMakeList file (which is borrowed heavily from diptest) is incorrect?

if(SKBUILD)
    INSTALL(TARGETS l0learn_core LIBRARY DESTINATION "${PROJECT_NAME}/lib")
ELSE ()
    INSTALL(
            TARGETS l0learn_core
            LIBRARY DESTINATION "${PROJECT_SOURCE_DIR}/l0learn_core/lib"
    )
ENDIF ()

Second, I get the following issue after compiling when I build on GH actions using cibuildwheel.

skbuild.exceptions.SKBuildError:
          CMake-installed files must be within the project root.
            Project Root  : /project/_skbuild/linux-x86_64-3.7/cmake-install
            Violating File: /usr/local/include/armadillo

I am relatively new to cmake, so I have been basing a decent amount of my work on https://gitlab.com/jason-rumengan/pyarma/. Therefore, I have decided to include armadillo as a subdirectory in my CMakeList file, similar to pyarma. There is an additional option, CMAKE_INSTALL_PREFIX, which is discussed in the armadillo documentation directly, https://gitlab.com/conradsnicta/armadillo-code#5a-installation-via-cmake, but I am unsure of how to make it play nicely with scikit-build and carma.

This code can be found on the use-carma branch of L0Learn (https://github.com/TNonet/L0Learn/tree/use-carma).

[RUrlus]

Hi @TNonet, sure happy to help and having a package from you as an example would be great.

One thing that's not entirely clear to me is if you want to install Armadillo in the python package or not?
I.e. do you want to ship the lib with the Python extension or are you statically linking?

If you're statically linking you could set the CMAKE_INSTALL_PREFIX to somewhere locally, e.g. the _skbuild dir.
If you're dynamically linking I would install Armadillo in the lib directory, you would have to manually set the CMAKE_INSTALL_PREFIX and clear it after.

Regarding the extension installation:

You're missing INCLUDE(GNUInstallDirs) in CMakeLists.txt which the install command relies on.
The 'magic' in installing without setting the CMAKE_INSTALL_PREFIX in the below:

if(SKBUILD)
    INSTALL(TARGETS l0learn_core LIBRARY DESTINATION "${PROJECT_NAME}/lib")

relies on the python package name and the CMake project name being the same.
In your case the CMake project name is l0learn_core vs l0learn on the python side.
I've never used the src/<python-package> structure with skbuild but I remember seeing something in their docs on how to handle this.

General tip, carma requires finding the python headers, which doesn't always work perfectly. Setting:

f"-DPython3_EXECUTABLE:STRING={sys.executable}",

as a cmake_args in the setup.py helps those cases

[TNonet]

Hi @RUrlus, Thank you for such a fast response. I will be looking at this today and tomorrow!

Updated with Comments:

First, to simplify the initial stages, I will assume armadillo is installed ('brew install armadillo`, or similar"). I will hopefully relax this constraint later.

I adjusted my CMakeList file to account for this by removing my armadillo-code subdirectory and the related steps. However, It seems when letting carma collect armadillo-code, it fails?

-- carma: Setting Armadillo version to 'v11.0.x' as none was specified.
-- carma: collecting Armadillo 11.0.x
[1/9] Creating directories for 'carmaarmadillo-populate'
[1/9] Performing download step (git clone) for 'carmaarmadillo-populate'
Cloning into 'armadillo-code'...
fatal: invalid reference: 11.0.x
CMake Error at /Users/tnonet/Documents/GitHub/L0Learn/python/_skbuild/macosx-10.16-x86_64-3.10/cmake-build/_deps/carmaarmadillo-subbuild/carmaarmadillo-populate-prefix/tmp/carmaarmadillo-populate-gitclone.cmake:40 (message):
  Failed to checkout tag: '11.0.x'

I was able to get around this issue by adding "-DUSE_ARMA_VERSION=10.6.x" to my cmake_args.

Secondly, I have moved my package from src/<python-package> to <python-package> to make this process easier. I may experiment with moving it back later.

Thirdly, I included the Python3_EXECUTABLE in the cmake-arg as you suggested.

Finally, I believe I followed your extension installation steps, but I am still getting a permissions issue installing.

-- Build files have been written to: /Users/tnonet/Documents/GitHub/L0Learn/python/_skbuild/macosx-10.16-x86_64-3.10/cmake-build
[4/5] Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:51 (file):
  file cannot create directory: /usr/local/l0learn/lib.  Maybe need
  administrative privileges.

I will be looking into this issue (and GNUInstallDirs and skbuild documentation) more tomorrow, but I wanted to get the update out soon.

I have updated the code on the 'use-carma' branch if you are interested.

[TNonet]

Hi @RUrlus,

I decided to take a step back and work on a more straightforward project. Therefore, I created an example project by merging your example https://github.com/RUrlus/carma/tree/stable/examples and pybind's https://github.com/pybind/scikit_build_example. This resulted in https://github.com/TNonet/carma-py-example.

An issue/inconsistency I have observed is that carma doesn't find python3 when carma is installed on manylinux and 'macosx_x86_64`. See this GH action build (https://github.com/TNonet/carma-py-example/actions/runs/2367460445).

However, it does find it when I include it from a subdirectory. See this GH action build (https://github.com/TNonet/carma-py-example/actions/runs/2367478858).

The commit diff between these two builds is found: TNonet/carma-py-example@4b7d7f4.

Is it possible I am doing something incorrectly concerning carma finding Python3?

I know there are other issues with this example project at the moment, but I am working on sorting them out as I learn more!

Thanks!

[RUrlus]

Hi @TNonet,

shit sorry. I just realised I forgot to change the finding of Python in the cmake targets when it is installed.
Just changed it in the unstable branch, will create a new patch release that should resolve it.

[TNonet]

Hi @RUrlus,

No worries. Thanks for being so helpful and responsive!

[TNonet]

Hi @RUrlus,

I wanted to follow up on my offer for example packages:

If you are willing to continue to help, I am running into two issues. I don't want to overstay my welcome, but I would be happy to help write up some beginner documentation or have two of my packages that hopefully will use carma as extended examples. I would like to give back for all your help!

Over the next few months, the following packages should be released/updated to use carma and hopefully have attached publications:

1. L0Learn (https://github.com/hazimehh/L0Learn), which has been using RcppArmadillo for quite a while, will be updated with python bindings using carma.
2. gL0Learn (https://github.com/TNonet/gL0Learn), which is soon to go into beta release, which will be released with Python bindings using carma.

I can follow up when they are officially released/updated.