CMakeNotFoundError with pyodide: cmake -E capabilities failed

[pthom]

Hello,

I am brand new to pyodide, but I'm trying to compile a project (imgui_bundle) that uses scikit-build-core.

I'm trying to build it from inside the pyodide build tree, and inside the Docker container, by running this command:

export CMAKE_EXECUTABLE=$(which cmake)   # failed attempt to help cmake.py find cmake
PYODIDE_PACKAGES="imgui_bundle" make

And I get this error:

File "/tmp/build-env-azbe70tj/lib/python3.12/site-packages/scikit_build_core/build/__init__.py", line 31, in build_wheel                
    return _build_wheel_impl(                                                                                                             
           ^^^^^^^^^^^^^^^^^^                                                                                                             
  File "/tmp/build-env-azbe70tj/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 151, in _build_wheel_impl            
    cmake = CMake.default_search(version=settings.cmake.version, env=os.environ)                                                          
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                          
  File "/tmp/build-env-azbe70tj/lib/python3.12/site-packages/scikit_build_core/cmake.py", line 63, in default_search                      
    raise CMakeNotFoundError(msg)                                                                                                         
scikit_build_core.errors.CMakeNotFoundError: Could not find CMake with version >=3.26.1                                                   
                                                                                                                                          
ERROR Backend subprocess exited when trying to invoke build_wheel                                                                         
[2024-04-19 18:51:33] Failed building package imgui_bundle in 2.4 seconds.                                                                
                                                                                                                                          
                                                                                                                                          
ERROR: cancelling buildall due to error building imgui_bundle                                

Here is an extract from my pyproject.toml:

[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "imgui-bundle"
version = "1.5.0"
requires-python = ">=3.10"

dependencies = [    "numpy",    "munch",    "glfw",    "PyOpenGL",    "pillow"]

...

[tool.scikit-build]
wheel.expand-macos-universal-tags = true
wheel.packages = ["bindings/imgui_bundle"]

cmake.version = ">=3.26.1"
cmake.verbose = true
logging.level = "INFO"

And the meta.yaml file I am using:

package:
  name: imgui_bundle
  version: 1.3.0
  top-level:
    - imgui_bundle
source:
  path: /workspaces/pyodide/_bundle/imgui_bundle
about:
  home: https://github.com/pthom/imgui_bundle
  PyPI: https://pypi.org/project/imgui_bundle
  summary:
    "Dear ImGui Bundle: easily create ImGui applications in Python and C++.
    Batteries included!"
  license: ""
extra:
  recipe-maintainers:
    - pthom
requirements:
  # host:
  #   #Dependencies that are needed to build the package
  #   - scikit-build-core>=0.3.3
  run:
    # Dependencies that are needed to run the package
    - numpy
    - munch
    - Pillow

I did find where the exception is raised:

scikit-build-core/src/scikit_build_core/cmake.py

Lines 53 to 63 in f844088

	cmake_executable = env.get("CMAKE_EXECUTABLE", "")
	candidates: Iterable[Program] = (
	[get_cmake_program(Path(cmake_executable))]
	if cmake_executable
	else get_cmake_programs(module=module)
	)
	cmake_program = best_program(candidates, version=version)
	if cmake_program is None:
	msg = f"Could not find CMake with version {version}"
	raise CMakeNotFoundError(msg)

It seems that it might perhaps be possible to pass the cmake location via the env variable CMAKE_EXECUTABLE, but this does not work.

I'm afraid I might be missing something obvious. However, I can not find a way to debug the internals of scikit_build_core, Since it is installed in a temporary isolated environment. Help would be appreciated.

Many thanks!

[pthom]

I was able to debug inside scikit-build-core, by using backend-path inside pyproject.toml (cf cf https://peps.python.org/pep-0517/#in-tree-build-backends)

[build-system]
requires = ["pybind11", "packaging", "pathspec"]
build-backend = "scikit_build_core.build"
backend-path = ["_backends/scikit-build-core/src/"]

As a matter of fact get_cmake_program will call cmake -E capabilities which fails.
I Instrumented its code as below:

def get_cmake_program(cmake_path: Path) -> Program:
    """
    Get the Program (with version) for CMake given a path. The version will be
    None if it cannot be determined.
    """
    # get version and capabilities with subprocess (my instrumentation)
    import subprocess
    version_result = subprocess.run([cmake_path, "--version"], capture_output=True)
    capabilities_result = subprocess.run([cmake_path, "-E", "capabilities"], capture_output=True)
    raise Exception(f"""
                    get_cmake_program 
                    {cmake_path=} 
                    
                    {version_result.stdout=} 
                    {version_result.returncode=}) 

                    {capabilities_result.stdout=} 
                    {capabilities_result.returncode=}
                    {capabilities_result.stderr=}
) 

                            """)

    # original code below
    try:
        result = Run().capture(cmake_path, "-E", "capabilities")
    except (subprocess.CalledProcessError, PermissionError):
        return Program(cmake_path, None)

    try:
        version = Version(json.loads(result.stdout)["version"]["string"])
    except (json.decoder.JSONDecodeError, KeyError, InvalidVersion):
        logger.warning("Could not determine CMake version, got {!r}", result.stdout)
        return Program(cmake_path, None)

    logger.info("CMake version: {}", version)
    return Program(cmake_path, version)

And I get the following exception as an output:

Exception:                                                                                                                                
                    get_cmake_program                                                                                                     
                    cmake_path=PosixPath('/tmp/tmpxg_inkrx/cmake')                                                                        
                                                                                                                                          
                    version_result.stdout=b'cmake version 3.28.3\n\nCMake suite maintained and supported by Kitware                       
(kitware.com/cmake).\n'                                                                                                                                                      
                    version_result.returncode=0)                                                                                          
                                                                                                                                          
                    capabilities_result.stdout=b''                                                                                        
                    capabilities_result.returncode=1
                    capabilities_result.stderr=b"configure: cmake -DCMAKE_C_COMPILER=/tmp/tmpxg_inkrx/cc                                  
-DCMAKE_CXX_COMPILER=/tmp/tmpxg_inkrx/c++ -DCMAKE_AR=/tmp/tmpxg_inkrx/ar -DCMAKE_C_COMPILER_AR=/tmp/tmpxg_inkrx/ar                        
-DCMAKE_CXX_COMPILER_AR=/tmp/tmpxg_inkrx/ar --fresh -E capabilities                                                                       
-DCMAKE_CROSSCOMPILING_EMULATOR=/workspaces/pyodide/emsdk/emsdk/node/16.20.0_64bit/bin/node\nCMake Error: Unknown argument -E\nCMake      
Error: Run 'cmake --help' for all supported options.\nemcmake: error: 'cmake -DCMAKE_C_COMPILER=/tmp/tmpxg_inkrx/cc                       
-DCMAKE_CXX_COMPILER=/tmp/tmpxg_inkrx/c++ -DCMAKE_AR=/tmp/tmpxg_inkrx/ar -DCMAKE_C_COMPILER_AR=/tmp/tmpxg_inkrx/ar                        
-DCMAKE_CXX_COMPILER_AR=/tmp/tmpxg_inkrx/ar --fresh -E capabilities                                                                       
-DCMAKE_CROSSCOMPILING_EMULATOR=/workspaces/pyodide/emsdk/emsdk/node/16.20.0_64bit/bin/node' failed (returned 1)\n"                       
)                                 

capabilities_result.stderr is a bit strange since it mixes errrors from elsewhere. However it contains this interesting piece:

CMake Error: Unknown argument -E
CMake Error: Run 'cmake --help' for all supported options.

So as a conclusion, I can see that the command cmake -E capabilities might fail (perhaps because some compilation options of CMake do not support this kind of output), even with cmake 3.28.3

[LecrisUT]

Probably #715?

[pthom]

Probably #715?

oh yes, after a deep dive in the internal of the code, I now see that it it the same.

[henryiii]

I'll be fixing shortly. Really pyodide's fault, but I can fix here faster.

[henryiii]

(0.8.x works until then with Pyodide)

[henryiii]

Fix in pyodide is #717, FYI.

[henryiii]

Fix released as 0.9.1!

[pthom]

Thanks a lot !!!