Multiple calls operating on the same build folder leak resources from isolated environment #26

ghost · 2021-08-26T17:22:28Z

Hello,

Ninja is currently the only generator supported by cmake-build-extension:

cmake-build-extension/src/cmake_build_extension/build_extension.py

Line 78 in bb836e4

raise RuntimeError("Required command 'ninja' not found")

Unfortunately it is not installed on all platform and a great benefit of CMake is to be agnostic of the final build tool that is used.

It would be nice to add a configuration option that select a different generator in replacement of Ninja (e.g. Make)

diegoferigo · 2021-10-17T12:54:29Z

Hi @remy-phelipot, sorry for the late reply, I missed this issue somehow. ninja is a dependency, and it should provide ootb the missing command.

cmake-build-extension/setup.cfg

Line 50 in bb836e4

ninja

Can you provide more info on your use case?

ghost · 2021-10-19T08:51:26Z

Hello @diegoferigo.

I tested again and indeed ninja seems to be correctly installed, so my original message do not correctly reflect the actual problem.

I'm encountering the following error when building the example module (I added ninja as build dependency in pyproject.toml to be sure it is installed, but is not necessary):

$ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45, setuptools_scm[toml]>=6.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing src/mymath.egg-info/PKG-INFO
writing dependency_links to src/mymath.egg-info/dependency_links.txt
writing entry points to src/mymath.egg-info/entry_points.txt
writing requirements to src/mymath.egg-info/requires.txt
writing top-level names to src/mymath.egg-info/top_level.txt
writing manifest file 'src/mymath.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
running build_ext

==> Configuring:
$ cmake -S /home/phelipot/projects/cmake-build-extension/example -B /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig -DPython3_ROOT_DIR=/tmp/build-env-j2geah8k -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=ON -DEXAMPLE_WITH_PYBIND11:BOOL=OFF

==> Building:
$ cmake --build /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings --config Release

==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings

-- Found Python3: /tmp/build-env-j2geah8k/bin/python3 (found version "3.8.10") found components: Interpreter Development.Module NumPy 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings
[4/4] Linking CXX shared module bindings_swig/mymath/_swig_bindings.so
-- Install configuration: "Release"
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/libmymath.a
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/include/mymath.h
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bin/print_answer_swig
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets-release.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfig.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfigVersion.cmake
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bindings.py
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/_swig_bindings.so

==> Configuring:
$ cmake -S /home/phelipot/projects/cmake-build-extension/example -B /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11 -DPython3_ROOT_DIR=/tmp/build-env-j2geah8k -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=OFF -DEXAMPLE_WITH_PYBIND11:BOOL=ON

==> Building:
$ cmake --build /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings --config Release

==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings

CMake Error at CMakeLists.txt:2 (project):
  Running

   '/tmp/pip-build-env-rr5z5dv1/overlay/bin/ninja' '--version'

  failed with:

   No such file or directory


-- Configuring incomplete, errors occurred!
See also "/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 43, in <module>
    setuptools.setup(
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 110, in run
    self.build_extension(ext)
  File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 224, in build_extension
    subprocess.check_call(configure_command)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-S', '/home/phelipot/projects/cmake-build-extension/example', '-B', '/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11', '-DPython3_ROOT_DIR=/tmp/build-env-j2geah8k', '-DCALL_FROM_SETUP_PY:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=OFF', '-DEXAMPLE_WITH_SWIG:BOOL=OFF', '-DEXAMPLE_WITH_PYBIND11:BOOL=ON']' returned non-zero exit status 1.

ERROR Backend subproccess exited when trying to invoke build_wheel

If I produce the sdist and the wheel using python -m build, then the build process is executed successfully.

Do you know what is going on and if there is any solution?

diegoferigo · 2021-10-19T09:16:38Z

Calling python -m build triggers the generation of the sdist and, from it, the wheel. Passing the --wheel option skips the sdist generation and the wheel is built directly from the sources. In your case, there's a mismatch in the two behaviors.

I couldn't reproduce your problem on my PC (see details below). The example has two CMakeExtensions, one for SWIG and the other for pybind11. What's strange is that the former works, but the process fails when the latter is configured. It's odd that it complains about ninja, since it has already successfully built the first extension in the very same isolated environment.

Let's keep investigating, I suspect that is something specific to your setup since it's not happening in mine nor in CI that builds installs and tests in all major OSs.

python -m build --wheel

cmake-build-extension/example on  master via △ v3.21.3 via 🐍 v3.8.10 🅒 /conda  
❯ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, numpy, pybind11, setuptools>=45, setuptools_scm[toml]>=6.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing src/mymath.egg-info/PKG-INFO
writing dependency_links to src/mymath.egg-info/dependency_links.txt
writing entry points to src/mymath.egg-info/entry_points.txt
writing requirements to src/mymath.egg-info/requires.txt
writing top-level names to src/mymath.egg-info/top_level.txt
writing manifest file 'src/mymath.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_ext

==> Configuring:
$ cmake -S /home/dferigo/git/cmake-build-extension/example -B /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig -DPython3_ROOT_DIR=/tmp/build-env-hlr9nns_ -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=ON -DEXAMPLE_WITH_PYBIND11:BOOL=OFF

==> Building:
$ cmake --build /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings --config Release

==> Installing:
$ cmake --install /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /conda/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /conda/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /tmp/build-env-hlr9nns_/bin/python3 (found version "3.8.10") found components: Interpreter Development.Module NumPy 
-- Found SWIG: /conda/bin/swig (found suitable version "4.0.2", minimum required is "4.0")  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings
[7/7] Linking CXX shared module bindings_swig/mymath/_swig_bindings.so
-- Install configuration: "Release"
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/libmymath.a
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/include/mymath.h
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bin/print_answer_swig
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets-release.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfig.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfigVersion.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bindings.py
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/_swig_bindings.so

==> Configuring:
$ cmake -S /home/dferigo/git/cmake-build-extension/example -B /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11 -DPython3_ROOT_DIR=/tmp/build-env-hlr9nns_ -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=OFF -DEXAMPLE_WITH_PYBIND11:BOOL=ON

==> Building:
$ cmake --build /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings --config Release

==> Installing:
$ cmake --install /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /conda/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /conda/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /tmp/build-env-hlr9nns_/bin/python3 (found version "3.8.10") found components: Interpreter Development.Module NumPy 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /tmp/build-env-hlr9nns_/lib/python3.8/site-packages/pybind11/include (found version "2.8.0" )
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dferigo/git/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings
[6/6] Linking CXX shared module bindings_pybind11/mymath/bindings.cpython-38-x86_64-linux-gnu.so
-- Install configuration: "Release"
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/lib/libmymath.a
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/include/mymath.h
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/bin/print_answer_pybind11
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathTargets.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathTargets-release.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathConfig.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathConfigVersion.cmake
-- Installing: /home/dferigo/git/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11/bindings.cpython-38-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/mymath_swig
creating build/bdist.linux-x86_64/wheel/mymath_swig/lib
copying build/lib.linux-x86_64-3.8/mymath_swig/lib/libmymath.a -> build/bdist.linux-x86_64/wheel/mymath_swig/lib
creating build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake
creating build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets.cmake -> build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets-release.cmake -> build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfigVersion.cmake -> build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfig.cmake -> build/bdist.linux-x86_64/wheel/mymath_swig/lib/cmake/MyMath
creating build/bdist.linux-x86_64/wheel/mymath_swig/bin
copying build/lib.linux-x86_64-3.8/mymath_swig/bin/print_answer_swig -> build/bdist.linux-x86_64/wheel/mymath_swig/bin
copying build/lib.linux-x86_64-3.8/mymath_swig/bin/__main__.py -> build/bdist.linux-x86_64/wheel/mymath_swig/bin
copying build/lib.linux-x86_64-3.8/mymath_swig/__init__.py -> build/bdist.linux-x86_64/wheel/mymath_swig
creating build/bdist.linux-x86_64/wheel/mymath_swig/include
copying build/lib.linux-x86_64-3.8/mymath_swig/include/mymath.h -> build/bdist.linux-x86_64/wheel/mymath_swig/include
copying build/lib.linux-x86_64-3.8/mymath_swig/bindings.py -> build/bdist.linux-x86_64/wheel/mymath_swig
copying build/lib.linux-x86_64-3.8/mymath_swig/_swig_bindings.so -> build/bdist.linux-x86_64/wheel/mymath_swig
creating build/bdist.linux-x86_64/wheel/mymath_pybind11
creating build/bdist.linux-x86_64/wheel/mymath_pybind11/lib
copying build/lib.linux-x86_64-3.8/mymath_pybind11/lib/libmymath.a -> build/bdist.linux-x86_64/wheel/mymath_pybind11/lib
creating build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake
creating build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathTargets.cmake -> build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathTargets-release.cmake -> build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathConfigVersion.cmake -> build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake/MyMath
copying build/lib.linux-x86_64-3.8/mymath_pybind11/lib/cmake/MyMath/MyMathConfig.cmake -> build/bdist.linux-x86_64/wheel/mymath_pybind11/lib/cmake/MyMath
creating build/bdist.linux-x86_64/wheel/mymath_pybind11/bin
copying build/lib.linux-x86_64-3.8/mymath_pybind11/bin/print_answer_pybind11 -> build/bdist.linux-x86_64/wheel/mymath_pybind11/bin
copying build/lib.linux-x86_64-3.8/mymath_pybind11/bin/__main__.py -> build/bdist.linux-x86_64/wheel/mymath_pybind11/bin
copying build/lib.linux-x86_64-3.8/mymath_pybind11/__init__.py -> build/bdist.linux-x86_64/wheel/mymath_pybind11
creating build/bdist.linux-x86_64/wheel/mymath_pybind11/include
copying build/lib.linux-x86_64-3.8/mymath_pybind11/include/mymath.h -> build/bdist.linux-x86_64/wheel/mymath_pybind11/include
copying build/lib.linux-x86_64-3.8/mymath_pybind11/bindings.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/mymath_pybind11
running install_egg_info
running egg_info
writing src/mymath.egg-info/PKG-INFO
writing dependency_links to src/mymath.egg-info/dependency_links.txt
writing entry points to src/mymath.egg-info/entry_points.txt
writing requirements to src/mymath.egg-info/requires.txt
writing top-level names to src/mymath.egg-info/top_level.txt
writing manifest file 'src/mymath.egg-info/SOURCES.txt'
Copying src/mymath.egg-info to build/bdist.linux-x86_64/wheel/mymath-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/mymath-0.0.0.dist-info/WHEEL
creating '/home/dferigo/git/cmake-build-extension/example/dist/tmpj8dg_w_n/mymath-0.0.0-cp38-cp38-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'mymath_pybind11/__init__.py'
adding 'mymath_pybind11/bindings.cpython-38-x86_64-linux-gnu.so'
adding 'mymath_pybind11/bin/__main__.py'
adding 'mymath_pybind11/bin/print_answer_pybind11'
adding 'mymath_pybind11/include/mymath.h'
adding 'mymath_pybind11/lib/libmymath.a'
adding 'mymath_pybind11/lib/cmake/MyMath/MyMathConfig.cmake'
adding 'mymath_pybind11/lib/cmake/MyMath/MyMathConfigVersion.cmake'
adding 'mymath_pybind11/lib/cmake/MyMath/MyMathTargets-release.cmake'
adding 'mymath_pybind11/lib/cmake/MyMath/MyMathTargets.cmake'
adding 'mymath_swig/__init__.py'
adding 'mymath_swig/_swig_bindings.so'
adding 'mymath_swig/bindings.py'
adding 'mymath_swig/bin/__main__.py'
adding 'mymath_swig/bin/print_answer_swig'
adding 'mymath_swig/include/mymath.h'
adding 'mymath_swig/lib/libmymath.a'
adding 'mymath_swig/lib/cmake/MyMath/MyMathConfig.cmake'
adding 'mymath_swig/lib/cmake/MyMath/MyMathConfigVersion.cmake'
adding 'mymath_swig/lib/cmake/MyMath/MyMathTargets-release.cmake'
adding 'mymath_swig/lib/cmake/MyMath/MyMathTargets.cmake'
adding 'mymath-0.0.0.dist-info/METADATA'
adding 'mymath-0.0.0.dist-info/WHEEL'
adding 'mymath-0.0.0.dist-info/entry_points.txt'
adding 'mymath-0.0.0.dist-info/top_level.txt'
adding 'mymath-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built mymath-0.0.0-cp38-cp38-linux_x86_64.whl

ghost · 2021-10-19T09:42:24Z

I reset my environment and do:

$ virtualenv -p python3 .venv
$ . .venv/bin/activate
(.venv) $ pip install --upgrade pip build
(.venv) $ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45, 

...
Successfully built mymath-0.0.0-cp38-cp38-linux_x86_64.whl

So far so good. But if I execute the command again it fails:

$ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45, 

...

==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings

CMake Error at CMakeLists.txt:2 (project):
  Running

   '/tmp/build-env-y54elrvx/bin/ninja' '--version'

  failed with:

   No such file or directory


-- Configuring incomplete, errors occurred!
See also "/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 43, in <module>
    setuptools.setup(
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 110, in run
    self.build_extension(ext)
  File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 224, in build_extension
    subprocess.check_call(configure_command)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-S', '/home/phelipot/projects/cmake-build-extension/example', '-B', '/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig', '-DPython3_ROOT_DIR=/tmp/build-env-bil93mdf', '-DCALL_FROM_SETUP_PY:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=OFF', '-DEXAMPLE_WITH_SWIG:BOOL=ON', '-DEXAMPLE_WITH_PYBIND11:BOOL=OFF']' returned non-zero exit status 1.

ERROR Backend subproccess exited when trying to invoke build_wheel

It seems temporary files are produced in the build directory that interfere between each consecutive build. If I delete the directory, then it works again like a charm.

(.venv) $ rm -rf build && python -m build --wheel
...
Successfully built mymath-0.0.0-cp38-cp38-linux_x86_64.whl

diegoferigo · 2021-10-19T09:51:42Z

Ow ok I finally understood the problem after your last comment! The CMakeCache.txt file stored in the build folder contains the path to the ninja command, which is installed in a virtualenv in /tmp. The virtualenv is deleted after the first time the wheel is built, but the build folder, being stored in-source, is kept. The second time you build the wheel, the CMakeCache.txt entry does not get updated with the new virtualenv folder in /tmp, therefore it fails.

I suspect that a workaround is passing in your setup.py the following option: f"-DCMAKE_MAKE_PROGRAM={shutil.which('ninja')}". Can you please try to check if it solves the problem?

ghost · 2021-10-19T10:04:35Z

Yeap your solution works at least on linux platforms 🙂 As the use of ninja is internal of cmake-build-extension and the end user have no idea of what is going on in this tricky case, may I suggest to add this code in the project? For instance here:

diff --git a/src/cmake_build_extension/build_extension.py b/src/cmake_build_extension/build_extension.py
index a27d144..c4667c7 100644
--- a/src/cmake_build_extension/build_extension.py
+++ b/src/cmake_build_extension/build_extension.py
@@ -148,6 +148,7 @@ class BuildExtension(build_ext):
             "-GNinja",
             f"-DCMAKE_BUILD_TYPE={ext.cmake_build_type}",
             f"-DCMAKE_INSTALL_PREFIX:PATH={cmake_install_prefix}",
+            f"-DCMAKE_MAKE_PROGRAM={shutil.which('ninja')}",
         ]
 
         # Extend the configure arguments with those passed from the extension

diegoferigo · 2021-10-19T10:10:36Z

Yes this was the idea, thanks for trying! Do you have specific reasons to say that this works only on linux platforms? The shutil module should work also on Windows and macOS.

Feel free to open a PR with this change, otherwise I'll take care of it in the next days.

ghost · 2021-10-19T12:02:15Z

I agree about Windows compatibility, it should be ok.

Fix #26: Pass ninja path to CMakeCache through CMAKE_MAKE_PROGRAM

diegoferigo self-assigned this Oct 19, 2021

diegoferigo added the bug Something isn't working label Oct 19, 2021

ghost mentioned this issue Oct 19, 2021

Pass ninja path to CMakeCache through CMAKE_MAKE_PROGRAM #29

Merged

diegoferigo changed the title ~~Configurable generator~~ Multiple packaging runs operating on the same build folder leak resources from isolated environment Oct 19, 2021

diegoferigo changed the title ~~Multiple packaging runs operating on the same build folder leak resources from isolated environment~~ Multiple calls operating on the same build folder leak resources from isolated environment Oct 19, 2021

diegoferigo closed this as completed in #29 Oct 19, 2021

diegoferigo added a commit that referenced this issue Oct 19, 2021

Merge pull request #29 from remy-phelipot/26_fix_ninja_path_cache

8972036

Fix #26: Pass ninja path to CMakeCache through CMAKE_MAKE_PROGRAM

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple calls operating on the same build folder leak resources from isolated environment #26

Multiple calls operating on the same build folder leak resources from isolated environment #26

ghost commented Aug 26, 2021

diegoferigo commented Oct 17, 2021

ghost commented Oct 19, 2021 •

edited by ghost

Loading

diegoferigo commented Oct 19, 2021

ghost commented Oct 19, 2021

diegoferigo commented Oct 19, 2021 •

edited

Loading

ghost commented Oct 19, 2021

diegoferigo commented Oct 19, 2021 •

edited

Loading

ghost commented Oct 19, 2021

Multiple calls operating on the same build folder leak resources from isolated environment #26

Multiple calls operating on the same build folder leak resources from isolated environment #26

Comments

ghost commented Aug 26, 2021

diegoferigo commented Oct 17, 2021

ghost commented Oct 19, 2021 • edited by ghost Loading

diegoferigo commented Oct 19, 2021

ghost commented Oct 19, 2021

diegoferigo commented Oct 19, 2021 • edited Loading

ghost commented Oct 19, 2021

diegoferigo commented Oct 19, 2021 • edited Loading

ghost commented Oct 19, 2021

ghost commented Oct 19, 2021 •

edited by ghost

Loading

diegoferigo commented Oct 19, 2021 •

edited

Loading

diegoferigo commented Oct 19, 2021 •

edited

Loading