Building pybind11 error.

[lunathanael]

Description

I cannot build c++ for pybind11 with poetry build. I've looked at multiple issues referencing this such as #7553, but I'm meeting errors that haven't been documented.

It looks like my installations might've been mixed with another project? Honestly, I'm not sure what is going on or if this is an issue with build.

I'm not sure what direction to go next, any help is welcome and appreciated!

# pyproject.toml
[tool.poetry]
name = "my-library"
version = "0.1.1"
description = "A cool library"
authors = ["Hello World <hello@helloworld.dev>"]
license = "MIT"
readme = "README.md"
packages = [{ include = "my_library" }]

[tool.poetry.dependencies]
python = "^3.10"
colorama = "^0.4.6"
protobuf = "^5.27.3"
websockets = "^12.0"
pydantic = "^2.8.2"
pillow = "^10.4.0"
setuptools = "60.0.0"
pybind11 = "^2.13.3"

[tool.poetry.group.dev.dependencies]
isort = "^5.13.2"
build = "^1.2.1"
packaging = "^24.1"
typing-extensions = "^4.12.2"

[tool.poetry.build]
generate-setup-file = true
script = "build.py"

[build-system]
requires = ["poetry-core", "pybind11", "setuptools"]
build-backend = "poetry.core.masonry.api"

# build.py
from pybind11.setup_helpers import Pybind11Extension, build_ext

def build(setup_kwargs):
    ext_modules = [
        Pybind11Extension("cutils", ["my_library/cutils/pext.cpp"]),
    ]
    setup_kwargs.update({
        "ext_modules": ext_modules,
        "cmd_class": {"build_ext": build_ext},
        "zip_safe": False,
    })

Preparing build environment with build-system requirements poetry-core, pybind11, setuptools
Building my-library (0.1.21)
  - Building sdist
  - Built my-library-0.1.21.tar.gz
  - Building wheel
C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\_distutils\dist.py:268: UserWarning: Unknown distribution option: 'cmd_class'
  warnings.warn(msg)
Traceback (most recent call last):
  File "C:\Users\USERNAME\Desktop\my-library\setup.py", line 42, in <module>
    setup(**setup_kwargs)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\__init__.py", line 108, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 158, in setup
    dist.parse_config_files()
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\_virtualenv.py", line 22, in parse_config_files
    result = old_parse_config_files(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\dist.py", line 603, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 70, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 135, in read_configuration
    validate(subset, filepath)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmpm01_ujlg\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 59, in validate
    raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project`.
configuration error: `project` must contain ['name'] properties

Command '['C:/Users/USERNAME/AppData/Local/Temp/tmpm01_ujlg/.venv/Scripts/python.exe', 'C:\\Users\\USERNAME\\Desktop\\my-library\\setup.py', 'build', '-b', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build', '--build-purelib', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build\\lib', '--build-platlib', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build\\lib.win-amd64-cpython-312']' returned non-zero exit status 1.

Workarounds

Don't build.

Poetry Installation Method

pipx

Operating System

Windows 11

Poetry Version

1.8.3

Poetry Configuration

cache-dir = "C:\\Users\\USERNAME\\AppData\\Local\\pypoetry\\Cache"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\USERNAME\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "venv"

Paths:
        data = "C:\Users\USERNAME\Desktop\some_other_project\.venv"
        include = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Include"
        platinclude = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Include"
        platlib = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Lib\site-packages"
        platstdlib = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Lib"
        purelib = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Lib\site-packages"
        scripts = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Scripts"
        stdlib = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib"

Variables:
        BINDIR = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Scripts"
        BINLIBDEST = "C:\Users\USERNAME\Desktop\some_other_project\.venv\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp312-win_amd64.pyd"
        INCLUDEPY = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Include"
        LIBDEST = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib"
        TZPATH = ""
        VERSION = "312"
        VPATH = "..\.."
        abiflags = ""
        base = "C:\Users\USERNAME\Desktop\some_other_project\.venv"
        exec_prefix = "C:\Users\USERNAME\Desktop\some_other_project\.venv"
        installed_base = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312"
        installed_platbase = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312"
        platbase = "C:\Users\USERNAME\Desktop\some_other_project\.venv"
        platlibdir = "DLLs"
        prefix = "C:\Users\USERNAME\Desktop\some_other_project\.venv"
        projectbase = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312"
        py_version = "3.12.1"
        py_version_nodot = "312"
        py_version_nodot_plat = "312"
        py_version_short = "3.12"
        srcdir = "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312"
        userbase = "C:\Users\USERNAME\AppData\Roaming\Python"

Example pyproject.toml

# pyproject.toml
[tool.poetry]
name = "my-library"
version = "0.1.1"
description = "A cool library"
authors = ["Hello World <hello@helloworld.dev>"]
license = "MIT"
readme = "README.md"
packages = [{ include = "my_library" }]

[tool.poetry.dependencies]
python = "^3.10"
colorama = "^0.4.6"
protobuf = "^5.27.3"
websockets = "^12.0"
pydantic = "^2.8.2"
pillow = "^10.4.0"
setuptools = "60.0.0"
pybind11 = "^2.13.3"

[tool.poetry.group.dev.dependencies]
isort = "^5.13.2"
build = "^1.2.1"
packaging = "^24.1"
typing-extensions = "^4.12.2"

[tool.poetry.build]
generate-setup-file = true
script = "build.py"

[build-system]
requires = ["poetry-core", "pybind11", "setuptools"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

logs: logs.txt

The logs are pretty long, so I attached the command output here, and logs.txt in attached.

PS C:\Users\USERNAME\Desktop\my-library> poetry -vvv build > logs.txt
Preparing build environment with build-system requirements poetry-core, pybind11, setuptools
Collecting poetry-core
  Using cached poetry_core-1.9.0-py3-none-any.whl.metadata (3.5 kB)
Collecting pybind11
  Using cached pybind11-2.13.3-py3-none-any.whl.metadata (9.5 kB)
Collecting setuptools
  Using cached setuptools-72.2.0-py3-none-any.whl.metadata (6.6 kB)
Using cached poetry_core-1.9.0-py3-none-any.whl (309 kB)
Using cached pybind11-2.13.3-py3-none-any.whl (240 kB)
Using cached setuptools-72.2.0-py3-none-any.whl (2.3 MB)
Installing collected packages: setuptools, pybind11, poetry-core
Successfully installed poetry-core-1.9.0 pybind11-2.13.3 setuptools-72.2.0
C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\_distutils\dist.py:268: UserWarning: Unknown distribution option: 'cmd_class'
  warnings.warn(msg)
Traceback (most recent call last):
  File "C:\Users\USERNAME\Desktop\my-library\setup.py", line 42, in <module>
    setup(**setup_kwargs)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\__init__.py", line 108, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 158, in setup
    dist.parse_config_files()
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\_virtualenv.py", line 22, in parse_config_files
    result = old_parse_config_files(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\dist.py", line 603, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 70, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 135, in read_configuration
    validate(subset, filepath)
  File "C:\Users\USERNAME\AppData\Local\Temp\tmppmy64v10\.venv\Lib\site-packages\setuptools\config\pyprojecttoml.py", line 59, in validate
    raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project`.
configuration error: `project` must contain ['name'] properties

  Stack trace:

  12  ~\pipx\venvs\poetry\Lib\site-packages\cleo\application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  11  ~\pipx\venvs\poetry\Lib\site-packages\poetry\console\application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  10  ~\pipx\venvs\poetry\Lib\site-packages\cleo\application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

   9  ~\pipx\venvs\poetry\Lib\site-packages\cleo\application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

   8  ~\pipx\venvs\poetry\Lib\site-packages\cleo\application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   7  ~\pipx\venvs\poetry\Lib\site-packages\cleo\commands\base_command.py:117 in run
       115│         io.input.validate()
       116│
     → 117│         return self.execute(io) or 0
       118│
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

   6  ~\pipx\venvs\poetry\Lib\site-packages\cleo\commands\command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

   5  ~\pipx\venvs\poetry\Lib\site-packages\poetry\console\commands\build.py:66 in handle
        64│             if not dist_dir.is_absolute():
        65│                 dist_dir = self.poetry.pyproject_path.parent / dist_dir
     →  66│             self._build(fmt, executable=env.python, target_dir=dist_dir)
        67│
        68│         return 0

   4  ~\pipx\venvs\poetry\Lib\site-packages\poetry\console\commands\build.py:49 in _build
        47│
        48│         for builder in builders:
     →  49│             builder(self.poetry, executable=executable).build(target_dir)
        50│
        51│     def handle(self) -> int:

   3  ~\pipx\venvs\poetry\Lib\site-packages\poetry\core\masonry\builders\wheel.py:121 in build
       119│             elif self._poetry.package.build_should_generate_setup():
       120│                 self._copy_module(zip_file)
     → 121│                 self._build(zip_file)
       122│             else:
       123│                 self._build(zip_file)

   2  ~\pipx\venvs\poetry\Lib\site-packages\poetry\core\masonry\builders\wheel.py:182 in _build
       180│                     try:
       181│                         os.chdir(self._path)
     → 182│                         self._run_build_command(setup)
       183│                     finally:
       184│                         os.chdir(current_path)

   1  ~\pipx\venvs\poetry\Lib\site-packages\poetry\core\masonry\builders\wheel.py:248 in _run_build_command
       246│                 "--inplace",
       247│             ])
     → 248│         subprocess.check_call([
       249│             self.executable.as_posix(),
       250│             str(setup),

  CalledProcessError

  Command '['C:/Users/USERNAME/AppData/Local/Temp/tmppmy64v10/.venv/Scripts/python.exe', 'C:\\Users\\USERNAME\\Desktop\\my-library\\setup.py', 'build', '-b', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build', '--build-purelib', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build\\lib', '--build-platlib', 'C:\\Users\\USERNAME\\Desktop\\my-library\\build\\lib.win-amd64-cpython-312']' returned non-zero exit status 1.

  at ~\AppData\Local\Programs\Python\Python312\Lib\subprocess.py:413 in check_call
       409│     if retcode:
       410│         cmd = kwargs.get("args")
       411│         if cmd is None:
       412│             cmd = popenargs[0]
    →  413│         raise CalledProcessError(retcode, cmd)
       414│     return 0
       415│
       416│
       417│ def check_output(*popenargs, timeout=None, **kwargs):

[dimbleby]

project must contain ['name'] properties

seems pretty clear?

You are using tooling that expects a PEP 621 compliant pyproject.toml, so you had better provide one.

[lunathanael]

I already have a [tool.poetry] section with a name provided. Stack overflow on poetry suggested that only one section should be provided. I've also previously tried adding a separate [project] section with name and etc defined, but it also had. It's own slew of errors.

Let me know if I am mistaken or what I should try! Thank you for the suggestion and insight!

Stack Overflow

And, #3332

[dimbleby]

please see PEP621

[lunathanael]

The pep you specified does require a Project section, with a note about this, what should a correct pyproject.toml for poetry look like then? As I've never come across one with a [project] section. Thank you for the response!

[dimbleby]

I cannot give a more precise answer than: please see PEP621

[lunathanael]

Using Poetry's init command, doesn't specifiy a [Project] section, I understand PEP621 as a resource for understanding the pyproject metadata specifications, however, PEP621 is a historical document, and even within PEP621, it does not explicitly require a [project] section to be defined, specifically if the build backend will dynamically provide the keys. From my understanding, I am using poetry-core as a build backend, but also using setup tools via generate-setup-file = true, I'm not sure what exactly I am looking for within the resource you provided. If you could provide something more specific in terms of what I am missing either in a file or a mistake I've made elsewhere, I would be very appreciative. Thank you for your guidance!

[dimbleby]

you are also using setuptools, apparently setuptools requires a pep621 pyproject.toml

nb build.py is unsupported and at your own risk / discretion #2740

[lunathanael]

I understand, thank you for the concise answer and fast response as always. Without using build.py or setup.py, how can one build extensions such as pybind11 with poetry? Resources I've found all point to build.py.

[dimbleby]

yes, but it is not supported. If you want to use a tool where maintainers do support building extensions - not poetry.

[lunathanael]

If you want to use a tool where maintainers do support building extensions - not poetry.

Thank you for the explanation.