Cannot add numpy when requires-python = ">=3.8,<3.13"

[mforbes]

Describe the bug

I cannot use pdm to create a working environment with numpy that supports python back to 3.8 but which works with python 3.12. It seems like the locking mechanism is using -S direct_minimal_versions even if it is disabled, resulting in numpy 1.24.4, which is incompatible with python 3.12.

One potential workaround seems to be to lock specific versions of python.

pdm lock --python "==3.8.*"
pdm lock --python "==3.9.*" --append
pdm lock --python 3.10 --append

Now pdm install works, but this fails if the lock file does not exist. What I am supposed to include in pyproject.toml for my package to work in the wild?

Note: pip install seems to do the right thing, so maybe the latter concern is not an issue, but I was hoping to use PDM for everything (i.e. installing code for tests).

To reproduce

pdm init --python 3.12 -n 
sed -i '' "s/==3\.12\.\*/>=3.8,<3.13/g" pyproject.toml  # requires-python = ">=3.8,<3.13"
pdm add numpy

Workaround

This works

rm pdm.lock
pdm lock --python "==3.8.*"
pdm lock --python "==3.9.*" --append
pdm lock --python 3.10 --append
pdm install

Abridged Output

...
⠴ 0:00:00 Resolve for environment (>=3.8,<3.13) 2 resolved, 0 to resolveINFO: Use `-q/--quiet` to suppress these warnings, or ignore them per-package with `ignore_package_warnings` config in [tool.pdm] table.
  0:00:00 🔒 Lock successful.  
Changes are written to pyproject.toml.
Synchronizing working set with resolved packages: 1 to add, 0 to update, 0 to remove

  ✖ Install numpy 1.24.4 failed
  ✖ Install numpy 1.24.4 failed

ERRORS:
add numpy failed:
Traceback (most recent call last):
...
  File ".../pdm/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 402, in _call_hook
    raise BackendUnavailable(
pyproject_hooks._impl.BackendUnavailable: Cannot import 'setuptools.build_meta'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
...
File ".../pdm/lib/python3.11/site-packages/pdm/builders/base.py", line 86, in wrapper
    raise BuildError(str(e)) from e
pdm.exceptions.BuildError: Cannot import 'setuptools.build_meta'

  0:00:02 ✖ Some package operations failed. 1/1
See /Users/mforbes/Library/Logs/pdm/pdm-install-146cih8y.log for detailed debug log.
[InstallationError]: Some package operations failed.
WARNING: Add '-v' to see the detailed traceback

Expected Behavior

PDM to install a version of numpy that works with python 3.12.

Environment Information

PDM version:
  2.23.1
Python Interpreter:
  .../tmp/pdm/bug_numpy/.venv/bin/python (3.12)
Project Root:
  .../tmp/pdm/bug_numpy
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.9",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "23.6.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 23.6.0: Thu Dec 19 20:44:10 PST 2024; root:xnu-10063.141.1.703.2~1/RELEASE_ARM64_T6000",
  "python_full_version": "3.12.9",
  "platform_python_implementation": "CPython",
  "python_version": "3.12",
  "sys_platform": "darwin"
}

Verbose Command Output

After removing the lock file and with "numpy>=1.24.4":

$ pdm install -v
WARNING: Lockfile does not exist
Updating the lock file...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   numpy>=1.24.4
pdm.termui:   Adding requirement python<3.13,>=3.8
pdm.termui:   Adding requirement numpy>=1.24.4
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.2.4 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.2.4, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.2.3 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.2.3, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.2.2 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.2.2, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.2.1 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.2.1, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.2.0 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.2.0, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.1.3 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.1.3, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.1.2 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.1.2, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.1.1 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.1.1, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.1.0 because it requires Python>=3.10 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.1.0, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.10" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.0.2 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.0.2, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.0.1 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.0.1, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@2.0.0 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@2.0.0, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.26.4 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.26.4, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.26.3 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.26.3, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.26.2 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.26.2, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.26.1 because it requires Python<3.13,>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.26.1, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.26.0 because it requires Python<3.13,>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.26.0, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.25.2 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.25.2, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.25.1 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.25.1, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py:190: PackageWarning: Skipping numpy@1.25.0 because it requires Python>=3.9 but the lock targets to work with Python<3.13,>=3.8. Instead, another version of numpy that supports Python<3.13,>=3.8 will be used.
If you want to install numpy@1.25.0, narrow down the `requires-python` range to include this version. For example, "<3.13,>=3.9" should work.
  found = self.repository.find_candidates(
pdm.termui: ======== Starting round 0 ========
pdm.termui: Adding new pin: python None
pdm.termui: ======== Starting round 1 ========
pdm.termui: Adding new pin: numpy 1.24.4
pdm.termui: ======== Starting round 2 ========
pdm.termui: ======== Resolution Result ========
pdm.termui:   python None
pdm.termui:    numpy 1.24.4
INFO: Use `-q/--quiet` to suppress these warnings, or ignore them per-package with `ignore_package_warnings` config in [tool.pdm] table.
pdm.termui: Fetching hashes for numpy@1.24.4
Changes are written to pdm.lock.
STATUS: Resolving packages from lockfile...
Synchronizing working set with resolved packages: 0 to add, 1 to update, 0 to remove

pdm.termui: Overwriting distribution numpy
pdm.termui: Using cached response for https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz
unearth.preparer: Downloading <Link https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz (from https://pypi.org/simple/numpy/)> (10.9 MB)
pdm.termui: Preparing environment(Isolated mode) for PEP 517 build...
pdm.termui: Saving wheel to cache: <Link https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz (from https://pypi.org/simple/numpy/)>
pdm.termui: Running PEP 517 backend to build a wheel for <Link https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz (from https://pypi.org/simple/numpy/)>
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   Adding requirement python==3.12.9
pdm.termui:   Adding requirement Cython<3.0,>=0.29.30
pdm.termui:   Adding requirement wheel==0.37.0
pdm.termui:   Adding requirement setuptools==59.2.0
pdm.termui: ======== Resolution Result ========
pdm.termui:       python None
pdm.termui:   setuptools 59.2.0
pdm.termui:        wheel 0.37.0
pdm.termui:       cython 0.29.37
pdm.termui: Fetching hashes for wheel@0.37.0
pdm.termui: Fetching hashes for setuptools@59.2.0
pdm.termui: Fetching hashes for Cython@0.29.37
pdm.termui: Installing cython@0.29.37...
pdm.termui: Using cached response for https://files.pythonhosted.org/packages/7e/26/9d8de10005fedb1eceabe713348d43bae1dbab1786042ca0751a2e2b0f8c/Cython-0.29.37-py2.py3-none-any.whl
unearth.preparer: Downloading <Link https://files.pythonhosted.org/packages/7e/26/9d8de10005fedb1eceabe713348d43bae1dbab1786042ca0751a2e2b0f8c/Cython-0.29.37-py2.py3-none-any.whl (from https://pypi.org/simple/cython/)> (989 kB)
pdm.termui: Installing setuptools@59.2.0...
pdm.termui: Using cached response for https://files.pythonhosted.org/packages/18/ad/ec41343a49a0371ea40daf37b1ba2c11333cdd121cb378161635d14b9750/setuptools-59.2.0-py3-none-any.whl
unearth.preparer: Downloading <Link https://files.pythonhosted.org/packages/18/ad/ec41343a49a0371ea40daf37b1ba2c11333cdd121cb378161635d14b9750/setuptools-59.2.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/)> (952 kB)
pdm.termui: Installing wheel@0.37.0...
pdm.termui: Using cached response for https://files.pythonhosted.org/packages/04/80/cad93b40262f5d09f6de82adbee452fd43cdff60830b56a74c5930f7e277/wheel-0.37.0-py2.py3-none-any.whl
unearth.preparer: Downloading <Link https://files.pythonhosted.org/packages/04/80/cad93b40262f5d09f6de82adbee452fd43cdff60830b56a74c5930f7e277/wheel-0.37.0-py2.py3-none-any.whl (from https://pypi.org/simple/wheel/)> (35 kB)
pdm.termui: Synchronization complete.
  ✖ Update numpy 2.2.4 -> 1.24.4 failed
pdm.termui: Error occurs updating numpy: 
Traceback (most recent call last):
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/base.py", line 84, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/wheel.py", line 24, in build
    requires = self._hook.get_requires_for_build_wheel(self.config_settings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 196, in get_requires_for_build_wheel
    return self._call_hook(
           ^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 402, in _call_hook
    raise BackendUnavailable(
pyproject_hooks._impl.BackendUnavailable: Cannot import 'setuptools.build_meta'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Volumes/Data/apps/conda_arm64/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 50, in update_candidate
    self.manager.overwrite(dist, can)
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/manager.py", line 63, in overwrite
    installed = self.install(candidate)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    prepared.build(),
    ^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/models/candidates.py", line 417, in build
    self._cached = Path(builder.build(build_dir, metadata_directory=self._metadata_dir))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/base.py", line 86, in wrapper
    raise BuildError(str(e)) from e
pdm.exceptions.BuildError: Cannot import 'setuptools.build_meta'
  Retry failed jobs(2/2)
pdm.termui: Overwriting distribution numpy
pdm.termui: Preparing environment(Isolated mode) for PEP 517 build...
pdm.termui: Saving wheel to cache: <Link https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz (from https://pypi.org/simple/numpy/)>
pdm.termui: Running PEP 517 backend to build a wheel for <Link https://files.pythonhosted.org/packages/a4/9b/027bec52c633f6556dba6b722d9a0befb40498b9ceddd29cbe67a45a127c/numpy-1.24.4.tar.gz (from https://pypi.org/simple/numpy/)>
  ✖ Update numpy 2.2.4 -> 1.24.4 failed
pdm.termui: Error occurs updating numpy: 
Traceback (most recent call last):
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/base.py", line 84, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/wheel.py", line 24, in build
    requires = self._hook.get_requires_for_build_wheel(self.config_settings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 196, in get_requires_for_build_wheel
    return self._call_hook(
           ^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 402, in _call_hook
    raise BackendUnavailable(
pyproject_hooks._impl.BackendUnavailable: Cannot import 'setuptools.build_meta'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Volumes/Data/apps/conda_arm64/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 50, in update_candidate
    self.manager.overwrite(dist, can)
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/manager.py", line 63, in overwrite
    installed = self.install(candidate)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    prepared.build(),
    ^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/models/candidates.py", line 417, in build
    self._cached = Path(builder.build(build_dir, metadata_directory=self._metadata_dir))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/builders/base.py", line 86, in wrapper
    raise BuildError(str(e)) from e
pdm.exceptions.BuildError: Cannot import 'setuptools.build_meta'
  ✖ Some package operations failed.
Traceback (most recent call last):
  File "/data/apps/pipx/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 392, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 270, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 265, in main
    self.handle(project, options)
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 195, in handle
    command.handle(project, options)
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/commands/install.py", line 103, in handle
    actions.do_sync(
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 304, in do_sync
    synchronizer.synchronize()
  File "/Volumes/Data/apps/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 225, in synchronize
    raise InstallationError("Some package operations failed.")
pdm.exceptions.InstallationError: Some package operations failed.

Additional Context

No response

Are you willing to submit a PR to fix this bug?

• Yes, I would like to submit a PR.

[mforbes]

Note: the following pyproject.toml file works:

[project]
name = "tmp"
version = "0.1.0"
description = "Default template for PDM package"
authors = []
requires-python = ">=3.8,<3.13"
dependencies = [
    'numpy >= 1.24.4; python_version == "3.8.*"',
    'numpy >= 2.0.2; python_version == "3.9.*"',
    'numpy >= 2.2.4; python_version >= "3.10"',
]
readme = "README.md"
license = {text = "MIT"}

Frustratingly, pdm lock fails, as does trying to install things (e.g. to use with nox) unless I make the lock file with

pdm lock --python "==3.8.*"
pdm lock --python "==3.9.*" --append
pdm lock --python ">=3.10" --append

[klmcwhirter]

@mforbes , As a fellow developer I am not sure I would expect things to work any differently than what you are experiencing.

Please do wait for an authoritative answer from @frostming .

And please note #3439 which may influence behavior in this area.

But here is how I would think through the problem.

Think about what you are asking pdm to do - support different versions of dependencies for different versions of python. This a many-to-many relationship because the reality is that it is not just 1 dependency it has to evaluate in practice, but dozens. And they will all have a different spec in terms of package version to python version mapping.

The good news is that pdm does have support for this many-to-many mapping via the command line invocations you mention.

This does equate to additional overhead in the CI/CD pipeline, but also in the project code. Because inevitably there will be API differences in one or more of the dependencies for which adapters (used in the project) will need to be written and maintained over time in order to support all the different python versions you are trying to support.

This also means the project would need to contain multiple dependencies lines for each dependency where they are needed. And, as you have noticed, one would need to live with the complexity of managing the lock file and preparing for testing each python version.

I would look closer at the tox-pdm package to see what it can do to help with this or not. Note there is no separate nox-pdm package currently available of which I am aware.

So unless I really had a need to do so (e.g., publishing a widely used library where it is known that all these python versions are in use) - I would avoid it. Note that in my case widely-used means across teams within the large company I work. This is a real world scenario that I live with in a couple cases. Most of the time I am just assertive with my deprecation policies with those other teams though. I have my own budget constraints to contend with.

See EOL dates for python versions - collects dates across multiple python versions; the dates are documented by the Python project as separate PEPs - e.g., PEP-569 - 3.8 Release schedule.

• 3.8 is at EOL already (Oct 2024) - I would drop support for it
• 3.9 is also EOL and only receiving security patches, and in my experience 3.9 is not used by many teams; I would drop support for it as well.

Once that conclusion has been reached then (at least for numpy) my requires-python becomes simply ">=3.10".

Hopefully that also works for the rest of the dependencies and the problem has been wrestled back under control.

But, I do realize that there are cases where an EOL version cannot be replaced as quickly as one would like - other team priorities, lack of project funding, etc.

In that case there are no silver bullets. I would just accept the complexity.

I truly hope this helps and is received in the spirit it is offered. For your sake, I hope you really do not need to support all those older versions.

[mforbes]

@klmcwhirter Thanks for the comments. I agree that ">=3.10" might be fine, though have been bitten in the past by some people trying to use my libraries on older systems.

That being said, I think there are some real issues here: after I have crafted a functioning pyproject.toml with version specifiers, I basically cannot use PDM any more.

Potential bug: Without a lock file, pdm install (fails) is very different from pip install (works). Is this really the intended behaviour?

• Workaround: use a lock file with different python versions and use pip for testing.

Feature request: I cannot use pdm add to add new packages, even when they are supported. For example: pdm add cowsay fails with my pyproject.toml file above.

• Workaround: Unknown.
  A possible solution would be for pdm addto support the--python` flag like lock does:

  pdm add cowsay --python "==3.8.*"
  

I guess I don't understand why, perhaps as a fallback, pdm cannot see the python_version restrictions in pyproject.toml and then do a version-by-version resolution. In its simplest incarnation, anything added to pyproject.toml would have a complete set of markers, but at least pdm add would work.

If these are reasonable, I can close this and open separate issues.

[mforbes]

@klmcwhirter

I truly hope this helps and is received in the spirit it is offered. For your sake, I hope you really do not need to support all those older versions.

The real issue is that I have some packages that work fine with older versions of python. Then a new python version comes out, so I want to just update python_requires, run my tests, change a few simple things (which sometimes requires conditional dependencies -- easy when you are right at the time of the change), bump the minor version and be done.

Then my students tell me I should really be using pdm, testing against locked dependencies, etc. so I bite the bullet, and try to do things "properly". Now my carefully crafted pyproject.toml file becomes a burden and I have to do more work, releasing a new major version because I am now dropping support for those older version of python when there really is no need. And the huge potential benefit of pdm add is completely gone.

As I think about it, it seems like adding --python and --platform selectors for pdm add might be the simplest thing, with the possibility of looping over these with a pdm lock flag. (Yes, there are an exponential number of combinations, but there are a limited set of environment markers, and in most projects, I expect that the full matrix would not be unreasonable.)

[frostming]

First I want to thank @klmcwhirter for the detailed answer, it saves my life as a non-native speaker. His statements are completely correct.

Secondly there is a limitation in pdm's resolver that it can't find multiple solutions in a single lock action. In your case, there is not a numpy version that satisfies all your dependency constraints. That's why you need to run several pdm lock commands with the correct platform parameter to get the correct solution. However, when the lock file doesn't exist, pdm install has to do lock itself, but because it only locks once, it can't get to the viable solution.

Potential bug: Without a lock file, pdm install (fails) is very different from pip install (works). Is this really the intended behaviour?

Of course, pip install doesn't have a lock file and it doesn't even remember what dependencies you have installed before. This can lead to conflict versions easily. Say you run pip install foo(foo depends on numpy<2) and pip install numpy>=2 in a row, you will get a foo package not working at all, while pdm can correctly throws the error and fail the lock`.

[mforbes]

@frostming Thanks for the response. I think I understand everything you have said, but something still does not make sense:

1. Why can't pdm run lock multiple times when there are environmental markers?
2. Whenpdm install is run, it knows the version of python is available (and which platform etc.) so pdm should be able to at least do something that works, adding an appropriate python/platform-specific entry to the lock file.

I.e.: Try resolving on all constraints. If that fails, try a limited resolve on the current platform/python version and lock just with those markers. I assume there is something difficult I am missing here, but I don't see it. Since I can run several pdm lock commands to get a solution, why is it not possible or undesirable for pdm to do this on its own when the overall lock fails? (With a switch of course.)

[klmcwhirter]

@frostming thanks for confirming my thinking ...

@mforbes if you had ever used poetry, you would really appreciate pdm. In my opinion, your students were right to introduce you to pdm.

It sounds like you are teaching (at least in part) less-fortunate kids. If that is the case, kudos to you for doing that. I have one kid (er, young man at this point) that I am still teaching myself. But that is a story for another time ...

A few more things ...

1. Deprecation policy

The trick to a deprecation policy that you can enforce is to be ahead of the need. Spend some time thinking about it. Of course, the goal is to keep your code base simple and limit the amount of maintenance you need to do. It will also help you (or maybe push you) to expose your students to the most recent stable versions of Python. For example, I am working with 3.14.0a6 right now because concurrent.futures.InterpreterPoolExecutor was added. Very cool class.

Establish a policy and publish it on your website, training materials. The students should know going in what to expect.

I have also, in the past, handed out thumb drives (as needed) that have a tiny linux distro with the software at the versions I need them to use. Something to think about

2. Why looping in pdm is not a great idea

At the heart of pdm is a design choice to operate on a single version of Python. You will see this felt in commands like pdm use and pdm python. All of the other commands assume that they can detect the single python version in effect for the project in whatever environment.

For example, when you do pdm init if you do not specify the python interpreter to use it asks you to select ONE from a list.

In the code you will see the PythonInfo class used for this purpose. Here's the code that is part of the python resolving process.

The effects of this decision are pervasive.

However, where does that kind of looping happen? tox or nox !

3. Suggestion for possible solution

With tox (or nox - I only have experience with tox so I will use it in my example) you configure "environments" with which to test. Part of that environment spec is the versions of Python to use.

That is where tox-pdm comes in. tox uses it to create a virtual env, sync the dependencies using the lock file and then run the tests.

Also, if you are using GitHub Actions, there is a setup-pdm action that installs pdm to enable the use of pdm in your actions (no pip needed). It has features to allow caching the python version and venv so the tests can run more efficiently - FYI.

I have a public project that is doing this - here is the workflow definition - https://github.com/klmcwhirter/nvidia-more-battery/blob/master/.github/workflows/tests.yml.

The tox.ini and pyproject.toml files are involved as well.

As tox iterates over the declared env_list (e.g., from the tox.ini in the project I linked above: env_list = py{311,312}) it will:

• use tox-pdm to create the venv using the version of python being tested at the time
• use tox-pdm to call either pdm sync (default - what my project is using) or pdm install (available as an option) - of course, as mentioned above the command being requested will start by detecting the python version configured for the project - when the venv was created
• use tox-pdm to run whatever pdm scripts are requested in tox.ini - again, in the example project those are testcov, flake8, mypy. All three of those are in the [tool.pdm.scripts] section of the pyproject.toml file.

This probably feels a little overwhelming while you are still learning. That is why I provided a working example.

Does that help?

Summary

1. Carefully limit the python versions you support to minimize the amount of work you need to do to maintain your code. You need to be spending your time designing and implementing lesson plans; not maintaining code. The amount of work required to support multiple versions of a tool chain is an exponential function. It is NOT linear! And the problem set only grows worse as time goes on.
2. Continue to use multiple invocations of pdm lock as needed. This only needs to be done one time (by you) to configure the project. And then please do check in the pdm.lock file so your students get the benefit of that work.
3. Use a combination of pdm, tox, tox-pdm and perhaps setup-pdm to accomplish what you are trying to do.
4. And finally, if you have cycles, consider contributing to one or more of the projects involved in producing pdm. Hey, why not as a class project? @frostming will be grateful. He is one guy - with a day job too.

Good luck!

And please do let me know if I can help. Even though I am retired now, I am involved in quite a few FOSS projects. And I do hang out here from time to time ...

[klmcwhirter]

As I think about it, it seems like adding --python and --platform selectors for pdm add might be the simplest thing ...

I agree - it does sound like the simplest for the user. Implementing it does seem challenging though.

Here is where the add command is defined.

This is one area that may need some refactoring - loop over requirements.

The parse_requirements function returns an instance of the Requirement dataclass. It uses that instance as a "key" a few lines below. And it is not immediately obvious to me if the Requirement honors the python_version constraint. If not, that would need to be added in Requirement. And the ripple effect of that change dealt with.

That's what I meant by it seems challenging ... thorough testing of all features would be needed.

[mforbes]

Thanks for the pointers to the code - I have not had a chance to look deeply yet, but I think in principle this should be almost trivial in principle since one can achieve the effect by just running pdm add multiple times on pyproject.toml files changing the requires_python line. Something like this (proof of concept):

PY_VERS="3.8 3.9 3.10 3.11 3.12 3.13 3.14"
PKGS="numpy"
mkdir tmp && cd tmp
pdm init --python "3.12.*" -n
sed -i '' "s/dependencies = \[\]/dependencies = [\n]/g" pyproject.toml
for py in ${PY_VERS}; do
  sed -i '' "s/requires-python = ['\"].*['\"]/requires-python = '==${py}.*'/g" pyproject.toml
  rm -f pdm.lock
  for pkg in ${PKGS}; do
    pdm add "${pkg}" --no-sync
    sed -i '' "s/ \"${pkg}>=\([^\"]*\)\"/ #'${pkg}>=\1; python_version == \"${py}.*\"',/g" pyproject.toml
  done
done

At the end of this, there will be list of the appropriately marked version:

    #'numpy>=1.24.4; python_version == "3.8.*"',
    #'numpy>=2.0.2; python_version == "3.9.*"',
    #'numpy>=2.2.4; python_version == "3.10.*"',
    #'numpy>=2.2.4; python_version == "3.11.*"',
    #'numpy>=2.2.4; python_version == "3.12.*"',
    #'numpy>=2.2.4; python_version == "3.13.*"',

From this it is fairly easy to generate the appropriate pyproject.toml file with environment markers. (Have not tried with platform dependencies yet.)

Is there something fundamentally more challenging that I am missing? It seems that all that would be needed is to use the pinned version of --python from the command line instead of what it picks up from requires_python. (I am not yet sure where this comes in ... probably in add_dependencies?) In any case, this must have be done somewhere with pdm lock.

The question for @frostming is if this functionality is desirable? As you say,

the heart of pdm is a design choice to operate on a single version of Python.

If I take this seriously, then perhaps I am spinning my wheels trying to get pdm to function as a tool for package managers who need to manage their project over multiple versions of python; (but I don't know anything better: tried Poetry, gave up there long ago. There is also an appeal to using/supporting the tool that the standards committee is behind.)

[klmcwhirter]

pdm is well suited for multiple python versions - it has the building blocks. I use it that way all the time. I just have not attempted to support such old versions before; to the extent that I had to specify alternate dependency versions.

But someone has ... otherwise the support for that would not exist.

What I meant by my statement is that within a single invocation of pdm it is expected to refer to a single version of python. The same is true for pip, poetry and uv.

There currently is no feature out there that will analyze all the constraints and for all possible matching versions of python do the thing.

That concept is supported but will take multiple venv creation steps and individual invocations to build up the pdm.lock (soon to be pylock.toml) file.

For example:

• create venv for py38
• use pdm for py38 (e.g., pdm add, install, lock, etc.)
• create venv for py39
• use pdm for py39
• ... though py313

After that process you will have built up the pdm.lock file and can commit it.

Have you thought about trying to write a pdm plugin? That might be a good way to get started.

Or simply write a utility in python that calls pdm in a sub-process. I suspect you will have quicker success with this approach.