Conflict between editable wheel and constraints #11718

stephenfin · 2023-01-10T12:59:02Z

Description

PEP-660 describes a mechanism to support editable installs by building and installing editable wheels. This will replace the legacy setup.py develop path.

This is behaviour that pip itself will most likely eventually default to itself (per #11457). It can be opted into today using the --use-pep517 flag. tox 4, meanwhile, utilises the pyproject-api project, a separate PEP-517 build frontend, and is already implementing this behaviour, which is where this issue was first spotted.

Unfortunately it appears there is an issue combining editable wheels and constraints file. Where the package being installed in editable mode is also listed in a constraints file (passed to pip install via the -c flag), pip's dependency resolver will raise a ResolutionImpossible error. Given this does not happen with the setup.py develop path, this seems to be a regression. This is already causing issues with tox 4 and when pip makes editable wheels the default, it will no longer be possible to test new versions of a given package using a single unified constraints file. This is something the OpenStack community, for one, relies on heavily.

Expected behavior

The constraint for the package being installed in editable mode should be ignored, ideally by default or else via a user-configurable flag.

pip version

22.3.1

Python version

3.11.1

OS

Fedora 37

How to Reproduce

❯ git clone https://github.com/openstack/os-vif
❯ cd os-vif
❯ virtualenv .venv
❯ python -m pip install -c https://releases.openstack.org/constraints/upper/master --use-pep517 -e .

You can also achieve this using tox 4. You'll need to modify the tox.ini file and set install_command so that tox uses this to install the package itself, rather than just the dependencies.

❯ git clone https://github.com/openstack/os-vif
❯ cd os-vif
❯ virtualenv .venv
❯ source .venv/bin/activate
❯ pip install 'tox>=4'
❯ cat > test.patch << EOF
diff --git a/tox.ini b/tox.ini
index e7b402c..9abe705 100644
--- a/tox.ini
+++ b/tox.ini
@@ -5,9 +5,8 @@ envlist = py3,pep8,docs,releasenotes,cover
 [testenv]
 usedevelop = True
 setenv =
+install_command = python -m pip install -c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} {opts} {packages}
 deps =
-  -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-  -r{toxinidir}/requirements.txt
   -r{toxinidir}/test-requirements.txt
 commands = stestr run --exclude-regex ".tests.functional" {posargs}
 
@@ -20,7 +19,6 @@ commands =
 
 [testenv:docs]
 deps =
-  -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
   -r{toxinidir}/doc/requirements.txt
 commands = sphinx-build -W -b html doc/source doc/build/html
 
EOF
❯ git apply test.patch
❯ tox -e py38

Output

Obtaining file:///tmp/os-vif
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
ERROR: Cannot install os-vif 3.1.1.dev3 (from /tmp/os-vif) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested os-vif 3.1.1.dev3 (from /tmp/os-vif)
    The user requested (constraint) os-vif===3.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

stephenfin · 2023-01-10T13:34:55Z

Hmm, actually, tell a lie: this also happens currently without without either -e or --use-pep517.

❯ python -m pip install -c https://releases.openstack.org/constraints/upper/master --use-pep517 .
Processing /tmp/os-vif
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Cannot install os-vif 3.1.1.dev3 (from /tmp/os-vif) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested os-vif 3.1.1.dev3 (from /tmp/os-vif)
    The user requested (constraint) os-vif===3.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

❯ python -m pip install -c https://releases.openstack.org/constraints/upper/master .
Processing /tmp/os-vif
  Preparing metadata (setup.py) ... done
ERROR: Cannot install os-vif 3.1.1.dev3 (from /tmp/os-vif) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested os-vif 3.1.1.dev3 (from /tmp/os-vif)
    The user requested (constraint) os-vif===3.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Looking further, it seems the reason we're seeing this behavior change is because tox 3 was not respecting the install_command value for installing the package itself. With tox 3:

❯ tox -e py38 -v
using tox.ini: /tmp/os-vif/tox.ini (pid 136252)
using tox-3.28.0 from /usr/lib/python3.11/site-packages/tox/__init__.py (pid 136252)
py38 cannot reuse: no previous config /tmp/os-vif/.tox/py38/.tox-config1
py38 create: /tmp/os-vif/.tox/py38
[136264] /tmp/os-vif/.tox$ /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3.8 py38 >py38/log/py38-0.log
py38 installdeps: -chttps://releases.openstack.org/constraints/upper/master, -r/tmp/os-vif/requirements.txt, -r/tmp/os-vif/test-requirements.txt
[136272] /tmp/os-vif$ /tmp/os-vif/.tox/py38/bin/python -m pip install -chttps://releases.openstack.org/constraints/upper/master -r/tmp/os-vif/requirements.txt -r/tmp/os-vif/test-requirements.txt >.tox/py38/log/py38-1.log
py38 develop-inst: /tmp/os-vif
write config to /tmp/os-vif/.tox/py38/.tox-config1 as '380c2a39af7b4a1f711ed12e65c7807ae74d2bd0291eb1a73c0eaf1de0941f21 /usr/bin/python3.8\n3.28.0 0 1 0\n00000000000000000000000000000000 -chttps://releases.openstack.org/constraints/upper/master\n00000000000000000000000000000000 -r/tmp/os-vif/requirements.txt\n00000000000000000000000000000000 -r/tmp/os-vif/test-requirements.txt'
[136306] /tmp/os-vif$ /tmp/os-vif/.tox/py38/bin/python -m pip install --exists-action w -e . >.tox/py38/log/py38-2.log
[136339] /tmp/os-vif$ /tmp/os-vif/.tox/py38/bin/python -m pip freeze >.tox/py38/log/py38-3.log
...
  py38: commands succeeded
  congratulations :)

Compared with tox 4:

❯ tox -e py38
py38: install_deps> python -m pip install -chttps://releases.openstack.org/constraints/upper/master -r /tmp/os-vif/test-requirements.txt
.pkg-cpython38: _optional_hooks> python /tmp/os-vif/.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython38: get_requires_for_build_editable> python /tmp/os-vif/.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython38: build_editable> python /tmp/os-vif/.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py38: install_package_deps> python -m pip install -chttps://releases.openstack.org/constraints/upper/master 'debtcollector>=1.19.0' 'netaddr>=0.7.18' 'oslo.concurrency>=3.20.0' 'oslo.config>=5.1.0' 'oslo.i18n>=3.15.3' 'oslo.log>=3.30.0' 'oslo.privsep>=1.23.0' 'oslo.versionedobjects>=1.28.0' 'ovsdbapp>=0.12.1' 'pbr!=2.1.0,>=2.0.0' 'pyroute2>=0.5.2' 'stevedore>=1.20.0'
py38: install_package> python -m pip install -chttps://releases.openstack.org/constraints/upper/master --force-reinstall --no-deps /tmp/os-vif/.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl
Processing ./.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl

The conflict is caused by:
    The user requested os-vif 3.1.1.dev4 (from /tmp/os-vif/.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl)
    The user requested (constraint) os-vif===3.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: Cannot install os-vif 3.1.1.dev4 (from /tmp/os-vif/.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl) because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

py38: exit 1 (1.58 seconds) /tmp/os-vif> python -m pip install -chttps://releases.openstack.org/constraints/upper/master --force-reinstall --no-deps /tmp/os-vif/.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl pid=137364
.pkg-cpython38: _exit> python /tmp/os-vif/.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  py38: FAIL code 1 (18.59 seconds)
  evaluation failed :( (18.67 seconds)

The important pieces being these lines:

[136306] /tmp/os-vif$ /tmp/os-vif/.tox/py38/bin/python -m pip install --exists-action w -e . >.tox/py38/log/py38-2.log

py38: install_package> python -m pip install -chttps://releases.openstack.org/constraints/upper/master --force-reinstall --no-deps /tmp/os-vif/.tox/.tmp/package/3/os_vif-3.1.1.dev4-0.editable-py3-none-any.whl

So this is not in fact a regression but closer to a feature request.

uranusjr · 2023-01-10T13:36:47Z

Duplicate of #7839

Constraint files are often shared across an organization. When developing a package included in such a constraint file, it is not possible to install the package with constraints since the constraint on the package prevents us installing a development version. ❯ cd my-amazing-package ❯ cat constraints.txt my-amazing-package==1.2.3 Jinja2==3.1.2 iso8601==1.1.0 msgpack==1.0.4 ❯ pip install -c constraints.txt . Processing /dev/my-amazing-package Preparing metadata (setup.py) ... done ERROR: Cannot install my-amazing-package 1.2.4.dev1 (from /dev/my-amazing-package) because these package versions have conflicting dependencies. The conflict is caused by: The user requested my-amazing-package 1.2.4.dev1 (from /dev/my-amazing-package) The user requested (constraint) my-amazing-package===1.2.4.dev1 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts Resolve this by allowing users to opt out of individual constraints to the 'install', 'wheel', and 'download' subcommands. This is rather manual but it's expected that tools like tox could automatically generate a value for this option when invoking 'pip install' command. ❯ pip install -c constraints.txt --ignore-constraint my-amazing-package . ❯ pip wheel -c constraints.txt --ignore-constraint my-amazing-package . ❯ pip download -c constraints.txt --ignore-constraint my-amazing-package . This is only added for the '2020-resolver' resolver, not the 'legacy-resolver' resolver, given the latter is deprecated for removal. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Fixes: pypa#11718

stephenfin added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jan 10, 2023

uranusjr marked this as a duplicate of #7839 Jan 10, 2023

uranusjr closed this as not planned Won't fix, can't repro, duplicate, stale Jan 10, 2023

uranusjr added resolution: duplicate Duplicate of an existing issue/PR and removed type: bug A confirmed bug or unintended behavior S: needs triage Issues/PRs that need to be triaged labels Jan 10, 2023

This was referenced Jan 10, 2023

Flag to ignore constraint for installing . #7839

Open

Add '--ignore-constraint' option #11723

Open

github-actions bot locked as resolved and limited conversation to collaborators Feb 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conflict between editable wheel and constraints #11718

Conflict between editable wheel and constraints #11718

stephenfin commented Jan 10, 2023 •

edited

stephenfin commented Jan 10, 2023 •

edited

uranusjr commented Jan 10, 2023

Conflict between editable wheel and constraints #11718

Conflict between editable wheel and constraints #11718

Comments

stephenfin commented Jan 10, 2023 • edited

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

stephenfin commented Jan 10, 2023 • edited

uranusjr commented Jan 10, 2023

stephenfin commented Jan 10, 2023 •

edited

stephenfin commented Jan 10, 2023 •

edited