new pip resolver "breaks" pip-compile with multiple inputs

[terencehonles]

This was likely a pre-existing issue, and I'm opening this issue to figure out if my expectation is valid and if I should look at making a code change.

Environment Versions

1. Python version: Python 3.8
2. pip version: pip 21.0.1 (python 3.8)
3. pip-tools version: pip-compile, version 6.0.1

Steps to replicate

1. requirements.in: httpx[http2]==0.16.1
2. requirements-test.in: pytest-httpx
3. run: pip-compile -Uo requirements.txt requirements.in requirements-test.in

Expected result

The requirements in requirements-test.in respect requirements.in.

Actual result

The new pip resolver breaks with:

Could not find a version that matches httpx[http2]==0.16.1,==0.17.* from https://files.pythonhosted.org/packages/2d/c6/59aa4188e7eddb9e89ec67a51598ca6bfc09f1b38c9b45f7ee45af7a4df4/httpx-0.16.1-py3-none-any.whl#sha256=9cffb8ba31fac6536f2c8cde30df859013f59e4bcc5b8d43901cb3654a8e0a5b (from -r r.in (line 1))
Tried: 0.6.7, 0.6.8, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.5, 0.7.6, 0.7.6, 0.7.7, 0.7.7, 0.7.8, 0.7.8, 0.8.0, 0.8.0, 0.9.0, 0.9.0, 0.9.1, 0.9.1, 0.9.2, 0.9.2, 0.9.3, 0.9.3, 0.9.4, 0.9.4, 0.9.5, 0.9.5, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 0.12.0, 0.12.0, 0.12.1, 0.12.1, 0.13.0, 0.13.0, 0.13.1, 0.13.1, 0.13.2, 0.13.2, 0.13.3, 0.13.3, 0.14.0, 0.14.0, 0.14.1, 0.14.1, 0.14.2, 0.14.2, 0.14.3, 0.14.3, 0.15.0, 0.15.0, 0.15.1, 0.15.1, 0.15.2, 0.15.2, 0.15.3, 0.15.3, 0.15.4, 0.15.4, 0.15.5, 0.15.5, 0.16.0, 0.16.0, 0.16.1, 0.16.1, 0.17.0, 0.17.0, 0.17.1, 0.17.1
Skipped pre-versions: 0.13.dev0, 0.13.dev0, 0.13.0.dev1, 0.13.0.dev1, 0.13.0.dev2, 0.13.0.dev2
There are incompatible versions in the resolved dependencies:
  httpx[http2]==0.16.1 from https://files.pythonhosted.org/packages/2d/c6/59aa4188e7eddb9e89ec67a51598ca6bfc09f1b38c9b45f7ee45af7a4df4/httpx-0.16.1-py3-none-any.whl#sha256=9cffb8ba31fac6536f2c8cde30df859013f59e4bcc5b8d43901cb3654a8e0a5b (from -r r.in (line 1))
  httpx==0.17.* (from pytest-httpx==0.11.0->-r r-test.in (line 1))

This is because pytest-httpx is unpinned and httpx is then pinned to an incompatible version.

[terencehonles]

A quick test of pip-compile -Uo requirements.txt <(cat requirements.in requirements-test.in) suggests this might be only(?) an issue with the new pip resolver. However, the code which processes the requirements serially

pip-tools/piptools/scripts/compile.py

Lines 338 to 377 in 019516a

	for src_file in src_files:
	is_setup_file = os.path.basename(src_file) in METADATA_FILENAMES
	if src_file == "-":
	# pip requires filenames and not files. Since we want to support
	# piping from stdin, we need to briefly save the input from stdin
	# to a temporary file and have pip read that. also used for
	# reading requirements from install_requires in setup.py.
	tmpfile = tempfile.NamedTemporaryFile(mode="wt", delete=False)
	tmpfile.write(sys.stdin.read())
	comes_from = "-r -"
	tmpfile.flush()
	reqs = list(
	parse_requirements(
	tmpfile.name,
	finder=repository.finder,
	session=repository.session,
	options=repository.options,
	)
	)
	for req in reqs:
	req.comes_from = comes_from
	constraints.extend(reqs)
	elif is_setup_file:
	dist = meta.load(os.path.dirname(os.path.abspath(src_file)))
	comes_from = f"{dist.metadata.get_all('Name')[0]} ({src_file})"
	constraints.extend(
	[
	install_req_from_line(req, comes_from=comes_from)
	for req in dist.requires or []
	]
	)
	else:
	constraints.extend(
	parse_requirements(
	src_file,
	finder=repository.finder,
	session=repository.session,
	options=repository.options,
	)
	)

may also be an issue if/when the pip resolver can handle searching different pytest-httpx versions.

[atugushev]

pip-tools has no proper resolver for now and tries to install the latest versions, which causes a dependency conflict. This should be fixed once we get adopted the new resolver.

However, looks like pytest-httpx has pretty retrictive dependencies. You could open an issue against their tracker and ask to relax restrictions.

[terencehonles]

However, looks like pytest-httpx has pretty retrictive dependencies. You could open an issue against their tracker and ask to relax restrictions.

It's likely because httpx is beta at this point and they don't want to support maintaining older versions. I was able to work around this by manually pushing pytest-httpx backwards, but I expected since pytest-httpx was not pinned pip/pip-compile would try other versions in order to find out what works.

It's not that big of an issue for me right now, but I wanted to make sure this potential issue was documented in the issue tracker and also clarify if multiple requirements files are resolved in parallel or if there was some sort of priority when resolving all of the dependencies. From your answer it seems like this question may still not be answered because installing the latest version instead of resolving a version would likely not matter if requirements files are resolved in parallel or serially.

[terencehonles]

@atugushev quick question (hopefully), I am actually running into this again and I've made some changes to my requirements files to try to see if we can use renovate, and it looks like we're running into something very similar to what I original submitted, but I'm wondering if there is anything I can do to fix this (or if it's really the same problem looking a little different). The renovate bot runs a very specific command and I'm not sure if that needs to be updated, something here can be updated or something in pip needs to be updated.

tail -n +1 requirements*
==> requirements-dev.in <==
pytest-httpx

==> requirements.in <==
# updated by renovate from 0.18.2 (requirements.txt below is from when this was 0.18.2)
httpx[http2]==0.19.0

-r requirements-dev.in

==> requirements.txt <==
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile
#
anyio==3.3.0
    # via httpcore
attrs==21.2.0
    # via pytest
certifi==2021.5.30
    # via httpx
h11==0.12.0
    # via httpcore
h2==3.2.0
    # via httpx
hpack==3.0.0
    # via h2
httpcore==0.13.6
    # via httpx
httpx[http2]==0.18.2
    # via
    #   -r requirements.in
    #   pytest-httpx
hyperframe==5.2.0
    # via h2
idna==3.2
    # via
    #   anyio
    #   rfc3986
iniconfig==1.1.1
    # via pytest
packaging==21.0
    # via pytest
pluggy==0.13.1
    # via pytest
py==1.10.0
    # via pytest
pyparsing==2.4.7
    # via packaging
pytest==6.2.4
    # via pytest-httpx
pytest-httpx==0.12.1
    # via -r requirements-dev.in
rfc3986[idna2008]==1.5.0
    # via httpx
sniffio==1.2.0
    # via
    #   anyio
    #   httpcore
    #   httpx
toml==0.10.2
    # via pytest

With those files I'm running pip-tools==6.2.0 and renovate will be running the command pip-compile and to debug further I used pip-compile -vv which will read requirements.in and write requirements.txt. Looking at the output from pip/pip-compile (I'm not sure which it is from honestly) I see:

> pip-compile -vv
Using indexes:
  https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  httpx[http2]==0.19.0 (from -r requirements.in (line 1))
  pytest-httpx (from -r requirements-dev.in (line 1))

Finding the best candidates:
  found candidate httpx[http2]==0.19.0 (constraint was ==0.19.0)
  found candidate pytest-httpx==0.12.1 (constraint was <any>)

Finding secondary dependencies:
  pytest-httpx==0.12.1      requires httpx==0.18.*, pytest==6.*
  httpx[http2]==0.19.0      requires certifi, charset-normalizer, h2<5,>=3; extra == "http2", httpcore<0.14.0,>=0.13.3, rfc3986[idna2008]<2,>=1.3, sniffio

New dependencies found in this round:
  adding ('certifi', '', [])
  adding ('charset-normalizer', '', [])
  adding ('h2', '<5,>=3', [])
  adding ('httpcore', '<0.14.0,>=0.13.3', [])
  adding ('httpx', '==0.18.*', [])
  adding ('pytest', '==6.*', [])
  adding ('rfc3986', '<2,>=1.3', ['idna2008'])
  adding ('sniffio', '', [])
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  certifi (from httpx[http2]==0.19.0->-r requirements.in (line 1))
  charset-normalizer (from httpx[http2]==0.19.0->-r requirements.in (line 1))
  h2<5,>=3 (from httpx[http2]==0.19.0->-r requirements.in (line 1))
  httpcore<0.14.0,>=0.13.3 (from httpx[http2]==0.19.0->-r requirements.in (line 1))
  httpx[http2]==0.18.*,==0.19.0 (from -r requirements.in (line 1))
  pytest==6.* (from pytest-httpx==0.12.1->-r requirements-dev.in (line 1))
  pytest-httpx (from -r requirements-dev.in (line 1))
  rfc3986[idna2008]<2,>=1.3 (from httpx[http2]==0.19.0->-r requirements.in (line 1))
  sniffio (from httpx[http2]==0.19.0->-r requirements.in (line 1))

Finding the best candidates:
  found candidate certifi==2021.5.30 (constraint was <any>)
  found candidate charset-normalizer==2.0.4 (constraint was <any>)
  found candidate h2==3.2.0 (constraint was >=3,<5)
  found candidate httpcore==0.13.6 (constraint was >=0.13.3,<0.14.0)
Could not find a version that matches httpx[http2]==0.18.*,==0.19.0 (from -r requirements.in (line 1))
Tried: 0.6.7, 0.6.8, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.5, 0.7.6, 0.7.6, 0.7.7, 0.7.7, 0.7.8, 0.7.8, 0.8.0, 0.8.0, 0.9.0, 0.9.0, 0.9.1, 0.9.1, 0.9.2, 0.9.2, 0.9.3, 0.9.3, 0.9.4, 0.9.4, 0.9.5, 0.9.5, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 0.12.0, 0.12.0, 0.12.1, 0.12.1, 0.13.0, 0.13.0, 0.13.1, 0.13.1, 0.13.2, 0.13.2, 0.13.3, 0.13.3, 0.14.0, 0.14.0, 0.14.1, 0.14.1, 0.14.2, 0.14.2, 0.14.3, 0.14.3, 0.15.0, 0.15.0, 0.15.1, 0.15.1, 0.15.2, 0.15.2, 0.15.3, 0.15.3, 0.15.4, 0.15.4, 0.15.5, 0.15.5, 0.16.0, 0.16.0, 0.16.1, 0.16.1, 0.17.0, 0.17.0, 0.17.1, 0.17.1, 0.18.0, 0.18.0, 0.18.1, 0.18.1, 0.18.2, 0.18.2, 0.19.0, 0.19.0
Skipped pre-versions: 0.13.dev0, 0.13.dev0, 0.13.0.dev1, 0.13.0.dev1, 0.13.0.dev2, 0.13.0.dev2
There are incompatible versions in the resolved dependencies:
  httpx[http2]==0.19.0 (from -r requirements.in (line 1))
  httpx==0.18.* (from pytest-httpx==0.12.1->-r requirements-dev.in (line 1))

Everything looks good until:

Finding the best candidates:
  found candidate httpx[http2]==0.19.0 (constraint was ==0.19.0)
  found candidate pytest-httpx==0.12.1 (constraint was <any>)

pytest-httpx==0.13.0 has been released and that is actually the best candidate. When seeing this I realized that it looks like pip-compile is using the existing requirements.txt file and pinning to the last installed version of pytest-httpx instead of the new one which would work. I can manually specify -U/--upgrade but that will update many more packages than I'm expecting and renovate does not specify that regardless if it should be specified. I see there's a -P/--upgrade-package but I'm not sure how that's supposed to be used and how renovate would include that to work for the general case.

In case you're interested here's the renovate feature request for supporting pip-tools renovatebot/renovate#2334 and I'm looking for how I might proceed with this. I'm trying to avoid writing my own package updater, and I was hoping for our Python code I could at least use renovate (It's looking somewhat tricky for our Angular project), but it's possible I'll not be able to and supplying -U would be something I could do with my own bespoke package updater if needed.

[atugushev]

The upcoming backtracking resolver #1539 fixes the issue.

Details

$ pip-compile --resolver backtracking
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --resolver=backtracking
#
attrs==21.4.0
    # via pytest
certifi==2021.10.8
    # via httpx
h11==0.13.0
    # via httpcore
h2==3.2.0
    # via httpx
hpack==3.0.0
    # via h2
httpcore==0.12.3
    # via httpx
httpx[http2]==0.16.1
    # via
    #   -r requirements.in
    #   pytest-httpx
hyperframe==5.2.0
    # via h2
idna==3.3
    # via rfc3986
iniconfig==1.1.1
    # via pytest
packaging==21.3
    # via pytest
pluggy==1.0.0
    # via pytest
py==1.11.0
    # via pytest
pyparsing==3.0.7
    # via packaging
pytest==6.2.5
    # via pytest-httpx
pytest-httpx==0.10.1
    # via -r requirements.in
rfc3986[idna2008]==1.5.0
    # via httpx
sniffio==1.2.0
    # via
    #   httpcore
    #   httpx
toml==0.10.2
    # via pytest

[atugushev]

This has been fixed in #1539 with the backtracking resolver, try pip-compile --resolver backtracking. The resolver is released as part of pip-tools v6.8.0. Please let us know if it doesn't resolve your issue. Thanks!