pip-compile breaks with Pip 22.1.1 - AttributeError: 'RequirementSet' object has no attribute 'add_requirement'.

[kbarnes3]

Running pip-compile will break when using the latest Pip release, 22.1.1.

This commit appears to be the cause of the break.

Environment Versions

1. OS Type: Windows 11, fully up to date
2. Python version: Python 3.10.4
3. pip version: pip 22.1.1 from %MyProject%\venv\lib\site-packages\pip (python 3.10)
4. pip-tools version: pip-compile, version 6.6.1

Steps to replicate

0. These directions are using PowerShell. They should work on other shells as well with some adjustments
1. Create a fresh virtualenv and activate it
2. python -m pip install pip==22.1.1
3. python -m pip install pip-tools==6.6.1
4. Set-Content -Value 'django' -Path .\example.in
5. pip-compile .\example.in

Expected result

pip-compile produces a requirements.txt file without error

Actual result

No requirements.txt file is produced. This callstack is shown

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "%MyProject%\venv\Scripts\pip-compile.exe\__main__.py", line 7, in <module>
  File "%MyProject%\venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "%MyProject%\venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "%MyProject%\venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "%MyProject%\venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "%MyProject%\venv\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "%MyProject%\venv\lib\site-packages\piptools\scripts\compile.py", line 466, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "%MyProject%\venv\lib\site-packages\piptools\resolver.py", line 175, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "%MyProject%\venv\lib\site-packages\piptools\resolver.py", line 319, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "%MyProject%\venv\lib\site-packages\piptools\resolver.py", line 428, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "%MyProject%\venv\lib\site-packages\piptools\repositories\pypi.py", line 242, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(
  File "%MyProject%\venv\lib\site-packages\piptools\repositories\pypi.py", line 192, in resolve_reqs
    reqset.add_requirement(ireq)
AttributeError: 'RequirementSet' object has no attribute 'add_requirement'. Did you mean: 'all_requirements'?

[kbarnes3]

More of a general question, but this is the second Pip update in 9 days that has rendered pip-tools unusable. Is it a bad idea to be always updating Pip to the latest release?

[akuzminsky]

More of a general question, but this is the second Pip update in 9 days that has rendered pip-tools unusable. Is it a bad idea to be always updating Pip to the latest release?

IMO it's not pip-tool's fault. pip didn't follow semantic versioning convention when a) it removed a method (=backwards incompatible change) and b) it didn't upgrade the major version.

Under these circumstances the only reliable way to go is to pin to an exact pip version (which is not a good idea anyway).

[kbarnes3]

Sorry if I came across as blaming pip-tools for this, I recognize that the pip project is the one that made an incompatible change in both these instances. However, I haven't found great guidance on how to avoid things like this. Even if pip upgraded the major version for this change, the pip-tools README states that pip-tools 6.5+ is compatible with all versions of pip greater than or equal to 21.2. So based on this information, it would be reasonable to expect pip-tools to work with this new pip version, even it has a bigger version number than it actually did.

[johnthagen]

I feel like between this and #1617, there needs to be a fundamental change in how pip and pip-tools are tested/released to make this sustainable.

Some ideas:

• Put pip-tools testing into pip's CI so that pip cannot be released if it breaks the latest pip-tools
• Vendor the important parts of pip into pip-tools
• Move pip-tools into the pip repo and version/release them together

For users, there are also alternative locking solutions that don't depend on pip internals, such as Poetry to consider?

[starhel]

IMO it's not pip-tool's fault. pip didn't follow semantic versioning convention when a) it removed a method (=backwards incompatible change) and b) it didn't upgrade the major version.

pip-tools uses internals (from pip._internal.req import InstallRequirement, RequirementSet), so any versioning convention will not work here. pip is written as command line tool, not library.

Under these circumstances the only reliable way to go is to pin to an exact pip version (which is not a good idea anyway).

Maybe vendoring the important parts of pip is solution.

[adamchainz]

Would it work to run CI with pip's main branch daily, and learn about breakage before pip releases?

[atugushev]

Would it work to run CI with pip's main branch daily, and learn about breakage before pip releases?

We have the daily job for testing pip-tools against pip's main branch. We only need volunteers who can track and address failed jobs shortly.

[scolebrook]

Still experiencing this exception. pip-tools 6.6.2 and pip 22.1.1 installed. Python 3.9.12. But pip-compile --version shows version 6.6.1.

The change in https://github.com/jazzband/pip-tools/pull/1624/files is not in my installed copy. Manually adding that change gets pip-compile working.

Is pip-compile 6.6.1 supposed to be in the 6.6.2 distribution of the pip-tools package? Is that fix in pull 1624 supposed to be in pip-tools 6.6.2?

[Czaki]

Is pip-compile 6.6.1 supposed to be in the 6.6.2 distribution of the pip-tools package? Is that fix in pull 1624 supposed to be in pip-tools 6.6.2?

verify if you run pip-compile from proper environment.

[scolebrook]

@Czaki Very nice catch. I looked in the paths in exception and it was executing pip-compile from a global installation instead of the active virtualenv. The virtual env had 6.6.2 but the global was 6.6.1.

Thanks for pointing me in the right direction to find the root problem.

[AndydeCleyre]

I just made pip-and-pip-tools, for use in my own CI dependencies.

[shacker]

This just cropped up in a slightly different way today, on pip-compile 6.12.3 and pip 23.1.2

AttributeError: 'InstallRequirement' object has no attribute 'install_options'. Did you mean: 'global_options'?

Not sure what to do other than to downgrade both.

[atugushev]

@shacker upgrade both to the latest versions. See #1854.