Unexpected conflict resolution for flake / flake8-isort

[jaap3]

Description

With a requirements.txt listing only flake8 and flake8-isort pip will install a combination of packages that doesn't work.

The most recent version of flake8-isort pins flake8<4, the most recent version of flake8 is ~4.0. Instead of installing an older version of flake8, pip installs and older version of flake8-isort that doesn't pin flake8, but is not compatible with the rest of the installed dependencies.

Expected behavior

I'd expect pip to honor the pin of flake8-isort instead of looking for an older version that has not yet pinned the list of dependencies.

pip version

21.2.4

Python version

3.8.12

OS

macOS 10.15.7

How to Reproduce

$ pip install --no-cache-dir --progress-bar=off flake8 flake8-isort
[...]
$ flake8
[...]
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "I00" due to cannot import name 'SortImports' from 'isort'

Output

$ pip install --no-cache-dir --progress-bar=off flake8 flake8-isort
Collecting flake8
  Downloading flake8-4.0.0-py2.py3-none-any.whl (64 kB)
Collecting flake8-isort
  Downloading flake8_isort-4.0.0-py2.py3-none-any.whl (14 kB)
Collecting pyflakes<2.5.0,>=2.4.0
  Downloading pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)
Collecting pycodestyle<2.9.0,>=2.8.0
  Downloading pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting testfixtures<7,>=6.8.0
  Downloading testfixtures-6.18.3-py2.py3-none-any.whl (95 kB)
Collecting isort<6,>=4.3.5
  Downloading isort-5.9.3-py3-none-any.whl (106 kB)
Collecting flake8
  Downloading flake8-3.9.2-py2.py3-none-any.whl (73 kB)
  Downloading flake8-3.9.1-py2.py3-none-any.whl (73 kB)
  Downloading flake8-3.9.0-py2.py3-none-any.whl (73 kB)
  Downloading flake8-3.8.4-py2.py3-none-any.whl (72 kB)
  Downloading flake8-3.8.3-py2.py3-none-any.whl (72 kB)
  Downloading flake8-3.8.2-py2.py3-none-any.whl (72 kB)
  Downloading flake8-3.8.1-py2.py3-none-any.whl (72 kB)
  Downloading flake8-3.8.0-py2.py3-none-any.whl (72 kB)
  Downloading flake8-3.7.9-py2.py3-none-any.whl (69 kB)
  Downloading flake8-3.7.8-py2.py3-none-any.whl (70 kB)
  Downloading flake8-3.7.7-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.7.6-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.7.5-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.7.4-py2.py3-none-any.whl (69 kB)
  Downloading flake8-3.7.3-py2.py3-none-any.whl (69 kB)
  Downloading flake8-3.7.2-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.7.1-py2.py3-none-any.whl (70 kB)
  Downloading flake8-3.7.0-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.6.0-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.5.0-py2.py3-none-any.whl (69 kB)
  Downloading flake8-3.4.1-py2.py3-none-any.whl (68 kB)
  Downloading flake8-3.4.0-py2.py3-none-any.whl (67 kB)
  Downloading flake8-3.3.0-py2.py3-none-any.whl (66 kB)
  Downloading flake8-3.2.1-py2.py3-none-any.whl (66 kB)
INFO: pip is looking at multiple versions of flake8-isort to determine which version is compatible with other requirements. This could take a while.
Collecting flake8-isort
  Downloading flake8_isort-3.0.1-py2.py3-none-any.whl (13 kB)
Collecting isort[pyproject]<5,>=4.3.5
  Downloading isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting flake8-isort
  Downloading flake8_isort-3.0.0-py2.py3-none-any.whl (13 kB)
WARNING: isort 5.9.3 does not provide the extra 'pyproject'
Installing collected packages: pyflakes, pycodestyle, mccabe, isort, testfixtures, flake8, flake8-isort
Successfully installed flake8-4.0.0 flake8-isort-3.0.0 isort-5.9.3 mccabe-0.6.1 pycodestyle-2.8.0 pyflakes-2.4.0 testfixtures-6.18.3
$ flake8
Traceback (most recent call last):
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 161, in load_plugin
    self._load()
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 138, in _load
    self._plugin = self.entry_point.load()
  File "/Users/jroes/.pyenv/versions/3.8.12/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Users/jroes/.pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/path/to/lib/python3.8/site-packages/flake8_isort.py", line 3, in <module>
    from isort import SortImports
ImportError: cannot import name 'SortImports' from 'isort' (/path/to/lib/python3.8/site-packages/isort/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jroes/.pyenv/versions/bugbear/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/path/to/lib/python3.8/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/path/to/lib/python3.8/site-packages/flake8/main/application.py", line 375, in run
    self._run(argv)
  File "/path/to/lib/python3.8/site-packages/flake8/main/application.py", line 363, in _run
    self.initialize(argv)
  File "/path/to/lib/python3.8/site-packages/flake8/main/application.py", line 343, in initialize
    self.find_plugins(config_finder)
  File "/path/to/lib/python3.8/site-packages/flake8/main/application.py", line 163, in find_plugins
    self.check_plugins.load_plugins()
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 422, in load_plugins
    plugins = list(self.manager.map(load_plugin))
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 309, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 420, in load_plugin
    return plugin.load_plugin()
  File "/path/to/lib/python3.8/site-packages/flake8/plugins/manager.py", line 168, in load_plugin
    raise failed_to_load
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "I00" due to cannot import name 'SortImports' from 'isort' (/path/to/lib/python3.8/site-packages/isort/__init__.py).

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[jaap3]

Putting flake8-isort before flake8 in the list of packages to install results in pip honoring the pinned version of flake8 in flake8-isort. So it seems that the order in which packages are listed is significant.

[pfmoore]

Both combinations are valid. You say the older flake8-isort results in a version of flake8 that "is not compatible with the rest of the installed dependencies", but pip does not consider dependencies of already-installed packages in this situation - see #7744 for (extensive) discussion of this.

The order in which packages is listed influences the search order the resolver uses. When multiple "correct" solutions exist, this may influence which one is found first (and hence used).

[jaap3]

pip doesn't install an older version of flake8 though. It downloads a bunch of older version, but then installs 4.0 anyway.

The output I posted in my report is an installation in an empty virtualenv. Nothing is installed yet. Here's the output again with some of the "noise" cut out:

Collecting flake8
  Downloading flake8-4.0.0-py2.py3-none-any.whl (64 kB)
Collecting flake8-isort
  Downloading flake8_isort-4.0.0-py2.py3-none-any.whl (14 kB)
[...]
Collecting flake8
  Downloading flake8-3.9.2-py2.py3-none-any.whl (73 kB)
  [...]
  Downloading flake8-3.2.1-py2.py3-none-any.whl (66 kB)
INFO: pip is looking at multiple versions of flake8-isort to determine which version is compatible with other requirements. This could take a while.
Collecting flake8-isort
  Downloading flake8_isort-3.0.1-py2.py3-none-any.whl (13 kB)
INFO: pip is looking at multiple versions of flake8-isort to determine which version is compatible with other requirements. This could take a while.
Collecting flake8-isort
  Downloading flake8_isort-3.0.1-py2.py3-none-any.whl (13 kB)
Collecting isort[pyproject]<5,>=4.3.5
  Downloading isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting flake8-isort
  Downloading flake8_isort-3.0.0-py2.py3-none-any.whl (13 kB)
WARNING: isort 5.9.3 does not provide the extra 'pyproject'
Installing collected packages: pyflakes, pycodestyle, mccabe, isort, testfixtures, flake8, flake8-isort
Successfully installed flake8-4.0.0 flake8-isort-3.0.0 isort-5.9.3 mccabe-0.6.1 pycodestyle-2.8.0 pyflakes-2.4.0 testfixtures-6.18.3

If I'm following things correctly this happens:

• pip is asked to install flake8 and flake8-isort
• pip downloads the most recent versions of both packages
• pip figures out that flake8-isort forbids flake8~=4.0
• pip downloads older versions of flake8
• pip changes its mind and decides that maybe there is an older version of flake8-isort is somehow compatible with flake8~=4.0
• pip finds an old version of flake8-isort that doesn't pin dependencies
• pip installs and old version of flake8-isort with the most recent version of flake8

The first bit of this seems reasonable to me. The thing that confuses me is the decision to look at older versions of flake8-isort, even though the most recent version already declared itself to not be compatible with flake8~=4.0.

[pfmoore]

I imagine there's a conflict somewhere. I'd have to pull out the whole dependency tree to determine where. I don't have time to do that right now - I'll see if I can find some time later.

However, the key point here is that what pip installed is a solution to the requirements you gave. It's not the only solution, but we don't make any commitment as to which solution we find, just that we find one. As far as I'm aware, there aren't any algorithms for solving dependency issues while also guaranteeing to get the "best" solution under some particular metric (apart from getting all solutions and then picking one, but that's really expensive).

One thing you can do - if you know you want the latest version of flake8-isort, you can just specify that explicitly.

[Peque]

It seems this will be fixed (hopefully shortly) in flake8-isort: gforcada/flake8-isort#104

The issue was reported here: gforcada/flake8-isort#103

[uranusjr]

The first bit of this seems reasonable to me. The thing that confuses me is the decision to look at older versions of flake8-isort, even though the most recent version already declared itself to not be compatible with flake8~=4.0.

This is a common pitfall for us humans to wrap our heads around dependency resolution. The fact that the latest flake8-isort declares incompatibility against flake8~=4.0 does not in any way imply older versions have this incompatibility. We tend to assume that because we already know they are incompatible, but it's also equally easy to imagine a scenario that

• b depends on a.
• a releases a new major version (say 2.0) that does not cause any compatibility issues for the current version of b (say 1.0).
• A new b version (say 2.0) introduces a new feature that does not work on a 2.0 and therefore adds a<2.0 to its dependencies.

In this case, for pip install a b where the user directly both uses a and b in their code, not selecting the higher b over a would be unexpected since the fact that there's no explicit b>=2.0 specification implies the user does not care about that new feature introduced in b 2.0.

The bottom line is, where there are multiple possible solutions to a dependency specification, there's no way for pip to tell exact what solution is preferred, so you need to supply more hints to it. There are a few of ways to do that right now:

1. If you don't actually use flake8 directly in your code, remove that from your requirements.txt (probably does not apply in this case, but useful in some situations). This makes pip prefer higher versions of flake8-isort over flake8 and would select the solution you want.
2. Move flake8 after flake8-isort in your requirements.txt. This has a similar effect as the previous approach.
3. As mentioned previously, add a version range to flake8-isort so pip can better understand what you want.

I'm going to close this for now since there isn't really a good way to "fix" this. There are maybe some potential hacks we could try (e.g. we can sort of "tell" flake8 is a dependency of flake8-isort from prefix, but even with that information it's unclear why we should prefer either), so it's probably to instead wait for a concrete proposal instead of having an issue open with action items available.