A VersionConflict exception when a system installed package conflicts with a pinned requirements package

[atugushev]

Steps to reproduce:

1. pip install setuptools==40.0.0
2. cat strategy.ini:

[Licenses]
authorized_licenses:
        mit

3. cat requirements.txt

setuptools>=41.0.0

4. liccheck -s strategy.ini -r requirements.txt

gathering licenses...
Traceback (most recent call last):
  File "/private/tmp/liccheck/.venv/bin/liccheck", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/liccheck/.venv/lib/python3.8/site-packages/liccheck/command_line.py", line 301, in main
    sys.exit(run(args))
  File "/private/tmp/liccheck/.venv/lib/python3.8/site-packages/liccheck/command_line.py", line 296, in run
    return process(args.requirement_txt_file, strategy, args.level)
  File "/private/tmp/liccheck/.venv/lib/python3.8/site-packages/liccheck/command_line.py", line 231, in process
    pkg_info = get_packages_info(requirement_file)
  File "/private/tmp/liccheck/.venv/lib/python3.8/site-packages/liccheck/command_line.py", line 152, in get_packages_info
    packages = [transform(dist) for dist in pkg_resources.working_set.resolve(requirements)]
  File "/private/tmp/liccheck/.venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 782, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (setuptools 40.0.0 (/private/tmp/liccheck/.venv/lib/python3.8/site-packages), Requirement.parse('setuptools>=41.0.0'))

[ochedru]

Thank you for the bug report and the test case. I am looking into it.

[ochedru]

We are using pkg_resources.working_set to find required packages, but this working set includes what is already installed so that can raise unexpected version conflicts. It would be better to find required packages from a clean environment (kind of venv). Still digging to find a way to achieve that...

[atugushev]

It would be better to find required packages from a clean environment (kind of venv).

FWIW setuptools is installed alongside with every virtual environment (pip and wheel as well).

[atugushev]

The solution could be excluding pip, setuptools, and wheel from passing to pkg_resources.working_set, assuming they are MIT licensed 🤔

UPDATE: or get their licenses from JSON API.

[ochedru]

I love the idea of the JSON API. We would no longer have to hack the pip and pkg_resources APIs. But this is not a trivial update.

[atugushev]

Originally this issue was caught due to another issue with poetry python-poetry/poetry#1584, where poetry wouldn't update setuptools in virtualenv.

I love the idea of the JSON API. We would no longer have to hack the pip and pkg_resources APIs. But this is not a trivial update.

JSON API doesn't show dependencies for packages that list their dependencies in a non-declarative way (e.g., using setup(install_requires=[...])).

The --no-deps option (#48) would help, where resolving dependencies would be a dependency managers' job (e.g., pip-tools/pipenv/poetry/...).

@ochedru I wonder what would you think?

[ochedru]

I am fine with your PR; it looks like we cannot rely on the JSON API and the --no-deps option will help dealing with difficult cases.
So, 👍