Installing local packages with transitive dependencies and extras not working

[mariojonke]

Pip (version 20.3+) seems to have problems resolving dependencies when trying to install packages from the filesystem with transitive dependencies and extras defined if for these packages also a distribution package on pypi exists.

Lets say there are the packages with dependencies defined by the following setup.py files:

import setuptools

setuptools.setup(
    name="opentelemetry-api",
    install_requires=[],
    extras_require={"test": []},
    version="0.17.dev0",
)

import setuptools

setuptools.setup(
    name="opentelemetry-sdk",
    install_requires=["opentelemetry-api == 0.17.dev0"],
    extras_require={"test": []},
    version="0.17.dev0",
)

import setuptools

setuptools.setup(
    name="opentelemetry-instrumentation",
    install_requires=[
        "opentelemetry-api == 0.17.dev0",
        "opentelemetry-sdk == 0.17.dev0",
    ],
    extras_require={"test": []},
    version="0.17.dev0",
)

These setup.py files refer to the local development version which does not exist on pypi. However, there exist release versions of these packages on pypi.

When now trying to install these packages locally from the filesystem with the extras specifier

 pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]' -e './opentelemetry-instrumentation[test]'

pip (20.3) will fail with the following error message (pip 20.2.4 works however):

Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-api
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-sdk
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-instrumentation
INFO: pip is looking at multiple versions of opentelemetry-api[test] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement opentelemetry-sdk==0.17.dev0 (from opentelemetry-instrumentation[test])
ERROR: No matching distribution found for opentelemetry-sdk==0.17.dev0

Runing pip with the -v switch it seems that pip tries to lookup the opentelemetry-sdk package from pypi even though it was specified in the command line as to be installed from the local filesystem.

After trying around a little bit the following scenarios seem to work.

• Installing only the first two packages: pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]'.
• Installing without extras: pip install -e './opentelemetry-api' -e './opentelemetry-sdk' -e './opentelemetry-instrumentation'
• Rename the package names in the setup.py files (and also the dependencies) to something else that is not avialable on pypi (e.g. opentelemetry-api = py1, opentelemetry-sdk = py2 and opentelemetry-instrumentation = py3).
• Modify the install_requires dependencies to include the test extra (e.g. opentelemetry-api[test] == 0.17.dev0)

So the problem seems to be a combination of transitive dependencies, extras and package lookup on pypi for packages installed from the local filesystem.

[brainwane]

Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us.

While we're waiting for the resolver developers to look at your report and respond, I'll mention here one useful troubleshooting and workaround tip from the documentation:

If you run into resolution problems and need a workaround while you’re fixing their root causes, you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

(If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.)

[uranusjr]

Merging this into #8785 to avoid messages from scattering.

[mariojonke]

It appears that #8785 didn't fix this issue.

Running pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]' -e './opentelemetry-instrumentation[test]' with the setup.py files above and the latest pip version 21.1.1 still fails with the following message:

Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-api
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-sdk
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-instrumentation
INFO: pip is looking at multiple versions of opentelemetry-api to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of opentelemetry-api[test] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement opentelemetry-sdk==0.17.dev0 (from opentelemetry-instrumentation[test]) (from versions: 0.1a0, 0.2a0, 0.3a0, 0.4a0, 0.4a1, 0.5b0, 0.6b0, 0.7b1, 0.8b0, 0.9b0, 0.10b0, 0.11b0, 0.12b0, 0.13b0, 0.14b0, 0.15b0, 0.16b0, 0.16b1, 0.17b0, 1.0.0rc1, 1.0.0, 1.1.0, 1.2.0, 1.10a0)
ERROR: No matching distribution found for opentelemetry-sdk==0.17.dev0

[uranusjr]

I think putting './opentelemetry-sdk[test]' first should work.

[mariojonke]

I tried that but it is still producing the same output. I thought that the order of packages should be irrelevant with the new resolver?

Anyway, in the install command above the packages were already listed according to their dependencies among each other: api -> {}, sdk -> {api} and instrumentation -> {api, sdk}. Also, install works fine with this order when the extras are omitted.

[uranusjr]

Ordering is still relevant, they are resolved in the same order as they are requested. But if installing works fine without extras, ordering is likely not the issue. I'll find some time to look into this.

[bmw]

I'm also hitting this for https://github.com/certbot/certbot where we also have multiple packages depending on each other like this.

In case it's useful, here's a stripped down example reproducing the problem with only two packages:

$ tree .             
.
├── certbot
│   └── setup.py
└── certbot-apache
    └── setup.py

2 directories, 2 files

certbot/setup.py is https://gist.github.com/bmw/89b369a00f53b66157850e8097973e1e and certbot-apache/setup.py is https://gist.github.com/bmw/214e15753d92342cb4c37c898eebb818

$ pip install -e 'certbot[docs]' -e certbot-apache
Obtaining file:///private/var/folders/3b/zg8fdh5j71x92yyzc1tyllfw0000gp/T/tmp.19lG8kl23R/certbot
Obtaining file:///private/var/folders/3b/zg8fdh5j71x92yyzc1tyllfw0000gp/T/tmp.19lG8kl23R/certbot-apache
ERROR: Could not find a version that satisfies the requirement certbot>=99.99.0.dev0 (from certbot-apache) (from versions: 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.11.1, 0.12.0, 0.13.0, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.22.0, 0.22.1, 0.22.2, 0.23.0, 0.24.0, 0.25.0, 0.25.1, 0.26.0, 0.26.1, 0.27.0, 0.27.1, 0.28.0, 0.29.0, 0.29.1, 0.30.0, 0.30.1, 0.30.2, 0.31.0, 0.32.0, 0.33.0, 0.33.1, 0.34.0, 0.34.1, 0.34.2, 0.35.0, 0.35.1, 0.36.0, 0.37.0, 0.37.1, 0.37.2, 0.38.0, 0.39.0, 0.40.0, 0.40.1, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0)
ERROR: No matching distribution found for certbot>=99.99.0.dev0

EDIT: Tweaked version numbers so the example still works as we continue to do releases.

[uranusjr]

Figured out the issue. The reason why #8785 did not fix this is because the requirements provided in this issue are all unnamed (i.e. a bare path without #egg=pkg or PEP 508 pkg @). This means that the resolver cannot correctly detect the user’s specified ordering from the command line. (So for a two-requirement set-up, the success rate is actually 50-50 depending on the requirement’s path/URL, although all the examples here are failures for obvious reasons.) I’ll work on a fix for this.

Edit: Editable-ness is not actually relevant here (although editable requirements hit this issue more since they tend to be unnamed), so I’m removing that label.