pkg_resources hard-fails on install_requires for entry point.

[pganssle]

This has bitten us a few times now, I think that pkg_resources is a bit aggressive about install_requires, so when you attempt to invoke an entry point, the install_requires list is checked, and if any of the dependencies are missing, the script fails to start.

This problem has come up because we have a system that side-loads pre-built source (a la DPKG and presumably other package managers) where the dependencies are declared by the package manager (so install_requires is actually mostly ignored at install time). Occasionally we have needed to change the name of the package deployed by the package manager even though the importable path has not changed. If someone has the old package name in their install_requires and invokes their script via python -m mypkg, it works fine, but if they are invoking through an entry point, the script fails to run because the dependencies are not installed.

To demonstrate, I have prepared a small repository that demonstrates this behavior.

This is a real problem because it occurs at runtime and there doesn't seem to be any obvious way to "shim" around it other than by providing a dummy package (with no easy way to notify users that they are relying on this dummy package).

I suggest changing the DistributionNotFound error into a warning instead.

[jaraco]

Have you tried using the entrypoints project? I suspect because it bypasses pkg_resources, it also bypasses the working set consistency check that leads to the DistributionNotFound error.

I should note that there are other errors besides DistributionNotFound that can result from attempting to load entrypoints (or even just importing pkg_resources), such as a ContextualVersionConflict if the package(s) are present but don't satisfy the declared version.

[pganssle]

@jaraco Can we just turn those all into warnings for the purposes of entry_points?

We can maybe have an option to re-enable the old behavior, but I think by default this sort of thing should be enforced only at build time, not at runtime. If you want a hard version check at runtime, that should be opt-in, and we should let entry points with missing dependencies fail only if the missing dependency itself causes a failure.

[palao]

Hi,
Given a setup like this one:

setup(
    name = "Project-A"
    #...,
    entry_points={
        "console_scripts": [
            "rst2pdf = project_a.tools.pdfgen [PDF]",
            "rst2html = project_a.tools.htmlgen",
        ],
    }
)

the docs say in optional-dependencies section

When the script rst2pdf is run, it will trigger the installation of the two dependencies PDF maps to.

But if that does not happen, either the docs are wrong, or there is a bug.

Are there plans to fix it, or did I miss anything?

[palao]

Are there plans to fix it, or did I miss anything?

I mean, this seems to be still unresolved but there is no "bug" label on it...

[jaraco]

the docs say in optional-dependencies section

When the script rst2pdf is run, it will trigger the installation of the two dependencies PDF maps to.

The docs may well be wrong. There was a recent revamp where a passionate user attempted to describe what behavior they expected, but it may not map to the actual implementation. I suspect the actual implementation is that the indicated console script will only be installed if the relevant extras are requested. Here's what the old docs said, and that sounds closer to the implementation.

Hmm. Looking at the docs, that's also consistent with what's written here. In c657e82, I've corrected the documentation to declare that the behavior is implementation-specific.

In another comment, I'll address the larger issue.

[jaraco]

I see that this issue has been fixed in #2194 by bypassing pkg_resources in the easy_install scripts and instead relying on importlib.metadata:

setuptools main $ git clone gh://pganssle/pkg_resources_install_requires_demo
Cloning into 'pkg_resources_install_requires_demo'...
remote: Enumerating objects: 22, done.
remote: Total 22 (delta 0), reused 0 (delta 0), pack-reused 22
Receiving objects: 100% (22/22), 5.00 KiB | 5.00 MiB/s, done.
Resolving deltas: 100% (4/4), done.
setuptools main $ cd pkg_resources_install_requires_demo/
pkg_resources_install_requires_demo master $ source-bash initial_install.sh
(venv)  pkg_resources_install_requires_demo master $ venv/bin/pkg1script
Used from pkg2
(venv)  pkg_resources_install_requires_demo master $ cat venv/bin/pkg1script
#!/Users/jaraco/code/public/pypa/setuptools/pkg_resources_install_requires_demo/venv/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'pkg1','console_scripts','pkg1script'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'pkg1'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point


def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()


globals().setdefault('load_entry_point', importlib_load_entry_point)


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('pkg1', 'console_scripts', 'pkg1script')())
(venv)  pkg_resources_install_requires_demo master $ pip uninstall pkg2
Found existing installation: pkg2 0.0.0
Uninstalling pkg2-0.0.0:
  Would remove:
    /Users/jaraco/code/public/pypa/setuptools/pkg_resources_install_requires_demo/venv/lib/python3.9/site-packages/pkg2.egg-link
Proceed (y/n)? y
  Successfully uninstalled pkg2-0.0.0
(venv)  pkg_resources_install_requires_demo master $ pkg3script
Fell back to pkg1
(venv)  pkg_resources_install_requires_demo master $ pkg1script
Fell back to pkg1

All you need to do is make sure you have Setuptools 47.2 or later and Python 3.8 or importlib_metadata installed.