Plugins using setuptools entry points resolve to system and not setup_requires

[anntzer]

Assume that you have e.g. setuptools_scm 3 installed system-wide, and try to to run the following setup.py:

from setuptools import setup
setup(name="foo", setup_requires=["setuptools_scm>=4"])

you get the warning

/usr/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /usr/lib/python3.8/site-packages/setuptools_scm/__init__.py, but /tmp/foo/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path

(i.e. the system-wide setuptools_scm was loaded, even though it is too old). One can confirm that the "correct" newer setuptools_scm indeed doesn't get used by changing setup.py to

from setuptools import setup
setup(name="foo",
      setup_requires=["setuptools_scm>=4"],
      use_scm_version={"parentdir_prefix_version": "foo-"})

in which case one gets the same warning and then a traceback

TypeError: __init__() got an unexpected keyword argument 'parentdir_prefix_version'

I guess a solution may be e.g. to move the entry point (which is what gets eagerly loaded by setuptools) to a separate package (let's say _setuptools_scm_helper) which itself imports setuptools_scm only when actually needed with a pass-through shim so that setuptools only imports _setuptools_scm_helper and then the correct version of setuptools_scm can be loaded when needed.

[RonnyPfannschmidt]

@jaraco does this relate to the keyword hook usage?
@anntzer whats the setuptools version?

usually this is something i would take a look at on a weekend, but the next 2 weekends i am preoccupied with personal matters

[anntzer]

This is with setuptools 46.4.0.
No hurries :)

[jaraco]

Setuptools_scm and other setuptools plugins are not intended to be installed system-wide. Why not just not do that?

[jaraco]

does this relate to the keyword hook usage?

I'm not sure, but you can find out by testing with setuptools<42 and see if the behavior is any different. It's possible that it is affected, because the finalize_distribution_options hook does run (and thus enumerate entry points) earlier than other hooks.

[anntzer]

I had not ever seen that recommendation anywhere (I would be happy to see guidance to that effect).
I'm happy to take your word for it, but then 1) perhaps setuptools/pip could somehow recognize that (e.g. packages providing setuptools-related entry points) and indeed refuse to install them, and 2) should e.g. linux/conda packagers get a warning requesting them to remove the packages (https://packages.debian.org/sid/python3-setuptools-scm, https://www.archlinux.org/packages/community/any/python-setuptools-scm/, https://anaconda.org/conda-forge/setuptools_scm etc.)?

[jaraco]

Now that I think about it, your best option is probably to use pip's build isolation to bypass system packages altogether. pip --use-pep517 foo should work.

[anntzer]

PEP517 is not really an option for us because we need to support editable installs (because the package takes a while to compile).
By "we" I specifically meant Matplotlib, in this case; I was hoping to switch it from versioneer to setuptools_scm.

[jaraco]

I'm able to replicate the issue with:

draft $ cat > setup.py
__import__('setuptools').setup(name='foo', setup_requires=['setuptools_scm>=4'])
draft $ pip-run -q 'setuptools_scm<4' -- setup.py --name
WARNING: The wheel package is not available.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-0cfuhayx/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo

[jaraco]

Confirmed - the issue emerges at setuptools 42.

draft $ pip-run -q 'setuptools_scm<4' 'setuptools<42' -- setup.py --name
foo
draft $ pip-run -q 'setuptools_scm<4' 'setuptools==42' -- setup.py --name
WARNING: The wheel package is not available.
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools/dist.py:724: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo

[jaraco]

I've figured out the issue. When setup() is called, it fires up a minimal Distribution() class, just enough to fetch the build requirements. It then adds those to the working set and proceeds to run setup() normally.

However, because constructing a Distribution triggers finalize_distribution_options, that causes plugins to get initialized and thus things like setuptools_scm to load before setup_requires was honored.

This patch works around the issue:

diff --git a/setuptools/__init__.py b/setuptools/__init__.py
index 811f3fd2..dbcb1723 100644
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -128,7 +128,11 @@ if PY3:
 def _install_setup_requires(attrs):
     # Note: do not use `setuptools.Distribution` directly, as
     # our PEP 517 backend patch `distutils.core.Distribution`.
-    dist = distutils.core.Distribution(dict(
+    class NeuteredDistribution(distutils.core.Distribution):
+        def finalize_options(self):
+            pass
+
+    dist = NeuteredDistribution(dict(
         (k, v) for k, v in attrs.items()
         if k in ('dependency_links', 'setup_requires')
     ))

[jaraco]

I've transferred the issue to the setuptools project, as setuptools_scm is just one plugin affected.

[jaraco]

Released as v44.1.1 and v47.1.1.

[anntzer]

Thanks for the quick fix! :)