Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugins using setuptools entry points resolve to system and not setup_requires #2158

Closed
anntzer opened this issue May 28, 2020 · 13 comments
Closed

Comments

@anntzer
Copy link

anntzer commented May 28, 2020

Assume that you have e.g. setuptools_scm 3 installed system-wide, and try to to run the following setup.py:

from setuptools import setup
setup(name="foo", setup_requires=["setuptools_scm>=4"])

you get the warning

/usr/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /usr/lib/python3.8/site-packages/setuptools_scm/__init__.py, but /tmp/foo/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path

(i.e. the system-wide setuptools_scm was loaded, even though it is too old). One can confirm that the "correct" newer setuptools_scm indeed doesn't get used by changing setup.py to

from setuptools import setup
setup(name="foo",
      setup_requires=["setuptools_scm>=4"],
      use_scm_version={"parentdir_prefix_version": "foo-"})

in which case one gets the same warning and then a traceback

TypeError: __init__() got an unexpected keyword argument 'parentdir_prefix_version'

I guess a solution may be e.g. to move the entry point (which is what gets eagerly loaded by setuptools) to a separate package (let's say _setuptools_scm_helper) which itself imports setuptools_scm only when actually needed with a pass-through shim so that setuptools only imports _setuptools_scm_helper and then the correct version of setuptools_scm can be loaded when needed.

@RonnyPfannschmidt
Copy link
Contributor

RonnyPfannschmidt commented May 28, 2020

@jaraco does this relate to the keyword hook usage?
@anntzer whats the setuptools version?

usually this is something i would take a look at on a weekend, but the next 2 weekends i am preoccupied with personal matters

@anntzer
Copy link
Author

anntzer commented May 28, 2020

This is with setuptools 46.4.0.
No hurries :)

@jaraco
Copy link
Member

jaraco commented May 28, 2020

Setuptools_scm and other setuptools plugins are not intended to be installed system-wide. Why not just not do that?

@jaraco
Copy link
Member

jaraco commented May 28, 2020

does this relate to the keyword hook usage?

I'm not sure, but you can find out by testing with setuptools<42 and see if the behavior is any different. It's possible that it is affected, because the finalize_distribution_options hook does run (and thus enumerate entry points) earlier than other hooks.

@anntzer
Copy link
Author

anntzer commented May 28, 2020

I had not ever seen that recommendation anywhere (I would be happy to see guidance to that effect).
I'm happy to take your word for it, but then 1) perhaps setuptools/pip could somehow recognize that (e.g. packages providing setuptools-related entry points) and indeed refuse to install them, and 2) should e.g. linux/conda packagers get a warning requesting them to remove the packages (https://packages.debian.org/sid/python3-setuptools-scm, https://www.archlinux.org/packages/community/any/python-setuptools-scm/, https://anaconda.org/conda-forge/setuptools_scm etc.)?

@jaraco
Copy link
Member

jaraco commented May 28, 2020

Now that I think about it, your best option is probably to use pip's build isolation to bypass system packages altogether. pip --use-pep517 foo should work.

@anntzer
Copy link
Author

anntzer commented May 28, 2020

PEP517 is not really an option for us because we need to support editable installs (because the package takes a while to compile).
By "we" I specifically meant Matplotlib, in this case; I was hoping to switch it from versioneer to setuptools_scm.

@jaraco
Copy link
Member

jaraco commented May 28, 2020

I'm able to replicate the issue with:

draft $ cat > setup.py
__import__('setuptools').setup(name='foo', setup_requires=['setuptools_scm>=4'])
draft $ pip-run -q 'setuptools_scm<4' -- setup.py --name
WARNING: The wheel package is not available.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-0cfuhayx/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo

@jaraco
Copy link
Member

jaraco commented May 28, 2020

Confirmed - the issue emerges at setuptools 42.

draft $ pip-run -q 'setuptools_scm<4' 'setuptools<42' -- setup.py --name
foo
draft $ pip-run -q 'setuptools_scm<4' 'setuptools==42' -- setup.py --name
WARNING: The wheel package is not available.
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools/dist.py:724: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo

@jaraco
Copy link
Member

jaraco commented May 28, 2020

I've figured out the issue. When setup() is called, it fires up a minimal Distribution() class, just enough to fetch the build requirements. It then adds those to the working set and proceeds to run setup() normally.

However, because constructing a Distribution triggers finalize_distribution_options, that causes plugins to get initialized and thus things like setuptools_scm to load before setup_requires was honored.

This patch works around the issue:

diff --git a/setuptools/__init__.py b/setuptools/__init__.py
index 811f3fd2..dbcb1723 100644
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -128,7 +128,11 @@ if PY3:
 def _install_setup_requires(attrs):
     # Note: do not use `setuptools.Distribution` directly, as
     # our PEP 517 backend patch `distutils.core.Distribution`.
-    dist = distutils.core.Distribution(dict(
+    class NeuteredDistribution(distutils.core.Distribution):
+        def finalize_options(self):
+            pass
+
+    dist = NeuteredDistribution(dict(
         (k, v) for k, v in attrs.items()
         if k in ('dependency_links', 'setup_requires')
     ))

@jaraco jaraco transferred this issue from pypa/setuptools_scm May 28, 2020
@jaraco
Copy link
Member

jaraco commented May 28, 2020

I've transferred the issue to the setuptools project, as setuptools_scm is just one plugin affected.

@jaraco jaraco changed the title setuptools_scm uses systemwide install (if present), even if it is too to satisfy setup_requires Plugins using setuptools entry points resolve to system and not setup_requires May 28, 2020
jaraco added a commit that referenced this issue May 29, 2020
…r to invoking `_install_setup_requires`, broken since v42.0.0. Fixes #2158.
@jaraco jaraco mentioned this issue May 29, 2020
2 tasks
jaraco added a commit that referenced this issue May 29, 2020
…r to invoking `_install_setup_requires`, broken since v42.0.0. Fixes #2158.
@jaraco jaraco closed this as completed in 4020c9a May 29, 2020
@jaraco
Copy link
Member

jaraco commented May 29, 2020

Released as v44.1.1 and v47.1.1.

@anntzer
Copy link
Author

anntzer commented May 29, 2020

Thanks for the quick fix! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants