Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install -e fails on version 50: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command #2355

Closed
jay-karimi opened this issue Aug 31, 2020 · 12 comments · Fixed by #2381

Comments

@jay-karimi
Copy link

jay-karimi commented Aug 31, 2020

> pip install -e .
    ERROR: Command errored out with exit status 1:
     command: /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"'; __file__='"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /Users/jaykarimi/Documents/vanir/
    Complete output (19 lines):
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing 'v1.7.1' to '1.7.1'
      warnings.warn(tmpl.format(**locals()))
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/jaykarimi/Documents/vanir/setup.py", line 127, in <module>
        setup(**_conf)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/core.py", line 134, in setup
        ok = dist.parse_command_line()
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 484, in parse_command_line
        args = self._parse_command_opts(parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py", line 903, in _parse_command_opts
        nargs = _Distribution._parse_command_opts(self, parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 548, in _parse_command_opts
        "command class %s must subclass Command" % cmd_class)
    distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
@jay-karimi jay-karimi changed the title pip install -e fails on version 50 pip install -e fails on version 50: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command Aug 31, 2020
@wmdrichards
Copy link

wmdrichards commented Sep 1, 2020

Minimal reproducing example setup.py

import setuptools
import pip
setuptools.setup()

So seems related to #2365, though the symptoms are different

@moltob
Copy link

moltob commented Sep 1, 2020

Since v50 we also get this error when installing open source package doit from its TGZ package (logs from a Windows machine, same error on Linux):

(venv) λ pip --no-cache-dir install doit
Looking in indexes: https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/simple
Collecting doit
  Downloading https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/packages/packages/1f/89/b5e1dd4854b47c4f9e5250899c04e0cd0c9082c30ef619c0413ace20cf7a/doit-0.32.0.tar.gz (1.4 MB)
     |████████████████████████████████| 1.4 MB 1.7 MB/s
    ERROR: Command errored out with exit status 1:
     command: 'd:\temp\doit-install-issue\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"'; __file__='"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\q199925\AppData\Local\Temp\pip-pip-egg-info-fhnoqhon'
         cwd: C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\
    Complete output (17 lines):
    d:\temp\doit-install-issue\venv\lib\site-packages\_distutils_hack\__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\setup.py", line 44, in <module>
        setup(name = 'doit',
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 152, in setup
        _install_setup_requires(attrs)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 145, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 665, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 572, in _parse_config_files
        filenames = self.find_config_files()
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\_distutils\dist.py", line 353, in find_config_files
        sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
    KeyError: 'distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It also has an import pip in its setup.py. Please note that while this might be wrong, it's quite tough to force other projects to adapt by releasing a new version of setuptools.

Also, please note that official Python docs recommend to update setuptools to the latest version before installing anything for real, so pinning setuptools to overcome the current issues is not a real option.

Would it be possible to roll back, but release a new version such that all the pip install -U setuptools calls remain valid?

@ferraith FYI

@asmacdo
Copy link

asmacdo commented Sep 1, 2020

This can be worked around with the SETUPTOOLS_USE_DISTUTILS env var.
https://setuptools.readthedocs.io/en/latest/history.html#v50-0-0

I had this problem with ruamel.yaml.clib, but only when installing from source (not wheels).

Reproducer:
pip install --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib

Snip...
distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command

Workaround
export SETUPTOOLS_USE_DISTUTILS="stdlib"
pip install --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib

Collecting ruamel.yaml.clib
  Downloading https://files.pythonhosted.org/packages/92/28/612085de3fae9f82d62d80255d9f4cf05b1b341db1e180adcf28c1bf748d/ruamel.yaml.clib-0.2.0.tar.gz (178kB)
     |████████████████████████████████| 184kB 739kB/s 
Installing collected packages: ruamel.yaml.clib
    Running setup.py install for ruamel.yaml.clib ... done
Successfully installed ruamel.yaml.clib-0.2.0

@jay-karimi
Copy link
Author

jay-karimi commented Sep 1, 2020

@asmacdo I can confirm that the workaround you describe with export SETUPTOOLS_USE_DISTUTILS="stdlib" seems to resolve the error I am seeing as well.

@jaraco
Copy link
Member

jaraco commented Sep 1, 2020

Note there are seemingly two errors reported here. One relating to Command subclass, which since it's in the title and OP, I'll continue to address here. The other, relating to import of pip, was reported in #2365, so I'll use that ticket to track that issue.

@jaraco
Copy link
Member

jaraco commented Sep 1, 2020

Hi @jay-karimi . Sorry for the trouble you experienced and thanks for filing a report. @asmacdo Thanks for the reproducer. That helps a lot. I'm able to replicate the failure with this command:

pip-run -q setuptools==50 -- -m pip-run --no-binary :all: --no-deps ruamel.yaml.clib

@jaraco
Copy link
Member

jaraco commented Sep 1, 2020

Same when executing against the source repo:

$ pip-run -q setuptools==50 -- -m pip-run hg+http://hg.code.sf.net/p/ruamel-yaml-clib/code@0.2.0

@jaraco
Copy link
Member

jaraco commented Sep 1, 2020

I traced the issue to its source:

ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
> /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
(Pdb) c
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py", line 587, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 5, in <module>
    from __future__ import print_function, absolute_import, division, unicode_literals
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 885, in main
    nsp.check()
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 451, in check
    from pip.exceptions import InstallationError
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 74, in find_spec
    return method()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 94, in spec_for_pip
    clear_distutils()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 30, in clear_distutils
    warnings.warn("Setuptools is replacing distutils.")
UserWarning: Setuptools is replacing distutils.
(Pdb) w
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1703)main()
-> pdb._runscript(mainpyfile)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1572)_runscript()
-> self.run(statement)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py(587)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(885)main()
-> nsp.check()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check()
-> from pip.exceptions import InstallationError
  <frozen importlib._bootstrap>(991)_find_and_load()
  <frozen importlib._bootstrap>(971)_find_and_load_unlocked()
  <frozen importlib._bootstrap>(914)_find_spec()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(74)find_spec()
-> return method()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(94)spec_for_pip()
-> clear_distutils()
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")

Notice /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check() there in the call stack when the warning is raised? It seems the setup.py for that project is importing pip (and merely to get an InstallationError exception class).

By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the KeyError is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and distutils.cmd.Command now has two manifestations.

Replacing the use of import pip fixes the error:

ruamel-yaml-clib-code default $ hg diff
diff -r 52bf0eb35c43 setup.py
--- a/setup.py  Fri Sep 27 07:42:41 2019 +0200
+++ b/setup.py  Tue Sep 01 18:07:28 2020 -0400
@@ -447,10 +447,7 @@
             sys.exit(1)
 
     def check(self):
-        try:
-            from pip.exceptions import InstallationError
-        except ImportError:
-            return
+        InstallationError = Exception
         # arg is either develop (pip install -e) or install
         if self.command not in ['install', 'develop']:
             return

ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror setup.py egg_info
sys.argv ['setup.py', 'egg_info']
test compiling test_ruamel_yaml
running egg_info
writing ruamel.yaml.clib.egg-info/PKG-INFO
writing dependency_links to ruamel.yaml.clib.egg-info/dependency_links.txt
writing namespace_packages to ruamel.yaml.clib.egg-info/namespace_packages.txt
writing top-level names to ruamel.yaml.clib.egg-info/top_level.txt
reading manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'

My recommendation is for that project (and any other projects using that technique) avoid importing pip.

@asfaltboy
Copy link

asfaltboy commented Sep 2, 2020

@jaraco should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?

@chfw
Copy link

chfw commented Sep 2, 2020

I apply the same fix for Windowns build and it worked too. Here is Window statement:

set SETUPTOOLS_USE_DISTUTILS="stdlib"
pip install ...

chfw added a commit to moremoban/moban that referenced this issue Sep 2, 2020
* 🔥 split up the non-essential utilities to moban extensions where they could grow bigger

* 📰 add moban update action

* 📰 update moban update command

* This is an auto-commit, updating project meta data, such as changelog.rst, contributors.rst

* 🐛 generate the template from the same environment so that custom jinja2 thingies are visible. fix #396

* 🔥 remove github tests

* 🐛 apply temporary workaround for pypa/setuptools#2355

* 🔥 remove unused imports

Co-authored-by: chfw <chfw@users.noreply.github.com>
@AvdN
Copy link

AvdN commented Sep 3, 2020

> I traced the issue to its source:
> 
> ```
> ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
<snip>
> By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the `KeyError` is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and `distutils.cmd.Command` now has two manifestations.
> 
> Replacing the use of `import pip` fixes the error:
> 
<snip> 
> My recommendation is for that project (and any other projects using that technique) avoid importing pip.

Thanks for the analysis and suggested fix.
I am in the process of implementing this, this is a common setup.py for all projects ruamel distributes, and especially generating the wheels for C extensions (such as ruamel.yaml.clib) will take some effort. I will however prioritize the open source pages (~25 or so from the 150+ that need updating).

@jaraco
Copy link
Member

jaraco commented Sep 4, 2020

should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?

It is likely possible. Given that for the distutils hack, setuptools special-cases pip, it may make sense to be even more sophisticated there and not disable the override unless pip is being executed.

elemental-lf added a commit to elemental-lf/benji that referenced this issue Sep 5, 2020
commx pushed a commit to commx/ruamel-yaml that referenced this issue Dec 30, 2020
ssbarnea pushed a commit to pycontribs/ruyaml that referenced this issue May 21, 2021
* allow C extension install for Python 3.8

* Added tag 0.16.8 for changeset 0228e1a734b5

* remove duplicates from CHANGES

* Added tag 0.16.9 for changeset 17f2c2782abb

* update url in meta and .svg in README

* Added tag 0.16.10 for changeset 42b89940bc91

* udpate setup.py to workaround issue in setuptools

pypa/setuptools#2355 (comment)

* Added tag 0.16.11 for changeset 43b028851940

* update links in doc

* Added tag 0.16.12 for changeset 53ec5198e50a

* try to re-enable rtfd

* try to re-enable rtfd

* try to re-enable rtfd

* fix issue 359

* corrections on update()

* explain round-tripping

* fix #371

* fix #365

* fix #373

* last release supporting 2.7

* Added tag 0.16.13 for changeset ce172fcf4b61

* remove python 2 specific code
add future deprecation warning to old style functions

* remove superfluous comment assignment

* add scan, parser. compose, emit, serialize to YAML

* remove support for 2.7, prepare for f-strings

* Added tag 0.17.0 for changeset 7c1c4a4e17cf

* add Python 3 only classifier

* Added tag 0.17.1 for changeset 961613d5ea1a

* change py2.py3 wheel to py3 only wheel

* Added tag 0.17.2 for changeset a8a9141e7a90

* fix for issue 382, error in format string

* Added tag 0.17.3 for changeset f115bb2a18a0

* partial fix for 351

* Added tag 0.17.4 for changeset eb1a5e34fc54

* Code and tests fixed.

Hopefully.

* remove old compat code

* Appease the linters

Co-authored-by: Anthon van der Neut <anthon@mnt.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants