pip install -e fails on version 50: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command

[jay-karimi]

> pip install -e .
    ERROR: Command errored out with exit status 1:
     command: /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"'; __file__='"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /Users/jaykarimi/Documents/vanir/
    Complete output (19 lines):
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing 'v1.7.1' to '1.7.1'
      warnings.warn(tmpl.format(**locals()))
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/jaykarimi/Documents/vanir/setup.py", line 127, in <module>
        setup(**_conf)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/core.py", line 134, in setup
        ok = dist.parse_command_line()
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 484, in parse_command_line
        args = self._parse_command_opts(parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py", line 903, in _parse_command_opts
        nargs = _Distribution._parse_command_opts(self, parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 548, in _parse_command_opts
        "command class %s must subclass Command" % cmd_class)
    distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[wmdrichards]

Minimal reproducing example setup.py

import setuptools
import pip
setuptools.setup()

So seems related to #2365, though the symptoms are different

[moltob]

Since v50 we also get this error when installing open source package doit from its TGZ package (logs from a Windows machine, same error on Linux):

(venv) λ pip --no-cache-dir install doit
Looking in indexes: https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/simple
Collecting doit
  Downloading https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/packages/packages/1f/89/b5e1dd4854b47c4f9e5250899c04e0cd0c9082c30ef619c0413ace20cf7a/doit-0.32.0.tar.gz (1.4 MB)
     |████████████████████████████████| 1.4 MB 1.7 MB/s
    ERROR: Command errored out with exit status 1:
     command: 'd:\temp\doit-install-issue\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"'; __file__='"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\q199925\AppData\Local\Temp\pip-pip-egg-info-fhnoqhon'
         cwd: C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\
    Complete output (17 lines):
    d:\temp\doit-install-issue\venv\lib\site-packages\_distutils_hack\__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\setup.py", line 44, in <module>
        setup(name = 'doit',
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 152, in setup
        _install_setup_requires(attrs)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 145, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 665, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 572, in _parse_config_files
        filenames = self.find_config_files()
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\_distutils\dist.py", line 353, in find_config_files
        sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
    KeyError: 'distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It also has an import pip in its setup.py. Please note that while this might be wrong, it's quite tough to force other projects to adapt by releasing a new version of setuptools.

Also, please note that official Python docs recommend to update setuptools to the latest version before installing anything for real, so pinning setuptools to overcome the current issues is not a real option.

Would it be possible to roll back, but release a new version such that all the pip install -U setuptools calls remain valid?

@ferraith FYI

[asmacdo]

This can be worked around with the SETUPTOOLS_USE_DISTUTILS env var.
https://setuptools.readthedocs.io/en/latest/history.html#v50-0-0

I had this problem with ruamel.yaml.clib, but only when installing from source (not wheels).

Reproducer:
pip install --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib

Snip...
distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command

Workaround
export SETUPTOOLS_USE_DISTUTILS="stdlib"
pip install --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib

Collecting ruamel.yaml.clib
  Downloading https://files.pythonhosted.org/packages/92/28/612085de3fae9f82d62d80255d9f4cf05b1b341db1e180adcf28c1bf748d/ruamel.yaml.clib-0.2.0.tar.gz (178kB)
     |████████████████████████████████| 184kB 739kB/s 
Installing collected packages: ruamel.yaml.clib
    Running setup.py install for ruamel.yaml.clib ... done
Successfully installed ruamel.yaml.clib-0.2.0

[jay-karimi]

@asmacdo I can confirm that the workaround you describe with export SETUPTOOLS_USE_DISTUTILS="stdlib" seems to resolve the error I am seeing as well.

[jaraco]

Note there are seemingly two errors reported here. One relating to Command subclass, which since it's in the title and OP, I'll continue to address here. The other, relating to import of pip, was reported in #2365, so I'll use that ticket to track that issue.

[jaraco]

Hi @jay-karimi . Sorry for the trouble you experienced and thanks for filing a report. @asmacdo Thanks for the reproducer. That helps a lot. I'm able to replicate the failure with this command:

pip-run -q setuptools==50 -- -m pip-run --no-binary :all: --no-deps ruamel.yaml.clib

[jaraco]

Same when executing against the source repo:

$ pip-run -q setuptools==50 -- -m pip-run hg+http://hg.code.sf.net/p/ruamel-yaml-clib/code@0.2.0

[jaraco]

I traced the issue to its source:

ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
> /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
(Pdb) c
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py", line 587, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 5, in <module>
    from __future__ import print_function, absolute_import, division, unicode_literals
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 885, in main
    nsp.check()
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 451, in check
    from pip.exceptions import InstallationError
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 74, in find_spec
    return method()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 94, in spec_for_pip
    clear_distutils()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 30, in clear_distutils
    warnings.warn("Setuptools is replacing distutils.")
UserWarning: Setuptools is replacing distutils.
(Pdb) w
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1703)main()
-> pdb._runscript(mainpyfile)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1572)_runscript()
-> self.run(statement)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py(587)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(885)main()
-> nsp.check()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check()
-> from pip.exceptions import InstallationError
  <frozen importlib._bootstrap>(991)_find_and_load()
  <frozen importlib._bootstrap>(971)_find_and_load_unlocked()
  <frozen importlib._bootstrap>(914)_find_spec()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(74)find_spec()
-> return method()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(94)spec_for_pip()
-> clear_distutils()
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")

Notice /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check() there in the call stack when the warning is raised? It seems the setup.py for that project is importing pip (and merely to get an InstallationError exception class).

By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the KeyError is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and distutils.cmd.Command now has two manifestations.

Replacing the use of import pip fixes the error:

ruamel-yaml-clib-code default $ hg diff
diff -r 52bf0eb35c43 setup.py
--- a/setup.py  Fri Sep 27 07:42:41 2019 +0200
+++ b/setup.py  Tue Sep 01 18:07:28 2020 -0400
@@ -447,10 +447,7 @@
             sys.exit(1)
 
     def check(self):
-        try:
-            from pip.exceptions import InstallationError
-        except ImportError:
-            return
+        InstallationError = Exception
         # arg is either develop (pip install -e) or install
         if self.command not in ['install', 'develop']:
             return

ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror setup.py egg_info
sys.argv ['setup.py', 'egg_info']
test compiling test_ruamel_yaml
running egg_info
writing ruamel.yaml.clib.egg-info/PKG-INFO
writing dependency_links to ruamel.yaml.clib.egg-info/dependency_links.txt
writing namespace_packages to ruamel.yaml.clib.egg-info/namespace_packages.txt
writing top-level names to ruamel.yaml.clib.egg-info/top_level.txt
reading manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'

My recommendation is for that project (and any other projects using that technique) avoid importing pip.

[asfaltboy]

@jaraco should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?

[chfw]

I apply the same fix for Windowns build and it worked too. Here is Window statement:

set SETUPTOOLS_USE_DISTUTILS="stdlib"
pip install ...

[AvdN]

> I traced the issue to its source:
> 
> ```
> ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
<snip>
> By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the `KeyError` is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and `distutils.cmd.Command` now has two manifestations.
> 
> Replacing the use of `import pip` fixes the error:
> 
<snip> 
> My recommendation is for that project (and any other projects using that technique) avoid importing pip.

Thanks for the analysis and suggested fix.
I am in the process of implementing this, this is a common setup.py for all projects ruamel distributes, and especially generating the wheels for C extensions (such as ruamel.yaml.clib) will take some effort. I will however prioritize the open source pages (~25 or so from the 150+ that need updating).

[jaraco]

should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?

It is likely possible. Given that for the distutils hack, setuptools special-cases pip, it may make sense to be even more sophisticated there and not disable the override unless pip is being executed.