Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pip 21.2 release breaks python2 compatibilty #10214

Closed
1 task done
mayanks opened this issue Jul 26, 2021 · 5 comments
Closed
1 task done

Pip 21.2 release breaks python2 compatibilty #10214

mayanks opened this issue Jul 26, 2021 · 5 comments
Labels
resolution: no action When the resolution is to not do anything

Comments

@mayanks
Copy link

mayanks commented Jul 26, 2021 •
edited

Description

This commit ensured that python setup.py can be run only with python3. And this was released in pip 21.2 version.

There are other projects like Spark which are still using python2 but also depend on latest release of pip.

So, when python2 setup.py sdist is run in above spark build, it installs the latest pip and then installation fails since python2 cannot execute pip's setup.py

And python3 setup.py sdist fails since Spark's setup.py cannot run on python3.

Expected behavior

pip should not break python2 compatibility

pip version

21.2

Python version

2.7.4

OS

Linux

How to Reproduce

git clone https://github.com/apache/spark/
cd python
python setup.py sdist

Output

Could not import pypandoc - required to package PySpark
zip_safe flag not set; analyzing archive contents...
pypandoc.__init__: module references __file__

Searching for wheel>=0.25.0
Reading https://pypi.org/simple/wheel/
Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl#sha256=78b5b185f0e5763c26ca1e324373aadd49182ca90e825f7853f4b2509215dc0e
Best match: wheel 0.36.2
Processing wheel-0.36.2-py2.py3-none-any.whl
Installing wheel-0.36.2-py2.py3-none-any.whl to /home/bigtop/bigtop/output/spark/spark-core-2.4.8/python/.eggs
writing requirements to /home/bigtop/bigtop/output/spark/spark-core-2.4.8/python/.eggs/wheel-0.36.2-py2.7.egg/EGG-INFO/requires.txt

Searching for pip>=8.1.0
Reading https://pypi.org/simple/pip/
Downloading https://files.pythonhosted.org/packages/f7/ce/e359cf283c0c0f2e0af7df8f16c8d79047aa1887a00a5b39b27d8afc49e2/pip-21.2.1.tar.gz#sha256=303a82aaa24cdc01f7ebbd1afc7d1b871a4aa0a88bb5bedef1fa86a3ee44ca0a
Best match: pip 21.2.1
Processing pip-21.2.1.tar.gz
Writing /tmp/easy_install-yZrS3X/pip-21.2.1/setup.cfg
Running pip-21.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-yZrS3X/pip-21.2.1/egg-dist-tmp-SjJiNU
Traceback (most recent call last):
  File "setup.py", line 224, in <module>
    'Programming Language :: Python :: Implementation :: PyPy']
  File "/usr/lib/python2.7/dist-packages/setuptools/__init__.py", line 144, in setup
    _install_setup_requires(attrs)
  File "/usr/lib/python2.7/dist-packages/setuptools/__init__.py", line 139, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 724, in fetch_build_eggs
    replace_conflicting=True,
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 791, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 704, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 730, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 915, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1183, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1169, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 44, in _execfile
    code = compile(script, filename, 'exec')
  File "/tmp/easy_install-yZrS3X/pip-21.2.1/setup.py", line 7
    def read(rel_path: str) -> str:
                     ^
SyntaxError: invalid syntax

Code of Conduct
@mayanks mayanks added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jul 26, 2021
@domdfcoding
Copy link
Contributor

Per pip’s Python 2 support policy, pip 20.3.4 was the last version of pip that supported Python 2. Projects still requiring Python 2 should pin pip to <21.0

@domdfcoding
Copy link
Contributor

From the traceback you provided it looks like spark is using easy_install to download and install pip. easy_install is deprecated in favour of pip. I suspect there is an easy_install bug/missing feature where it is trying to install an incompatible version of pip for the current version of Python, but I doubt that it will be fixed in easy_install.

I can reproduce the issue by installing pypandoc with easy_install. It looks to me like this is one of the steps in the spark install process.

$ python2.7 -m easy_install pypandoc
WARNING: The easy_install command is deprecated and will be removed in a future version.
Searching for pypandoc
Reading https://pypi.org/simple/pypandoc/
Downloading https://files.pythonhosted.org/packages/56/4b/1962e8b460e509b5e23d47491a11a182e9a2eca3206bb3ce8dad0633ecaa/pypandoc-1.6.3.tar.gz#sha256=95a331f5ef74adb9557e9961d784c30ba73bd0d62e7736bb5d9ce74888b9f438
Best match: pypandoc 1.6.3
Processing pypandoc-1.6.3.tar.gz
Writing /tmp/easy_install-GflVXi/pypandoc-1.6.3/setup.cfg
Running pypandoc-1.6.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GflVXi/pypandoc-1.6.3/egg-dist-tmp-7rzJRt
zip_safe flag not set; analyzing archive contents...
pypandoc.__init__: module references __file__
creating /tmp/spark/venv/lib/python2.7/site-packages/pypandoc-1.6.3-py2.7.egg
Extracting pypandoc-1.6.3-py2.7.egg to /tmp/spark/venv/lib/python2.7/site-packages
Adding pypandoc 1.6.3 to easy-install.pth file

Installed /tmp/spark/venv/lib/python2.7/site-packages/pypandoc-1.6.3-py2.7.egg
Processing dependencies for pypandoc
Searching for pip>=8.1.0
Reading https://pypi.org/simple/pip/
Downloading https://files.pythonhosted.org/packages/f7/ce/e359cf283c0c0f2e0af7df8f16c8d79047aa1887a00a5b39b27d8afc49e2/pip-21.2.1.tar.gz#sha256=303a82aaa24cdc01f7ebbd1afc7d1b871a4aa0a88bb5bedef1fa86a3ee44ca0a
Best match: pip 21.2.1
Processing pip-21.2.1.tar.gz
Writing /tmp/easy_install-95y9Wv/pip-21.2.1/setup.cfg
Running pip-21.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-95y9Wv/pip-21.2.1/egg-dist-tmp-q08hAr
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/spark/venv/lib/python2.7/site-packages/easy_install.py", line 5, in <module>
    main()
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 2321, in main
    **kw
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/__init__.py", line 162, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 424, in run
    self.easy_install(spec, not self.no_deps)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 685, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 713, in install_item
    self.process_distribution(spec, dist, deps)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 758, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/tmp/spark/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/tmp/spark/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/tmp/spark/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 685, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 711, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 896, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1164, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1150, in run_setup
    run_setup(setup_script, args)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/spark/venv/lib/python2.7/site-packages/setuptools/sandbox.py", line 44, in _execfile
    code = compile(script, filename, 'exec')
  File "/tmp/easy_install-95y9Wv/pip-21.2.1/setup.py", line 7
    def read(rel_path: str) -> str:
                     ^
SyntaxError: invalid syntax

pypandoc specifies pip as one of its setup requirements, which is why easy_install is trying to install it. If a suitable version is already installed, easy_install skips the install step and pypandoc installs successfully.

@uranusjr
Copy link
Member

Closing since we are not going to do anything about this.

@uranusjr uranusjr added resolution: no action When the resolution is to not do anything and removed S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jul 26, 2021
@pfmoore
Copy link
Member

pfmoore commented Jul 26, 2021

I suspect there is an easy_install bug/missing feature where it is trying to install an incompatible version of pip for the current version of Python

I believe easy_install never got updated to include Requires-Python support.

I doubt that it will be fixed in easy_install.

AFAIK easy_install has been deprecated by the setuptools project, so no, it won't.

@mayanks
Copy link
Author

mayanks commented Jul 27, 2021

My bad. I should have read the README before raising this issue. It clearly states that Pip 21.2 onwards Python2 support is deprecated.

Thanks for your help and suggestions @domdfcoding @pfmoore

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
resolution: no action When the resolution is to not do anything
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mayanks @uranusjr @pfmoore @domdfcoding