pip 21.1 install with --no-binary fails while 21.0.1 succeeds (pep517)

[g8trbryant]

Description

Installing the pip package pex with --no-binary fails after pip version changed from 21.0.1 to 21.1. In the output log below, it seems like pep517 is involved. Steps below are successful if pip==21.0.1 is used.

Expected behavior

No response

pip version

21.1

Python version

3.6.4

OS

RHEL7 x86_64

How to Reproduce

python3 -m venv venv
. venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade setuptools
python3 -m pip install pex==2.1.30 --no-binary :all:

Output

Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached https://files.pythonhosted.org/packages/ac/cf/0cc542fc93de2f3b9b53cb979c7d1118cffb93204afb46299a9f858e113f/pip-21.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-21.1

Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (28.8.0)
Collecting setuptools
  Using cached setuptools-56.0.0-py3-none-any.whl (784 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-56.0.0

Collecting pex==2.1.30
  Downloading pex-2.1.30.tar.gz (3.3 MB)
     |████████████████████████████████| 3.3 MB 3.0 MB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: <path to venv>/venv/bin/python3 /tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-3b3hrzfg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2,<3'
       cwd: None
  Complete output (67 lines):
  Collecting flit_core<3,>=2
    Using cached flit_core-2.3.0.tar.gz (22 kB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: <path to venv>/venv/bin/python3 /tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpogp2e7mb
         cwd: /tmp/pip-install-71xy4ixf/flit-core_156db09074064d12968592391ad52e71
    Complete output (1 lines):
    <path to venv>/venv/bin/python3: can't find '__main__' module in '/tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip/_vendor/pep517/in_process/_in_process.py'

^repeated error while it tries earlier versions of flit_core

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

This is a side effect of pip 21.1 switching to use zipapp for PEP 517 dependency population (#9689). This crashes 3.6 in “PEP 517 inception” scenarios[see note] since pep517 falls back to “dumb” __file__ manipulation when importlib.resources is not available. I should have noticed this 😞

I think there are two solutions:

1. Fall back to the “old” way of self-invocation on 3.6. This means people on 3.6 won’t receive Run pip in isolated env by building zip #9689, but it was already only fixing a fairly obscure issue anyway.
2. Vendor importlib-resources (backport of the stdlib module) and patch pep517 to use it instead of the stdlib module.

Either way, the workaround will only be short-term, since we will be dropping 3.6 in a year.

---

Note: This will not crash without --no-binary, even if a package is sdist-only, as long as the build dependency has a wheel. When pep517 is invoked to build the sdist, _in_process.py is a plain file, so everything works. With --no-binary, build dependencies required by the sdist are also only fetched by source only, and pep517 by that point lives in a zip, causing the issue.