Pex 2.1.93 pex --lock artifact downloading fails for sdists with ext_modules or PEP-517 build systems.

[jsirois]

Even though we download lock artifacts using pip download --no-deps, Pip still tries to execute the build system. Maybe to gather metadata? I'm really not sure why it does this; no metadata is needed:

$ pex -V
2.1.93
$ pex3 lock create --no-wheel psutil==5.9.1 -o lock.json

lock.json:

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": false,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "57f1819b5d9e95cdfb0c881a8a5b7d542ed0b7c522d575706a80bedc848c8954",
              "url": "https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz"
            }
          ],
          "project_name": "psutil",
          "requires_dists": [
            "enum34; python_version <= \"3.4\" and extra == \"test\"",
            "ipaddress; python_version < \"3.0\" and extra == \"test\"",
            "mock; python_version < \"3.0\" and extra == \"test\"",
            "pywin32; sys_platform == \"win32\" and extra == \"test\"",
            "wmi; sys_platform == \"win32\" and extra == \"test\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7",
          "version": "5.9.1"
        }
      ],
      "platform_tag": [
        "cp311",
        "cp311",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.93",
  "prefer_older_binary": false,
  "requirements": [
    "psutil==5.9.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "transitive": true,
  "use_pep517": null
}

Leads to:

$ rm -rf ~/.pex && pex --lock lock.json -o psutil.pex
There was 1 error downloading required artifacts:
1. psutil 5.9.1 from https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz
    ERROR: Command errored out with exit status 1: /home/jsirois/.pex/venvs/s/0490b204/venv/bin/python3 /home/jsirois/.pex/venvs/s/0490b204/venv/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /home/jsirois/.pex/pip_cache/.tmp/pip-build-env-whexefy7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

So #1811, which switched Pex to using pip download --no-deps to download lock artifacts, fails for various sdists.

[jsirois]

This was spotted over in pantsbuild/pants#15990

[jsirois]

Aha, the explanation is here: pypa/pip#1884 (comment)
Pip needs to generate metadata to discover Requires-Python. IOW even though we say --no-deps; so Pip does not need Requires-Dist metadata, it does reasonably need Requires-Python in the generic case of downloading a requirement. Granted our requirements in this case are explicit URLs.

[jsirois]

But then you'd think this would work with modern Pip that supports --ignore-requires-python:

^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ python -mvenv pip.venv
^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ source pip.venv/bin/activate
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip -q install -U pip
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip -V
pip 22.1.2 from /home/jsirois/dev/pantsbuild/jsirois-pex/pip.venv/lib/python3.11/site-packages/pip (python 3.11)
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip download --no-deps --ignore-requires-python --dest /tmp/pip-download https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
Collecting https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
  File was already downloaded /tmp/pip-download/PyYAML-5.4.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Successfully downloaded PyYAML

[jsirois]

Aha, the more on-point explanation is here: pypa/pip#1884 (comment) - basically, legacy code in Pip mooshes download and prepare together and that's hard to untangle easily.