Universal lock creation failure for disjoint IC and Requires-Python is correct but the error message from Pip / Pex is confusing.

[jsirois]

The issue was surfaced / demonstrated in https://github.com/tobni/dummy-pants-repo by running:

$ ./pants generate-lockfiles
20:15:56.82 [INFO] Completed: Generate lockfile for python-default
20:15:56.83 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 22775 -> /home/jsirois/.cache/pants/named_caches/pex_root/venvs/c7244f8b84e19e8bd6c885094b310b0b5b2a01ee/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.cache/pants/named_caches/pex_root/venvs/c7244f8b84e19e8bd6c885094b310b0b5b2a01ee/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-Ylww3s/.tmp/pex-pip-log.5ntpw7pn/pip.log download --dest /tmp/pants-sandbox-Ylww3s/.tmp/tmpkgqxtrcm/usr.bin.python3.10 --requirement __pip_args.txt numpy==1.21.4 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement numpy==1.21.4
ERROR: No matching distribution found for numpy==1.21.4




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

The underlying relevant Pex command line is:

$ pex3 lock create --resolver-version pip-2020-resolver --style universal --target-system linux --target-system mac numpy==1.21.4 --interpreter-constraint "~=3.10" -o lock.json --indent 2 --preserve-pip-download-log
pex: Preserving `pip download` log at /tmp/pex-pip-log.z4dpx96l/pip.log
pid 22816 -> /home/jsirois/.pex/venvs/d71f2a9856ffe0051f78792ca856ec7cc0e434b3/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/d71f2a9856ffe0051f78792ca856ec7cc0e434b3/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.pex/pip_cache --log /tmp/pex-pip-log.z4dpx96l/pip.log download --dest /tmp/tmpmqwlvz6o/home.jsirois.bin.pex.venv.bin.python numpy==1.21.4 --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement numpy==1.21.4
ERROR: No matching distribution found for numpy==1.21.4

That reveals:

$ grep -i "link requires a different python" /tmp/pex-pip-log.z4dpx96l/pip.log | grep "1\.21\.4"
2022-10-02T20:17:36,606   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/76/9a/a245290ae642f78caf80d1454045d19119dd6f266a81f02a1aa7aa89ba99/numpy-1.21.4-cp310-cp310-macosx_10_9_universal2.whl#sha256=8890b3360f345e8360133bc078d2dacc2843b6ee6059b568781b15b97acbe39f (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2022-10-02T20:17:36,608   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/de/54/3f7d10f5030195be5fb2adc689239d2e5356e3cf6be4d2c046f1ab635589/numpy-1.21.4-cp310-cp310-macosx_10_9_x86_64.whl#sha256=69077388c5a4b997442b843dbdc3a85b420fb693ec8e33020bb24d647c164fa5 (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2022-10-02T20:17:36,609   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5d/0a/96864521e42ad6a81069a4cc19ae31f6e4eae956ed03970b6174ffdac8b8/numpy-1.21.4-cp310-cp310-macosx_11_0_arm64.whl#sha256=e89717274b41ebd568cd7943fc9418eeb49b1785b66031bc8a7f6300463c5898 (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2022-10-02T20:17:36,611   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/d5/99/7d0271fac620d938ad4fc06a53d9d42245529a1ced5416dd6744b93afe78/numpy-1.21.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=0b78ecfa070460104934e2caf51694ccd00f37d5e5dbe76f021b1b0b0d221823 (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2022-10-02T20:17:36,612   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/ab/94/2fc54cc791846812318080a4f86f9afcf661891163779684d1dd1fe018f9/numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=615d4e328af7204c13ae3d4df7615a13ff60a49cb0d9106fde07f541207883ca (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2022-10-02T20:17:36,615   Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip#sha256=e6c76a87633aa3fa16614b61ccedfae45b91df2767cf097aa9c933932a7ed1e0 (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)

Now, ~=3.10 means >=3.10,==3.10.* (see here: https://peps.python.org/pep-0440/#compatible-release) which means any Python 3.10 compatible distribution. Since numpy 1.21.4 is compatible with all Python 3.10 versions (requires-python:>=3.7,<3.11), the lock should succeed. Instead we see the error message from Pip: Link requires a different Python (3.10.4 not in: '>=3.7,<3.11').

If we use the equivalent IC syntax, the lock does succeed; so the issue is specifically with the ~= operator:

$ pex3 lock create --resolver-version pip-2020-resolver --style universal --target-system linux --target-system mac numpy==1.21.4 --interpreter-constraint ">=3.10,==3.10.*" -o lock.json --indent 2 --preserve-pip-download-log
pex: Preserving `pip download` log at /tmp/pex-pip-log.uozyqz1k/pip.log

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "615d4e328af7204c13ae3d4df7615a13ff60a49cb0d9106fde07f541207883ca",
              "url": "https://files.pythonhosted.org/packages/ab/94/2fc54cc791846812318080a4f86f9afcf661891163779684d1dd1fe018f9/numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "e89717274b41ebd568cd7943fc9418eeb49b1785b66031bc8a7f6300463c5898",
              "url": "https://files.pythonhosted.org/packages/5d/0a/96864521e42ad6a81069a4cc19ae31f6e4eae956ed03970b6174ffdac8b8/numpy-1.21.4-cp310-cp310-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "8890b3360f345e8360133bc078d2dacc2843b6ee6059b568781b15b97acbe39f",
              "url": "https://files.pythonhosted.org/packages/76/9a/a245290ae642f78caf80d1454045d19119dd6f266a81f02a1aa7aa89ba99/numpy-1.21.4-cp310-cp310-macosx_10_9_universal2.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "0b78ecfa070460104934e2caf51694ccd00f37d5e5dbe76f021b1b0b0d221823",
              "url": "https://files.pythonhosted.org/packages/d5/99/7d0271fac620d938ad4fc06a53d9d42245529a1ced5416dd6744b93afe78/numpy-1.21.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "69077388c5a4b997442b843dbdc3a85b420fb693ec8e33020bb24d647c164fa5",
              "url": "https://files.pythonhosted.org/packages/de/54/3f7d10f5030195be5fb2adc689239d2e5356e3cf6be4d2c046f1ab635589/numpy-1.21.4-cp310-cp310-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "e6c76a87633aa3fa16614b61ccedfae45b91df2767cf097aa9c933932a7ed1e0",
              "url": "https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip"
            }
          ],
          "project_name": "numpy",
          "requires_dists": [],
          "requires_python": "<3.11,>=3.7",
          "version": "1.21.4"
        }
      ],
      "platform_tag": null
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.107",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "numpy==1.21.4"
  ],
  "requires_python": [
    "==3.10.*,>=3.10"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [
    "linux",
    "mac"
  ],
  "transitive": true,
  "use_pep517": null
}

[jsirois]

OK, re-read https://peps.python.org/pep-0440/#compatible-release and ~=3.10 does mean >=3.10,==3.* aka >=3.10,<4 and so the failure to lock is legitimate and this is just an error message issue.

[jsirois]

Circling back to the confusing error message in the Pip download logs of Link requires a different Python (3.10.4 not in: '>=3.7,<3.11'). That text is generated here:
https://github.com/pantsbuild/pex/blob/196b4cd5b8dd4b4af2586460530e9a777262be7d/pex/vendor/_vendored/pip/pip/_internal/index/package_finder.py#L60-L92

Pex monkey-patches the check_requires_python in packaging here:
https://github.com/pantsbuild/pex/blob/196b4cd5b8dd4b4af2586460530e9a777262be7d/pex/vendor/_vendored/pip/pip/_internal/index/package_finder.py#L76-L78

But that leaves the error message using the version of the current Python interpreter (3.10.4 in this case) which is confusing since the monkey-patch actually ran over all versions 3.10.* and 3.11.* and it's the 3.11.* which did not match the requirement.