[v2.20.0] Lockfile generation with multiple indexes fails due to mismatching hashes

[seifertm]

Describe the bug
Upgrading Pants from v2.19.1 (PEX pinned to v2.1.137) to v2.20.0 breaks lockfile generation for PyTorch when multiple indexes are involved:

cat << EOF >pants.toml
[GLOBAL]
pants_version = "2.20.0"
backend_packages = [
  "pants.backend.python",
]
[python]
enable_resolves = true
interpreter_constraints = ["==3.11.*"]

[python.resolves]
python-default = "3rdparty/python/lock.txt"

[python-repos]
indexes = [
    "https://pypi.org/simple/",
    "https://download.pytorch.org/whl/cpu",
]
EOF

cat << EOF >BUILD
python_requirement(
    name="torch_cuda",
    requirements=[
        "torch==2.2.1,!=2.2.1+cpu",
    ],
)
EOF

pants generate-lockfiles

Actual output:

14:36:40.10 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
Expected sha256 hash of 26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed when downloading torch but hashed to b90669b162984e302fbd05c9c270ef796e467903944ecefa7457babe9611607e.



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

Apparently, torch-2.2.1-cp39-none-macosx_11_0_arm64.whl has different content on both indexes.

$ sha256sum torch-2.2.1-cp39-none-macosx_11_0_arm64.whl  # PyPI version
26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed  torch-2.2.1-cp39-none-macosx_11_0_arm64.whl
$ sha256sum torch-2.2.1-cp39-none-macosx_11_0_arm64.whl  # PyTorch repository version
b90669b162984e302fbd05c9c270ef796e467903944ecefa7457babe9611607e  torch-2.2.1-cp39-none-macosx_11_0_arm64.whl

Pants version
v2.20.0

OS
Linux

Additional information
I suspected the PEX upgrade as part of the Pants upgrade to be the culprit, so I started testing different PEX versions. I found that PEX v2.1.148 creates the lockfile as expected, whereas v2.1.149, v2.1.150, and v2.3.1 don't. The diff look unsuspicious to me, though.

Related to #18965.

[cburroughs]

I'm a little confused on two fronts here:

First you have interpreter_constraints = ["==3.11.*"] but the quoted hash mismatches look like Python 3.9?

Second, when I run this example I also get hash mismatches with Pants 2.16 through 2.19.

[seifertm]

I can reproduce the same hash mismatch with the following PEX call (with Pex v2.3.1, line breaks added for clarity):

$ PEX_VERBOSE=3 pex3 lock create --style universal --pypi --index https://download.pytorch.org/whl/cpu --interpreter-constraint "==3.11.*" -o lock.json --indent 2 --pip-version 24.0 "torch==2.2.1,!=2.2.1+cpu"

[…]

pex: Resolving interpreters :: Indexing downloads :: Downloading FileArtifact(
  url=ArtifactURL(
    raw_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    download_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    normalized_url='https://files.pythonhosted.org/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    scheme='https',
    path='/packages/d7/23/e4d34a78adc26ea0415fbe78fb2f940bac2c0b637db34f4fa0538a57f832/torch-2.2.1-cp39-none-macosx_11_0_arm64.whl',
    fragment_parameters={},
    fingerprints=()
  ),
  fingerprint=Fingerprint(
    algorithm='sha256',
    hash='26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed'
  ),
  verified=False,
Expected sha256 hash of 26bd2272ec46fc62dcf7d24b2fb284d44fcb7be9d529ebf336b9860350d674ed when downloading torch but hashed to b90669b162984e302fbd05c9c270ef796e467903944ecefa7457babe9611607e.

When I leave out both the --pip-version and --interpreter-constraint options from the above command, the lockfile is generated successfully.

[seifertm]

I tried the different pip versions listed in Pex's --pip-version option. The lockfile creation was unsuccessful with all of them.

[seifertm]

The reproducer has just one resolve, but two indexes. In reality, I work with multiple resolves for PyTorch, one that includes the CPU only version, and another one that includes the CUDA version.

In case someone else runs into the same issue: I work around the described problem by generating the lockfile for the CPU resolve, disabling the second package index in pants.toml, generating the CUDA resolve, and resetting pants.toml to its original state.