Newer versions of pip fail to install from PyPI proxy

[tieneupin]

Description:

• Newer versions of pip are unable to find the wheel metadata needed to build the package from the proxy PyPI index.
• Using pip install --index-url --extra-index-url, but with the Simple PyPI Index as the URL instead, works without issue.
• This problem does not occur when using pip <=v22.2.2.

pip versions tested:

• 24.0
• 23.3.1
• 23.3
• 23.2.1
• 23.1.2
• 23.0.1
• 22.3.1
• 22.3 (fails above here)
• 22.2.2 (passes below here)
• 22.1.2
• 22.0

[tieneupin]

Narrowed issue down to this function in src/murfey/servery/bootstrap.py

@pypi.get("/{package}/", response_class=Response)
def get_pypi_package_downloads_list(package: str):
    """Obtain list of all package downloads from PyPI via the simple API (PEP 503),
    and rewrite all download URLs to point to this server,
    underneath the current directory."""
    full_path_response = requests.get(f"https://pypi.org/simple/{package}")

    # This function to rewrite the PyPI URL returns one that pip doesn't accept
    def rewrite_pypi_url(match):
        url = match.group(4)
        return (
            b"<a "
            + match.group(1)
            + b'href="'
            + url
            + b'"'
            + match.group(3)
            + b">"
            + match.group(4)
            + b"</a>"
        )

    content = re.sub(
        b'<a ([^>]*)href="([^">]*)"([^>]*)>([^<]*)</a>',
        rewrite_pypi_url,
        full_path_response.content,
    )
    return Response(
        content=content,
        media_type=full_path_response.headers.get("Content-Type"),
        status_code=full_path_response.status_code,
    )

[tieneupin]

Examining the HTML code behind the scenes, the issue seems to be with the difference in the <a href="..."> section of the code. Initial guess is that the ```#sha256=... string after the package name will have to be included as well.

PyPI Simple Index Entry

<a href="https://files.pythonhosted.org/packages/e6/33/de41e554e5a187d583906e10d53bfae5fd6c07e98cbf4fe5262bd37e739a/fastapi-0.111.0-py3-none-any.whl#sha256=97ecbf994be0bcbdadedf88c3150252bed7b2087075ac99735403b1b76cc8fc0" data-requires-python=">=3.8" data-dist-info-metadata="sha256=13886d9e031d8755cca2b4ce5c6e3d438fcd883e5d9126ac12f3b3cd2bbde5c1" data-core-metadata="sha256=13886d9e031d8755cca2b4ce5c6e3d438fcd883e5d9126ac12f3b3cd2bbde5c1">fastapi-0.111.0-py3-none-any.whl</a>
<br>
<a href="https://files.pythonhosted.org/packages/0e/1f/f4a99e92c583780787e04b05aa9d8a8db9ec76d091d81545948a006f5b44/fastapi-0.111.0.tar.gz#sha256=b9db9dd147c91cb8b769f7183535773d8741dd46f9dc6676cd82eab510228cd7" data-requires-python=">=3.8">fastapi-0.111.0.tar.gz</a>
<br>

Our Proxy

<a href="fastapi-0.111.0-py3-none-any.whl" data-requires-python=">=3.8" data-dist-info-metadata="sha256=13886d9e031d8755cca2b4ce5c6e3d438fcd883e5d9126ac12f3b3cd2bbde5c1" data-core-metadata="sha256=13886d9e031d8755cca2b4ce5c6e3d438fcd883e5d9126ac12f3b3cd2bbde5c1">fastapi-0.111.0-py3-none-any.whl</a>
<br>
<a href="fastapi-0.111.0.tar.gz" data-requires-python=">=3.8">fastapi-0.111.0.tar.gz</a>
<br>