Fix absolute links in pytorch repository and allow it to be proxied

[stephanbertl]

🐛 Describe the bug

The pip repositories like https://download.pytorch.org/whl/cu118 do not follow common standards like PEP503.

While for itself pip this is not an issue, we are having trouble proxying the repository using Sonatype Nexus.

When looking at the actual location of the wheels, https://download.pytorch.org/whl/torch/ the links to the wheel files use absolute paths like that:

<a href="/whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl">torch-2.0.1+cu117-cp310-cp310-linux_x86_64.whl</a>

So far so good.

Now I am proxying the pytorch repo to make use of caching and faster downloads in our corporate network. Additionally direct internet access is not allowed. We are using a so-called "raw repository" inside Sonatype Nexus. We cannot use their "PyPI" repository feature, because the pytorch repository does not follow PEP503 standard. But that is a separate issue.

Our cached repository ends up at https://nexus.corporate.info/repository/pytorch-raw/ with the wheels ending up at https://nexus.corporate.info/repository/pytorch-raw/whl/torch/. The page shows links the same way as the original repo.

<a href="/whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl">torch-2.0.1+cu117-cp310-cp310-linux_x86_64.whl</a>

But a link like /whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl ends up being resolved to https://nexus.corporate.info/whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl

The correct URL should be https://nexus.corporate.info/repository/pytorch-raw/whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl

Could you please fix the absolute links and use relative ones if possible?

Alternatively, can you implement a PEP503 simple repository? What is the issue with that?

Versions

2.0.1

[janeyx99]

cc @seemethere @osalpekar @ZainRizvi @malfet

[ZainRizvi]

cc @atalman

[malfet]

@stephanbertl hmm, can you please elaborate what makes https://download.pytorch.org/whl/cu118 non PEP503 complainant?

[stephanbertl]

@malfet You are probably right, PEP503 allows also absolute URLs:

https://peps.python.org/pep-0503/

URLs may be either absolute or relative as long as they point to the correct location.

However, proxying with Nexus is not possible with absolute links. PyTorch is the only repository where we experience problems.

https://community.sonatype.com/t/not-able-to-proxy-python-link/6745/11

If the WHL links were relative with "../" instead of absolute it would just work. PyPI main repo use this convention.

[stephanbertl]

Is there any plan to implement changes to this? Currently, this really makes a lot of problems for us because we are unable to proxy pip repos that use absolute links.

[ykfq]

Same problem here while proxying the repository https://download.pytorch.org using Sonatype Nexus.

Temporarily find a solution for this:

pip install \
    --find-links https://mirror.sjtu.edu.cn/pytorch-wheels/torch_stable.html \
    torch===2.0.0+cpu

#tuna/issues#363 (comment)

[hboutemy]

@stephanbertl @ykfq I came across the same issue as you and worked with Sonatype Community to create a mirror of PyTorch index that works with Nexus Repository
see https://sonatype-nexus-community.github.io/pytorch-pypi/whl/ for the indexes
and https://github.com/sonatype-nexus-community/pytorch-pypi for documentation

feedback appreciated