Pip package URL is not a valid index

[Korijn]

🐛 Bug

The URL presented on the Getting Started page (https://download.pytorch.org/whl/torch_stable.html) is not a valid index (i.e. the "simple" format). This means package managers like Poetry and Pipenv can't consume the packages.

Right now it works with pip because pip can search arbitrary HTML pages for download links that look like they lead to python packages. However, there are assumptions about the format in package managers like Poetry and Pipenv.

We work around this issue by mirrorring the packages in an index of our own but it seems like it would be much friendlier to users if this would be solved at your end.

I get that it may not matter much for "quick start" setups where you just want it to work on your machine, but it's a bigger problem in production deployments where reproducibility matters and thus lockfiles and dependency management are of greater importance.

PEP 503 -- Simple Repository API specifies the simple format, it only requires a couple of minor changes to the HTML you currently serve.

Edit The relevant code seems to be https://github.com/pytorch/builder/blob/master/cron/update_s3_htmls.sh
which is another repo, so perhaps the issue should be transferred?

To Reproduce

Steps to reproduce the behavior:

1. Using this pyproject.toml

[tool.poetry]
name = "bla"
version = "0.1.0"
description = ""
authors = ["Korijn van Golen <k.vangolen@clinicalgraphics.com>"]

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/torch_stable.html"
default = true

[tool.poetry.dependencies]
python = "^3.6"
torch = "1.5.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

2. poetry lock -vvv

λ poetry lock -vvv
Using virtualenv: C:\Users\korij_000\AppData\Local\pypoetry\Cache\virtualenvs\bla-8bWMJ8mE-py3.6
Updating dependencies
Resolving dependencies...
   1: fact: bla is 0.1.0
   1: derived: bla
   1: fact: bla depends on torch (1.5.0)
   1: selecting bla (0.1.0)
   1: derived: torch (1.5.0)
torch: 0 packages found for torch 1.5.0          <<<==== PROBLEM ====
   1: fact: no versions of torch match 1.5.0
   1: conflict: no versions of torch match 1.5.0
   1: ! torch (1.5.0) is satisfied by torch (1.5.0)
   1: ! which is caused by "bla depends on torch (1.5.0)"
   1: ! thus: version solving failed
   1: Version solving took 0.545 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because bla depends on torch (1.5.0) which doesn't match any versions, version solving failed.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @seemethere @malfet @walterddr

[albanD]

cc @soumith who I think originally set this up.

[soumith]

yea, @seemethere we should fix this. cc: @malfet

[ezyang]

RElated: #25639

[FatherShawn]

This could resolve a number of issues filed against pipenv

pypa/pipenv#4649
pypa/pipenv#4504
pypa/pipenv#4121

Clear answer from the pipenv team at pypa/pipenv#4649 (comment)

See also: pytorch/builder#615

[albanD]

Bumping priority as this trips up quite a few users.

[DanielPerezJensen]

Any updates on this thus far?

[mihajenko]

Would maintaining two indexes be acceptable (standard and non-standard)?

[seemethere]

This has already been resolved see #25639 (comment), closing.

[Arcitec]

Thanks a lot for creating that PEP 503 repo!

If anyone needs the correct solution for installing PyTorch via Pipenv, I have posted a guide and explanation here:

pypa/pipenv#4961 (comment)

It would be cool if the official pytorch website could list those install commands (the ones I've generated) as an option during the "roll your own selections" guide. I.e. having Pipenv as a choice next to Pip, and then showing the command-style that I'm using in my guide. Then projects that are based on Pipenv won't have to manually lookup the latest versions in the repo HTML in a browser anymore.