poetry add with --index-url option

[alanwilter]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

I do this with pip:

pip3 install torch --index-url https://download.pytorch.org/whl/cpu

I can't find an equivalent in poetry. Suggestion:

poetry add torch --index-url https://download.pytorch.org/whl/cpu

[dimbleby]

https://python-poetry.org/docs/repositories/, please close

[alanwilter]

Thanks!

[alanwilter]

Sorry, I read the docs and clearly I'm missing something.

So I have in pyproject.toml:

[[tool.poetry.source]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
default = false
secondary = true

I do:

poetry add --source pytorch-cpu torch

and see:

[tool.poetry.dependencies]
python = ">=3.9,<3.11"
typed-argument-parser = "^1.7.2"
opencv-python = "4.6.0.66"
pillow = "^9.3.0"
tqdm = "^4.64.1"
scikit-image = "^0.19.3"
numpy = "^1.23.5"
tornado = "^6.2"
parse = "^1.19.0"
more-itertools = "^9.0.0"
torch = {version = "^2.0.0+cpu", source = "pytorch-cpu"}

[tool.poetry.group.dev.dependencies]
isort = "^5.11.2"
pyupgrade = "^3.3.1"
black = "^22.12.0"
mypy = "^0.991"
flake8 = "^6.0.0"
pre-commit = "^2.20.0"
pytest-cov = "^4.0.0"
ipdb = "^0.13.11"
pdbpp = "^0.10.3"

All looks fine. Then I try poetry lock or install and I got this:

Resolving dependencies... (4.6s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/pdbpp/
Resolving dependencies... (4.9s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/ipdb/
Resolving dependencies... (5.2s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/pytest-cov/
Resolving dependencies... (5.6s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/pre-commit/
Resolving dependencies... (5.9s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/flake8/
Resolving dependencies... (6.3s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/mypy/
Resolving dependencies... (6.4s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/black/
Resolving dependencies... (6.9s)Source (pytorch-cpu): Authorization error accessing https://download.pytorch.org/whl/cpu/pyupgrade/
...

I tried to convert all my entries to:

isort = { version = "^5.11.2", source = "pypi" }
...

Play with secondary = to true of false and I always get the error message.

Basically, I want to use pypi as usual, just torch I want to install as a dependency from https://download.pytorch.org/whl/cpu. And if not asking much, I want to install the usual torch from pypi into [tool.poetry.group.dev.dependencies].

The rationale here is: I want to publish a package with torch-cpu dependency but I want to be able to work locally using the torch-gpu enabled.

[dimbleby]

please read the docs again - specifically the bit that says "All package sources (including secondary sources) will be searched during the package lookup process." - and subscribe to #6713

[alanwilter]

Ok, I got it, I just have to wait it searches...

I was having difficult to understand this bit:

Any package source not marked as secondary will take precedence over PyPI.

I still don't understand that statement, but I glad I got it to work as I wanted.

[tool.poetry.dependencies]
torch = {version = "^2.0.0+cpu", source = "pytorch-cpu"}
...

[tool.poetry.group.dev.dependencies]
torch = {version = "^2.0.0", source = "pypi"}
...

[hyliu1989]

Thought I could share some successful experience on the same issue.

After updating poetry to 1.5.1, there is a new package URL specification priority call explicit and that works more clear to me.

poetry source add --priority=explicit pytorch-gpu-src https://download.pytorch.org/whl/cu118
poetry add --source pytorch-gpu-src torch torchvision torchaudio

I can successfully generate a pyproject.toml with the following section

[tool.poetry.dependencies]
python = ">=3.9, <3.11"
torch = {version = "^2.0.1+cu118", source = "pytorch-gpu-src"}
torchvision = {version = "^0.15.2+cu118", source = "pytorch-gpu-src"}
torchaudio = {version = "^2.0.2+cu118", source = "pytorch-gpu-src"}

And the commandline (using PowerShell) does not complain anything

Hope this helps for people revisiting here.

ps. I was trying to install PyTorch

[Martins6]

I'm going to say what I think everyone that came to this issue is thinking. The answer from @hyliu1989 should be on the documentation of poetry add (https://python-poetry.org/docs/cli/#add) to give more visibility to users that use PyTorch and other specific repositories.

[dimbleby]

@Martins6 if you want to see docs changes then by far the best way to make that happen is to submit a merge request

[hyliu1989]

@dimbleby @Martins6 I just created a PR. Thanks for the suggestion.

[Aeolun]

The best way to make things happen has failed. What is the second best way?

[radoering]

if you want to see docs changes then by far the best way to make that happen is to submit a merge request

... and respond to maintainer's feedback on your merge request

[rbavery]

from #8246 (review)

I think I'd prefer to just add a short sentence with a link to https://python-poetry.org/docs/repositories/#installing-from-private-package-sources instead of having almost the same in two places.

As a newcomer to using Poetry, I like the call out to Pytorch in @hyliu1989 's PR. it is the most common case where private repositories would be used. Also before reading this thread, I wouldn't know to look at the private repository page, the first place I would look is the docs for poetry add so I like that the PR goes into detail here.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.