--find-links equivalent

[severinsimmler]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Using:

• Python 3.7.3
• Poetry 1.0.0b1

Feature Request

Because the CPU version of PyTorch is not on PyPI, they recommend installing it this way:

pip install torch==1.2.0+cpu --find-links https://download.pytorch.org/whl/torch_stable.html

However, there is no way to do this with Poetry?

Pointing directly to the wheel:

poetry add https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl

results in

Updating dependencies
Resolving dependencies... (2.0s)


Package operations: 1 install, 0 updates, 0 removals

  - Installing torch (1.2.0+cpu https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl)

[EnvCommandError]
Command ['/home/simmler/.cache/pypoetry/virtualenvs/epss-WeFenOJ_-py3.7/bin/python', '-m', 'pip', 'install', '--no-deps', 'torch==1.2.0+cpu'] errored with the following return code 1, and output: 
Collecting torch==1.2.0+cpu
  Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0)
No matching distribution found for torch==1.2.0+cpu

And adding https://download.pytorch.org/whl/torch_stable.html as a repository to the pyproject.toml does not work, because it (the PyTorch wheel index page) is not PEP conform.

[cc-a]

Hi. I was thinking of having a go at this for Hacktoberfest.

The approach I was thinking of:

• Implement a variation of the LegacyRepository class
• Extend the expected fields for the source toml section to add a flag that specifies this type of simple html repository
• Implement the relevant logic in poetry.py to determine the correct repository class to use

Sound about right?

[severinsimmler]

Sounds like a plan 👍

[woodlandhunter]

@cc-a a --find-links that is equivalent to pip's would also accept file paths (a directory). See pip docs here. We could really use this. It's also mentioned in these issues:

#558
#1289

Your comment doesn't mention this scenario, would --find-links=DIR be covered by what you are working on?

[cc-a]

I'll have a crack at it, probably in a second PR. In principle the same logic should be required just working with file paths instead of urls.

[golyalpha]

Right now the workaround is defining remote file dependencies (in pytorch's case, platform specific).
This works for me on the current release:

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl --platform win32

Unfortunately, this will limit you to only one platform, as you can't have the same dependency for multiple platforms with different file links.

[germn]

After installing torch with

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl

each time I add any package poetry does following:

- Updating torch (1.3.1 -> 1.3.1 https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl)

It doesn't break anything, but takes some time.

Anyone knows a way to avoid "updating"?

[golyalpha]

Not really, it's a workaround for a current limitation in Poetry, but until that limitation is fixed, you could probably limit the impact by using a local cache (like devpi) @germn

[allen-chin]

@germn Something like below should work fine for your case.
pyspark = {file = "pyspark-3.0.0.dev0.tar.gz"}

[golyalpha]

@allen-chin The sideeffect of that is continual reinstallations of the package. Just because the package is local, doesn't mean Poetry won't reinstall it.

[cpbotha]

I created a valid simple index that can be used with index-url and as a tool.poetry.source for all of the pytorch.org official packages. You can find the new index here: https://vxlabs.com/pypi/

I massaged the official PyTorch find-urls file to look more like a PEP 503 compatible simple index, but it's still pointing to all of the official packages at pytorch.org.

It's currently working like a charm in my poetry setup.

[aviramha]

@cpbotha Plan on adding 1.5.0 soon?
BTW Any news on this issue?

[cpbotha]

Currently swamped, have not had time to get back to this and probably won't for the coming weeks.

There's also the issue on pytorch's side to make available the right sort of index for their packages, see pytorch/pytorch#25639 , but no-one has had time yet to dive in.

[timorkal]

@cpbotha great job, but I have an issue when I am using your source with poetry when trying to lock.
pyproject.toml:
torch = "1.3.1+cpu"
torchvision = "0.4.2+cpu"

[SolverProblemError]
Because torchvision (0.4.2+cpu) depends on torch (1.3.1)
and package depends on torch (1.3.1+cpu), torchvision is forbidden.
So, because package on torchvision (0.4.2+cpu), version solving failed.

[lironmo]

same issue here.
@cpbotha i tried to add your private pip repository with no success:
poetry config repositories.torch-mirror https://vxlabs.com/pypi

[daniel-j-h]

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

[severinsimmler]

Most of us only have this problem because of PyTorch, right? To be honest, since I opened this issue I no longer see the problem here with Poetry, but with PyTorch. Why the hell isn't a big project like PyTorch published on a PEP-conform index? I opened an issue (pytorch/pytorch#26340) some time ago -- so far without any concrete reaction.

I actually like the strict policy of Poetry. PEP exists for a reason.

[kousu]

@severinsimmler I'm having the same issue (spinalcordtoolbox/spinalcordtoolbox#1526 (comment)). Thanks a lot for trying to get the ball rolling with pytorch!

I agree poetry should stick close to the core pypa specs, and they want to kill --find-links and dependency_links in favour of https://www.python.org/dev/peps/pep-0508/#environment-markers (pypa/pip#6162, pypa/pip#4187) so you shouldn't try to reintroduce that feature here. But PEP 508's URL mode doesn't cover wheels with compiled platform-specific versions :/ so we're stuck.

But if pytorch got themselves together and published to pypi this would be a non-issue.

[Korijn]

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

See #2543 for the separate issue

For what it's worth, I don't think poetry needs this --find-links feature. Rather, the pytorch team needs to adopt proper packaging practices..

[golyalpha]

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

[Korijn]

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

Fair enough, I have not digged deep enough to uncover this root cause. Apologies for my tone.

I do think that it's odd that hosting your own index is apparently too hard for such a project. Aren't there tons of such tools available to do that?

[golyalpha]

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

[Korijn]

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

Yes, well, I guess I was also not clear enough. One can also host a static HTML site that meets the PEP 503 format. That also allows it to be included as an index in pyproject.toml. See pytorch/pytorch#25639 and pytorch/pytorch#39779

[golyalpha]

Well, if the guys at PyTorch manage to figure it out, that's good. Poetry still probably should support the feature from pip.

[RafalSkolasinski]

Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

[Korijn]

Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

If it's just torch (and not also vision), you can if you set up a PEP 503 compatible index with the package on it. There's a public one hacked together by @cpbotha you can use: https://vxlabs.com/pypi/

[sinoroc]

I will add this as a potential workaround:

• Route as '--find-links' replacement uranusjr/simpleindex#2

It does not seem to be a valid workaround, yet. It might be that there is an issue with URL redirections in poetry, so that the workaround fails for now.

[espdev]

Any news on the PR?

For example, I need to use numpy+mkl on Windows 10 2004 Update because of this issue. I can download whl from gohlke web page only manually. And it would be nice to use --find-links and the local wheelhouse directory for searching wheel-packages instead of remote wheelhouse:

Something like this:

[tool.poetry.dependencies]
python = "^3.7"
numpy = { version = "^1.19.4+mkl", markers = "sys_platform == 'win32'" }
pandas = "^1.1.4"

[[tool.poetry.source]]
name = "numpy"
url = "wheels/"  # relative path to local directory with wheel files
find_links = true

[timothyjlaurent]

@Korijn's solution is good but it has become outdated.
https://eternalphane.github.io/pytorch-pypi/ seems to be up to date though.

[jhrmnn]

Just linking to a related issue: #2613

[Nathan-Furnal]

Hi! Is there any news on this issue?

[Sbozzolo]

I stumbled upon this issue because vtk does not provide wheels for Python 3.10, but unofficial ones exist: pyvista/pyvista#2064

[ColdTeapot273K]

Same problem when trying to install JAX via

pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html  # Note: wheels only available on linux.

Currently I've resorted to:

poetry run pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html    

Which effectively excludes JAX and its deps from tracking (i.e. they're not in poetry.lock)
Which in turn results in overwriting them with poetry-accessible dependencies once i try to install something that depends on them, i.e.:

> poetry add flax
...
Updating jaxlib (0.3.7+cuda11.cudnn82 -> 0.3.7) # "jax[cuda]" install gets overwritten effectively into "jax[cpu]"
...
Installing flax (0.4.1)

So i have to rerun the pip install command again after every poetry update. "Cat and mouse" basically 🐁🐈‍⬛ but at least it works.

---

UPD:
Found a community member has hosted a PEP 503 compliant index (similar to #1391 (comment)) for jax over here: google/jax#5410 (comment)

[mkniewallner]

Hi, in to-be-released Poetry 1.2, it will be possible to define single page repositories following https://python-poetry.org/docs/master/repositories/#single-page-link-source.

You should be able to test that by using the recently released 1.2.0b2 pre-release.

[mkniewallner]

Closing, since it is now implemented in the next release of Poetry, 1.2, but if you still encounter any issue, feel free to reopen.

[mboisson]

--find-links accepts a directory path, not a single-page index. A single-page index supposes running a web server. This is not equivalent.

[rashidnhm]

We are running in an environment (out of control), where the packages are spread across different local directories. Pip was configured with --find-links to search across all the directories when installing a package, however, it does not seem like poetry is able to do the same.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.