-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
--find-links equivalent #1391
Comments
Hi. I was thinking of having a go at this for Hacktoberfest. The approach I was thinking of:
Sound about right? |
Sounds like a plan 👍 |
I'll have a crack at it, probably in a second PR. In principle the same logic should be required just working with file paths instead of urls. |
Right now the workaround is defining remote file dependencies (in pytorch's case, platform specific).
Unfortunately, this will limit you to only one platform, as you can't have the same dependency for multiple platforms with different file links. |
After installing torch with
each time I add any package poetry does following:
It doesn't break anything, but takes some time. Anyone knows a way to avoid "updating"? |
Not really, it's a workaround for a current limitation in Poetry, but until that limitation is fixed, you could probably limit the impact by using a local cache (like devpi) @germn |
@germn Something like below should work fine for your case. |
@allen-chin The sideeffect of that is continual reinstallations of the package. Just because the package is local, doesn't mean Poetry won't reinstall it. |
I created a valid simple index that can be used with index-url and as a I massaged the official PyTorch find-urls file to look more like a PEP 503 compatible simple index, but it's still pointing to all of the official packages at pytorch.org. It's currently working like a charm in my poetry setup. |
@cpbotha Plan on adding 1.5.0 soon? |
Currently swamped, have not had time to get back to this and probably won't for the coming weeks. There's also the issue on pytorch's side to make available the right sort of index for their packages, see pytorch/pytorch#25639 , but no-one has had time yet to dive in. |
@cpbotha great job, but I have an issue when I am using your source with poetry when trying to lock. [SolverProblemError] |
same issue here. |
The same is true for the latest stable pytorch and torchvision packages. The officially recommended way is
but without an equivalent to the But adding the wheels manually fails because the solver thinks $ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies... (36.4s)
[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed. Should we split this into a new issue separate from the And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate. |
Most of us only have this problem because of PyTorch, right? To be honest, since I opened this issue I no longer see the problem here with Poetry, but with PyTorch. Why the hell isn't a big project like PyTorch published on a PEP-conform index? I opened an issue (pytorch/pytorch#26340) some time ago -- so far without any concrete reaction. I actually like the strict policy of Poetry. PEP exists for a reason. |
@severinsimmler I'm having the same issue (spinalcordtoolbox/spinalcordtoolbox#1526 (comment)). Thanks a lot for trying to get the ball rolling with pytorch! I agree poetry should stick close to the core pypa specs, and they want to kill But if pytorch got themselves together and published to pypi this would be a non-issue. |
See #2543 for the separate issue For what it's worth, I don't think poetry needs this |
@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves. The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package. |
Fair enough, I have not digged deep enough to uncover this root cause. Apologies for my tone. I do think that it's odd that hosting your own index is apparently too hard for such a project. Aren't there tons of such tools available to do that? |
@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.) |
Yes, well, I guess I was also not clear enough. One can also host a static HTML site that meets the PEP 503 format. That also allows it to be included as an index in pyproject.toml. See pytorch/pytorch#25639 and pytorch/pytorch#39779 |
Well, if the guys at PyTorch manage to figure it out, that's good. Poetry still probably should support the feature from pip. |
Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml? |
If it's just torch (and not also vision), you can if you set up a PEP 503 compatible index with the package on it. There's a public one hacked together by @cpbotha you can use: https://vxlabs.com/pypi/ |
I will add this as a potential workaround: It does not seem to be a valid workaround, yet. It might be that there is an issue with URL redirections in poetry, so that the workaround fails for now. |
Any news on the PR? For example, I need to use numpy+mkl on Windows 10 2004 Update because of this issue. I can download whl from gohlke web page only manually. And it would be nice to use Something like this: [tool.poetry.dependencies]
python = "^3.7"
numpy = { version = "^1.19.4+mkl", markers = "sys_platform == 'win32'" }
pandas = "^1.1.4"
[[tool.poetry.source]]
name = "numpy"
url = "wheels/" # relative path to local directory with wheel files
find_links = true |
@Korijn's solution is good but it has become outdated. |
Just linking to a related issue: #2613 |
Hi! Is there any news on this issue? |
I stumbled upon this issue because vtk does not provide wheels for Python 3.10, but unofficial ones exist: pyvista/pyvista#2064 |
Same problem when trying to install JAX via pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html # Note: wheels only available on linux.
Currently I've resorted to: poetry run pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html Which effectively excludes JAX and its deps from tracking (i.e. they're not in poetry.lock) > poetry add flax
...
Updating jaxlib (0.3.7+cuda11.cudnn82 -> 0.3.7) # "jax[cuda]" install gets overwritten effectively into "jax[cpu]"
...
Installing flax (0.4.1) So i have to rerun the UPD: |
Hi, in to-be-released Poetry 1.2, it will be possible to define single page repositories following https://python-poetry.org/docs/master/repositories/#single-page-link-source. You should be able to test that by using the recently released 1.2.0b2 pre-release. |
Closing, since it is now implemented in the next release of Poetry, 1.2, but if you still encounter any issue, feel free to reopen. |
--find-links accepts a directory path, not a single-page index. A single-page index supposes running a web server. This is not equivalent. |
We are running in an environment (out of control), where the packages are spread across different local directories. Pip was configured with --find-links to search across all the directories when installing a package, however, it does not seem like poetry is able to do the same. |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Using:
Feature Request
Because the CPU version of PyTorch is not on PyPI, they recommend installing it this way:
However, there is no way to do this with Poetry?
Pointing directly to the wheel:
results in
And adding
https://download.pytorch.org/whl/torch_stable.html
as a repository to thepyproject.toml
does not work, because it (the PyTorch wheel index page) is not PEP conform.The text was updated successfully, but these errors were encountered: