Poetry seems to ignore dependencies of pytorch 2.0.1 but correctly installs them from local whl file

[twoertwein]

• Poetry version: 1.4.2
• Python version: 3.11.3
• OS version and name: Fedora 38

The following pyproject.toml does not install pytorch's dependecies. When manually downloading the whl from pipy, poetry correctly installs all dependecies.

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["Foo Bar <foo@bar.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
torch = "2.0.1"  # does NOT install nvidia-cublas-cu11
# torch = { file = "path/to/torch-2.0.1-cp311-cp311-manylinux1_x86_64.whl"}  # installs nvidia-cublas-cu11


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

xref pytorch/pytorch#100974

[dimbleby]

That's not one of pytorch's dependencies, according to:

$ curl -s https://pypi.org/pypi/torch/2.0.1/json | jq '.info.requires_dist'
[
  "filelock",
  "typing-extensions",
  "sympy",
  "networkx",
  "jinja2",
  "opt-einsum (>=3.3) ; extra == 'opt-einsum'"
]

Per many other issues: poetry builds cross-platform solutions and expects all distributions of a particular version of a package to have the same requirements, with per-platform variation encoded in pep508 markers.

Please close

[twoertwein]

Thank you for the quick response!

[Queuecumber]

Hi @dimbleby I'm working with some pytorch people to try to fix this, would you know where I can get more information about how that json API works? We're not sure how to make sure it has the right information

[dimbleby]

As I understand it, dependencies on the JSON API are taken from the metadata in the first wheel that is uploaded.

But rather than relying on such wrinkles, it is better to put the same metadata in all uploaded distributions, and use markers for platform-specific variation

[Queuecumber]

Just wanted to give an update, this was supposed to be fixed in the latest pytorch release, but 2.1.0 just dropped and still has this issue so we need to loop back with Meta and figure out whats going wrong

[dimbleby]

it looks as though https://files.pythonhosted.org/packages/e1/24/f7fe3fe82583e6891cc3fceeb390f192f6c7f1d87e5a99a949ed33c96167/torch-2.1.0-cp38-cp38-manylinux1_x86_64.whl contains the cuda dependencies with platform-specific markers: but other wheels don't - and that one happens not to have been the first to be uploaded.

see #7902 (comment)

[MischaPanch]

@Queuecumber is there any issue in torch that is tracking this? It's a real bummer for all poetry-based projects

[Queuecumber]

I believe it's being tracked internally, I know for sure it's on our list (Nvidia), hopefully we can get it resolved but I wouldn't hold out hope for anything that's already been released

…

On Mon, Oct 9, 2023, 16:05 Michael Panchenko ***@***.***> wrote: @Queuecumber <https://github.com/Queuecumber> is there any issue in torch that is tracking this? It's a real bummer for all poetry-based projects — Reply to this email directly, view it on GitHub <#7902 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMX7K6USWFC76GRKVHG4D3X6RKJDAVCNFSM6AAAAAAX6FA2NGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJTGY3TGNBQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MischaPanch]

Can we at least reopen this issue then @twoertwein ? It seems the only place to which one can link for describing this problem, and it's confusing that the issue is closed. I undestand that it's not an issue of poetry, but still it could be kept open until torch+nvidia resolve this in a new release

[dimbleby]

If you want to track a torch problem, then do it in the torch repository - ie open an issue there

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.