-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Installing PyTorch from default conda command #1544
Comments
I have the same issue. The following
I can work around it by installing PyTorch via PIP:
|
Okay, the workaround stops working as soon as I add any other dependency like
|
And another update. Looks like this is related to the channel order. If I set
|
This gets even more complicated if you want to add
It is possible to resolve this issue by changing the channel order to
|
I think there is a more fundamental issue hiding here: If an installation fails due to the strict channel order, there might still be a way. I think that should be made more explicit in the failed solving. Either by automatically attempting an alternative channel order - maybe as a dry-run, and suggest the possible channel order (if any). Or maybe at the least suggest trying an alternative channel order if EDIT: Found out that is already an active issue: #1254 |
I also came across this issue, like @roaldarbol stated the error message is non obvious that its a channel ordering issue. By moving pytorch to the bottom of the list of channel orderings I managed to get py-opencv installed, but this error message made me think there was some other incompatibility default) pablo@pablo-ubuntu:~/0Dev/personal/sam2-dust3r$ pixi add py-opencv
WARN Defined custom mapping channel https://conda.anaconda.org/pytorch/ is missing from project channels
ERROR error=repodata not found
× failed to solve the conda requirements of 'default' 'linux-64'
╰─▶ Cannot solve the request because of: py-opencv * cannot be installed because there are no viable options:
└─ py-opencv 4.2.0 | 4.2.0 | 4.5.0 | 4.5.0 | 4.5.0 would require
└─ libopencv ==4.2.0 py36_5, for which no candidates were found.
The following packages are incompatible
├─ rerun-sdk >=0.17.0,<0.18 can be installed with any of the following options:
│ ├─ rerun-sdk 0.17.0 would require
│ │ └─ python_abi 3.10.* *_cp310, which can be installed with any of the following options:
│ │ └─ python_abi 3.10
│ ├─ rerun-sdk 0.17.0 would require
│ │ └─ python_abi 3.9.* *_cp39, which can be installed with any of the following options:
│ │ └─ python_abi 3.9 | 3.9
│ └─ rerun-sdk 0.17.0 would require
│ └─ python_abi 3.8.* *_cp38, which can be installed with any of the following options:
│ └─ python_abi 3.8
├─ py-opencv * can be installed with any of the following options:
│ ├─ py-opencv 4.2.0 | 4.3.0 | 4.4.0 would require
│ │ ├─ python_abi 3.6.* *_cp36m, which can be installed with any of the following options:
│ │ │ └─ python_abi 3.6
│ │ └─ python >=3.6,<3.7.0a0, which can be installed with any of the following options:
│ │ └─ python 3.6.9 | 3.6.9 | 3.6.9 | 3.6.12 | 3.6.12
│ ├─ py-opencv 4.2.0 | 4.3.0 | 4.4.0 would require
│ │ ├─ python_abi ==3.6 *_pypy36_pp73, which can be installed with any of the following options:
│ │ │ └─ python_abi 3.6 | 3.6
│ │ └─ python >=3.6,<3.7.0a0, which can be installed with any of the following options:
│ ├─ py-opencv 4.2.0 | 4.2.0 | 4.3.0 | 4.4.0 would require
│ │ ├─ python_abi 3.8.* *_cp38, which can be installed with any of the following options:
│ │ │ └─ python_abi 3.8
│ │ └─ python >=3.8,<3.9.0a0, which can be installed with any of the following options:
│ │ └─ python 3.8.12 | 3.8.13 | 3.8.16
│ └─ py-opencv 4.2.0 | 4.2.0 | 4.3.0 | 4.4.0 would require
│ ├─ python_abi 3.7.* *_cp37m, which can be installed with any of the following options:
│ │ └─ python_abi 3.7 | 3.7 | 3.7
│ └─ python >=3.7,<3.8.0a0, which can be installed with any of the following options:
│ └─ python 3.7.9 | 3.7.9 | 3.7.10 | 3.7.10 | 3.7.12
└─ python 3.11.* cannot be installed because there are no viable options:
└─ python 3.11.0 | 3.11.0 | 3.11.0 | 3.11.0 | 3.11.1 | 3.11.2 | 3.11.3 | 3.11.4 | 3.11.5 | 3.11.6 | 3.11.7 | 3.11.7 | 3.11.8 | 3.11.9
would constrain
└─ python_abi 3.11.* *_cp311 , which conflicts with any installable versions previously reported and all that was required was going from this channels = [
"https://prefix.dev/ai-demos",
"nvidia",
"nvidia/label/cuda-12.1.0",
"pytorch",
"conda-forge",
] to this channels = [
"https://prefix.dev/ai-demos",
"nvidia",
"nvidia/label/cuda-12.1.0",
"conda-forge",
"pytorch",
] The error message makes it difficult to tell exactly why its failing |
another example I have is with the
https://prefix.dev/channels/conda-forge/packages/mmcv for this pixi.toml [project]
authors = [{ name = "pablovela5620", email = "pablovela5620@gmail.com" }]
dependencies = []
description = "Add a short description here"
name = "sam2-dust3r"
requires-python = ">= 3.11"
version = "0.1.0"
[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]
[tool.pixi.project]
channels = [
"https://prefix.dev/ai-demos",
"nvidia",
"nvidia/label/cuda-12.1.0",
"conda-forge",
"pytorch",
]
conda-pypi-map = { "pytorch" = "torch_mapping.json" }
platforms = ["linux-64"]
[tool.pixi.system-requirements]
cuda = "12"
libc = { family = "glibc", version = "2.31" }
[tool.pixi.pypi-dependencies]
sam-2 = { git = "https://github.com/facebookresearch/segment-anything-2.git", rev = "0db838b11726893f151fa5826ecfa744e1a7760f", extras = [
"demo",
] }
sam2-dust3r = { path = ".", editable = true }
[tool.pixi.dependencies]
python = "3.11.*"
pytorch = { version = "2.3.1", channel = "pytorch" }
torchvision = { version = "0.18.1", channel = "pytorch" }
cuda = { version = "*", channel = "nvidia/label/cuda-12.1.0" }
pytorch-cuda = { version = "12.1.*", channel = "pytorch" }
huggingface_hub = ">=0.24.5,<0.25"
rerun-sdk = ">=0.17.0,<0.18"
simplecv = { version = ">=0.0.2,<0.0.3", channel = "https://prefix.dev/ai-demos" }
[tool.pixi.tasks] |
Hi all, this is a unfortunately still hard. I discussed it in this discussion: #1812 TL;DR: try if it solves with Warning
The original issue from @slobodaapl is something that will not work. [project]
channels = ["conda-forge", "pytorch"]
name = "pytorch"
platforms = ["linux-64"]
[dependencies]
pytorch = ">=2.3.1,<3"
torchvision = ">=0.18.0,<0.20"
torchaudio = ">=2.3.1,<3"
# Important to tell pixi that `cuda` is available and which version it can expect on all machines the project needs to run on.
[system-requirements]
cuda = "12.1"
[tasks]
cuda-available = "python -c 'import torch; print(torch.cuda.is_available())'"
---
|
Thank you for the context, this is helpful. One last question I have is, once I've turned off channel-priority Also there's some confusion on my end for the system-requirements. Does setting cuda ="12.1" mean exactly that version of cuda, or 12.1 and above? Lastly does getting cuda from the nvidia channel like this also satisfy the requirement? cuda = { version = "*", channel = "nvidia/label/cuda-12.1.0" } |
Hi @pablovela5620, TL;DR: System-requirements define minimal requirements. The cuda dependencies from the labeled channel will indeed get you |
No idea if this helps, but an FYI, the solution posted here doesn't seem to work for me and I still end up with a cpu pytorch :
and
|
This should be fixed once #2025 is merged. TDLR: Explanation: With the changes to the solver, the solver will not first decide on the transitive dependency for Before the above PR:
With the above PR:
Btw: You dont need the |
Are you also saying that maybe another version of |
Im not entirely sure which one should be prioritized over others. But I think it being a transitive dependency should not restrict the version we select for a direct dependency. |
Okay! With #2025 now merged this yields: [project]
authors = ["Tim de Jager <tim@prefix.dev>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "torch"
platforms = ["linux-64"]
version = "0.1.0"
[tasks]
[system-requirements]
cuda = "12.3"
[dependencies]
pytorch = "*" $ pixi list --platform linux-64 | grep pytorch
pytorch 2.4.0 cuda120_py312h26b3cf7_301 24.6 MiB conda pytorch-2.4.0-cuda120_py312h26b3cf7_301.conda |
Will this transparently also translate to a pixi global install?.. Or rather, are either of these possible: a) Specify all packages in pixi toml to be installed as global Thank you |
@slobodaapl We're not planning to support global installation of libraries like A) The next version of |
That's a valid point, I don't know the mechanics behind pixi enough, I had a misunderstanding of what the global install does. What I'm looking to do, which is something that even Poetry lacks right now without workarounds, is to install global site packages to use as a "base" environment, and which are considered "concrete". The goal is to develop a development environment with some predetermined fixed requirements that should be inherited and not touched, with the specific versions they're at, and then create virtual environments in projects that inherit the global packages, without pointlessly re-downloading different versions of them and reinstalling them in the .venv Is there currently any reasonable way to achieve this? |
The main purpose of That said, it’s doable. The most flexible approach is to create a
Important After setting this up, all Warning If you use P.S. If this approach seems too cumbersome, I’ve proposed another idea that leverages caching, which could solve your issue differently. Check out the discussion here: #1354. |
Hmm is it the point to avoid those workflows though?.. Conda by default uses hardlinks when cloning a 'base' environment to avoid taking up more space and using up more disk, or to avoid taking a long time to set up. The neat thing about Poetry, Pixi, etc., is the streamlined TOML files which allow a lot of flexibility and configuration in one place, plus various other benefits.. but it seems to actually lack that ability, in both cases. However the cache approach you mention on #1354 does actually seem like a very reasonable approach that gives us best of both worlds, saving on disk space, setup times, but also avoiding potentially breaking combination of versions. At any rate, #2025 fixes the issue of installing pytorch, so I'll close this issue. |
I believe that your goal is not to install globally but just to avoid the double download. |
Checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pixi, using
pixi --version
.Reproducible example
Issue description
Attempting to install PyTorch on Python 3.11 results in failure, due to a package not being detected
Expected behavior
Pytorch installs correctly
This is likely related to #1484
The text was updated successfully, but these errors were encountered: