Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installing PyTorch from default conda command #1544

Closed
2 tasks done
slobodaapl opened this issue Jun 26, 2024 · 21 comments
Closed
2 tasks done

Installing PyTorch from default conda command #1544

slobodaapl opened this issue Jun 26, 2024 · 21 comments
Labels
🐞 bug Something isn't working

Comments

@slobodaapl
Copy link

slobodaapl commented Jun 26, 2024

Checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi global install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Issue description

Attempting to install PyTorch on Python 3.11 results in failure, due to a package not being detected

..........
would require
0.478   │    └─ blas * mkl, for which no candidates were found.

Expected behavior

Pytorch installs correctly
This is likely related to #1484

@slobodaapl slobodaapl added the 🐞 bug Something isn't working label Jun 26, 2024
@tibuch
Copy link

tibuch commented Jul 1, 2024

I have the same issue. The following pixi.toml installs only PyTorch CPU:

[project]
name = "pixi-cuda"
version = "0.1.0"
description = "Add a short description here"
authors = ["Tim-Oliver Buchholz"]
channels = ["nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[tasks]
check_cuda = 'python -c "import torch; print(torch.cuda.is_available())"'

[system-requirements]
cuda = "11"

[dependencies]
python = "3.12.*"
pytorch-cuda = ">=11.4,<12"
pytorch = ">=2.3.1,<2.4"

pixi list | grep torch:

libtorch           2.3.1       cpu_mkl_h0bb0d08_100       47.5 MiB   conda  libtorch-2.3.1-cpu_mkl_h0bb0d08_100.conda
pytorch            2.3.1       cpu_mkl_py312h3b258cc_100  23.3 MiB   conda  pytorch-2.3.1-cpu_mkl_py312h3b258cc_100.conda
pytorch-cuda       11.8        h7e8668a_5                 3.5 KiB    conda  pytorch-cuda-11.8-h7e8668a_5.tar.bz2

I can work around it by installing PyTorch via PIP:

[project]
name = "pixi-cuda"
version = "0.1.0"
description = "Add a short description here"
authors = ["Tim-Oliver Buchholz"]
channels = ["nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[tasks]
check_cuda = 'python -c "import torch; print(torch.cuda.is_available())"'

[system-requirements]
cuda = "11"

[dependencies]
python = "3.12.*"

[pypi-dependencies]
torch = "*"

[pypi-options]
extra-index-urls = ["https://download.pytorch.org/whl/cu118"]

pixi list | grep torch:

torch                     2.3.1+cu118                      1.6 GiB    pypi   torch-2.3.1+cu118-cp312-cp312-linux_x86_64.http.whl

@tibuch
Copy link

tibuch commented Jul 1, 2024

Okay, the workaround stops working as soon as I add any other dependency like pytorch-lightning:

pixi.toml:

[project]
name = "pixi-cuda"
version = "0.1.0"
description = "Add a short description here"
authors = ["Tim-Oliver Buchholz"]
channels = ["nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[tasks]
check_cuda = 'python -c "import torch; print(torch.cuda.is_available())"'

[system-requirements]
cuda = "11"

[dependencies]
python = "3.12.*"
pytorch-lightning = ">=2.3.0,<2.4"

[pypi-dependencies]
torch = "*"

[pypi-options]
extra-index-urls = ["https://download.pytorch.org/whl/cu118"]

pixi list | grep torch:

libtorch             2.3.1         cpu_mkl_h0bb0d08_100       47.5 MiB   conda  libtorch-2.3.1-cpu_mkl_h0bb0d08_100.conda
pytorch              2.3.1         cpu_mkl_py312h3b258cc_100  23.3 MiB   conda  pytorch-2.3.1-cpu_mkl_py312h3b258cc_100.conda
pytorch-lightning    2.3.0         pyhd8ed1ab_0               432.1 KiB  conda  pytorch-lightning-2.3.0-pyhd8ed1ab_0.conda
torchmetrics         1.4.0.post0   pyhd8ed1ab_0               341.9 KiB  conda  torchmetrics-1.4.0.post0-pyhd8ed1ab_0.conda

@tibuch
Copy link

tibuch commented Jul 1, 2024

And another update. Looks like this is related to the channel order. If I set channels = ["nvidia", "pytorch", "conda-forge"] it works.


[project]
name = "pixi-cuda"
version = "0.1.0"
description = "Add a short description here"
authors = ["Tim-Oliver Buchholz"]
channels = ["nvidia", "pytorch", "conda-forge"]
platforms = ["linux-64"]

[tasks]
check_cuda = 'python -c "import torch; print(torch.cuda.is_available())"'

[system-requirements]
cuda = "11"

[dependencies]
python = "3.12.*"
pytorch-lightning = ">=2.3.0,<2.4"
pytorch = ">=2.3.1,<2.4"
pytorch-cuda = "11.8.*"

pixi list | grep torch:

pytorch              2.3.1         py3.12_cuda11.8_cudnn8.7.0_0  1.6 GiB    conda  pytorch-2.3.1-py3.12_cuda11.8_cudnn8.7.0_0.tar.bz2
pytorch-cuda         11.8          h7e8668a_5                    3.5 KiB    conda  pytorch-cuda-11.8-h7e8668a_5.tar.bz2
pytorch-lightning    2.3.0         pyhd8ed1ab_0                  432.1 KiB  conda  pytorch-lightning-2.3.0-pyhd8ed1ab_0.conda
pytorch-mutex        1.0           cuda                          2.8 KiB    conda  pytorch-mutex-1.0-cuda.tar.bz2
torchmetrics         1.4.0.post0   pyhd8ed1ab_0                  341.9 KiB  conda  torchmetrics-1.4.0.post0-pyhd8ed1ab_0.conda

@tibuch
Copy link

tibuch commented Jul 1, 2024

This gets even more complicated if you want to add opencv to the mix. With the above channel order (["nvidia", "pytorch", "conda-forge"]) we get:

pixi run check_cuda
  ⠖ cuda:linux-64        [00:00:01] resolving conda                                                                                                                                                                  × failed to solve the conda requirements of 'default' 'linux-64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      └─ opencv >=4.10.0,<4.11 cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py312h131ea51_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         │           └─ ffmpeg 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 |
      6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 | 6.1.1 is excluded because due to strict channel priority not using this option from: 'https://conda.anaconda.org/conda-forge/'
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py311h5a6cdeb_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py310hd1140c8_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         │           └─ ffmpeg 7.0.1 | 7.0.1 | 7.0.1 | 7.0.1 | 7.0.1 | 7.0.1 | 7.0.1 | 7.0.1 is excluded because due to strict channel priority not using this option from: 'https://conda.anaconda.org/conda-
      forge/'
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py39hb60e6cd_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py38heac1b63_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py39ha1d3367_601, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py312h21e1ac6_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py311h5fdf532_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py310hbbd9133_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py39h0ffffb9_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py38hb11e753_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt6_py39h33f55d2_600, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py312h23c775b_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py311h27bda42_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py310h1a23324_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py39h6d5955e_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py38h6e305d6_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py39h75a1c2e_501, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py312hdcfb50f_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py311h4b374c5_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py310hee144ac_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py39h678e54b_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py38hbe4c39b_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 qt5_py39hf0b7c19_500, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py312hc5075ff_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py311h5bd7e75_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py310h3d4b477_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py39h2f75a73_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py38h909be62_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py39h4ebaf0d_1, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py312h8908fee_0, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py311hf5cb60f_0, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py310hef7d0a5_0, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py39h1857587_0, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         ├─ opencv 4.10.0 would require
         │  └─ libopencv ==4.10.0 headless_py38he6231d4_0, which cannot be installed because there are no viable options:
         │     └─ libopencv 4.10.0 would require
         │        └─ ffmpeg >=6.1.1,<7.0a0, which cannot be installed because there are no viable options:
         └─ opencv 4.10.0 would require
            └─ libopencv ==4.10.0 headless_py39h7137bbe_0, which cannot be installed because there are no viable options:
               └─ libopencv 4.10.0 would require
                  └─ ffmpeg >=7.0.1,<8.0a0, which cannot be installed because there are no viable options:

It is possible to resolve this issue by changing the channel order to ["nvidia", "conda-forge", "pytorch"] and explicitly defining channel on the pytorch and pytorch-cuda dependency like this:

[project]
name = "pixi-cuda"
version = "0.1.0"
description = "Add a short description here"
authors = ["Tim-Oliver Buchholz"]
channels = ["nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[feature.cuda.tasks]
check_cuda = 'python -c "import torch; print(torch.cuda.is_available())"'

[dependencies]
python = "3.11.*"
opencv = ">=4.10.0,<4.11"

[feature.cuda]
platforms = ["linux-64"]
system-requirements = {cuda = "11"}

[feature.cuda.dependencies]
pytorch-cuda = {version="11.8.*", channel="pytorch"}
pytorch-lightning = ">=2.3.0,<2.4"
pytorch = {version = ">=2.3.1,<2.4", channel="pytorch"}

[environments]
cuda = ["cuda"]

@roaldarbol
Copy link

roaldarbol commented Jul 2, 2024

I think there is a more fundamental issue hiding here: If an installation fails due to the strict channel order, there might still be a way. I think that should be made more explicit in the failed solving. Either by automatically attempting an alternative channel order - maybe as a dry-run, and suggest the possible channel order (if any). Or maybe at the least suggest trying an alternative channel order if pixi.toml contains multiple channels? Maybe worth an issue on its own though...

EDIT: Found out that is already an active issue: #1254

@pablovela5620
Copy link
Contributor

I also came across this issue, like @roaldarbol stated the error message is non obvious that its a channel ordering issue. By moving pytorch to the bottom of the list of channel orderings I managed to get py-opencv installed, but this error message made me think there was some other incompatibility

default) pablo@pablo-ubuntu:~/0Dev/personal/sam2-dust3r$ pixi add py-opencv
 WARN Defined custom mapping channel https://conda.anaconda.org/pytorch/ is missing from project channels
ERROR error=repodata not found
  × failed to solve the conda requirements of 'default' 'linux-64'
  ╰─▶ Cannot solve the request because of: py-opencv * cannot be installed because there are no viable options:
      └─ py-opencv 4.2.0 | 4.2.0 | 4.5.0 | 4.5.0 | 4.5.0 would require
         └─ libopencv ==4.2.0 py36_5, for which no candidates were found.
      The following packages are incompatible
      ├─ rerun-sdk >=0.17.0,<0.18 can be installed with any of the following options:
      │  ├─ rerun-sdk 0.17.0 would require
      │  │  └─ python_abi 3.10.* *_cp310, which can be installed with any of the following options:
      │  │     └─ python_abi 3.10
      │  ├─ rerun-sdk 0.17.0 would require
      │  │  └─ python_abi 3.9.* *_cp39, which can be installed with any of the following options:
      │  │     └─ python_abi 3.9 | 3.9
      │  └─ rerun-sdk 0.17.0 would require
      │     └─ python_abi 3.8.* *_cp38, which can be installed with any of the following options:
      │        └─ python_abi 3.8
      ├─ py-opencv * can be installed with any of the following options:
      │  ├─ py-opencv 4.2.0 | 4.3.0 | 4.4.0 would require
      │  │  ├─ python_abi 3.6.* *_cp36m, which can be installed with any of the following options:
      │  │  │  └─ python_abi 3.6
      │  │  └─ python >=3.6,<3.7.0a0, which can be installed with any of the following options:
      │  │     └─ python 3.6.9 | 3.6.9 | 3.6.9 | 3.6.12 | 3.6.12
      │  ├─ py-opencv 4.2.0 | 4.3.0 | 4.4.0 would require
      │  │  ├─ python_abi ==3.6 *_pypy36_pp73, which can be installed with any of the following options:
      │  │  │  └─ python_abi 3.6 | 3.6
      │  │  └─ python >=3.6,<3.7.0a0, which can be installed with any of the following options:
      │  ├─ py-opencv 4.2.0 | 4.2.0 | 4.3.0 | 4.4.0 would require
      │  │  ├─ python_abi 3.8.* *_cp38, which can be installed with any of the following options:
      │  │  │  └─ python_abi 3.8
      │  │  └─ python >=3.8,<3.9.0a0, which can be installed with any of the following options:
      │  │     └─ python 3.8.12 | 3.8.13 | 3.8.16
      │  └─ py-opencv 4.2.0 | 4.2.0 | 4.3.0 | 4.4.0 would require
      │     ├─ python_abi 3.7.* *_cp37m, which can be installed with any of the following options:
      │     │  └─ python_abi 3.7 | 3.7 | 3.7
      │     └─ python >=3.7,<3.8.0a0, which can be installed with any of the following options:
      │        └─ python 3.7.9 | 3.7.9 | 3.7.10 | 3.7.10 | 3.7.12
      └─ python 3.11.* cannot be installed because there are no viable options:
         └─ python 3.11.0 | 3.11.0 | 3.11.0 | 3.11.0 | 3.11.1 | 3.11.2 | 3.11.3 | 3.11.4 | 3.11.5 | 3.11.6 | 3.11.7 | 3.11.7 | 3.11.8 | 3.11.9
      would constrain
            └─ python_abi 3.11.* *_cp311 , which conflicts with any installable versions previously reported

and all that was required was going from this

channels = [
    "https://prefix.dev/ai-demos",
    "nvidia",
    "nvidia/label/cuda-12.1.0",
    "pytorch",
    "conda-forge",
]

to this

channels = [
    "https://prefix.dev/ai-demos",
    "nvidia",
    "nvidia/label/cuda-12.1.0",
    "conda-forge",
    "pytorch",
]

The error message makes it difficult to tell exactly why its failing

@pablovela5620
Copy link
Contributor

another example I have is with the mmcv package, that installs perfectly fine if done through pypi but gives me a bunch of errors with conda-forge

(default) pablo@pablo-ubuntu:~/0Dev/personal/sam2-dust3r$ pixi add mmcv
 WARN Defined custom mapping channel https://conda.anaconda.org/pytorch/ is missing from project channels
ERROR error=repodata not found
  × failed to solve the conda requirements of 'default' 'linux-64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      ├─ pytorch ==2.3.1 can be installed with any of the following options:
      │  ├─ pytorch 2.3.1 | 2.3.1 | 2.3.1
      │  ├─ pytorch 2.3.1 | 2.3.1 would require
      │  │  └─ python >=3.9,<3.10.0a0, which can be installed with any of the following options:
      │  │     └─ python 3.9.10 | 3.9.12 | 3.9.16 | 3.9.17 | 3.9.18 | 3.9.18 would require
      │  │        └─ python_abi ==3.9 *_pypy39_pp73, which can be installed with any of the following options:
      │  │           └─ python_abi 3.9 | 3.9 | 3.9
      │  └─ pytorch 2.3.1 | 2.3.1 would require
      │     └─ python >=3.8,<3.9.0a0, which can be installed with any of the following options:
      │        └─ python 3.8.12 | 3.8.13 | 3.8.16 would require
      │           └─ python_abi ==3.8 *_pypy38_pp73, which can be installed with any of the following options:
      │              └─ python_abi 3.8 | 3.8 | 3.8
      ├─ pytorch-cuda 12.1.* can be installed with any of the following options:
      │  └─ pytorch-cuda 12.1 | 12.1 would require
      │     └─ libcublas >=12.1.0.26,<12.1.3.1, which can be installed with any of the following options:
      │        └─ libcublas 12.1.0.26
      ├─ mmcv * cannot be installed because there are no viable options:
      │  ├─ mmcv 2.2.0 | 2.2.0 would require
      │  │  └─ libcublas >=12.5.2.13,<13.0a0, which cannot be installed because there are no viable options:
      │  │     ├─ libcublas 12.5.2.13 | 12.5.3.2 | 12.6.0.22 is excluded because due to strict channel priority not using this option from:
      'https://conda.anaconda.org/conda-forge/'
      │  │     └─ libcublas 12.5.2.13 | 12.5.3.2 | 12.6.0.22, which conflicts with the versions reported above.
      │  ├─ mmcv 2.2.0 would require
      │  │  └─ libcublas >=12.4.5.8,<13.0a0, which cannot be installed because there are no viable options:
      │  │     ├─ libcublas 12.4.5.8 | 12.4.5.8 | 12.4.5.8 is excluded because due to strict channel priority not using this option from:
      'https://conda.anaconda.org/conda-forge/'
      │  │     └─ libcublas 12.4.5.8, which conflicts with the versions reported above.
      │  ├─ mmcv 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 would require
      │  │  └─ python_abi 3.12.* *_cp312, which cannot be installed because there are no viable options:
      │  │     └─ python_abi 3.12, which conflicts with the versions reported above.
      │  ├─ mmcv 2.1.0 would require
      │  │  └─ libcublas >=12.1.3.1,<13.0a0, which cannot be installed because there are no viable options:
      │  │     ├─ libcublas 12.4.5.8, which conflicts with the versions reported above.
      │  │     ├─ libcublas 12.1.3.1 | 12.2.5.6 | 12.3.4.1 | 12.3.4.1 | 12.4.2.65 is excluded because due to strict channel priority not using
      this option from: 'https://conda.anaconda.org/conda-forge/'
      │  │     └─ libcublas 12.1.3.1 | 12.2.1.16 | 12.2.4.5 | 12.2.5.6 | 12.3.2.9 | 12.3.4.1 | 12.4.2.65, which conflicts with the versions
      reported above.
      │  ├─ mmcv 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0
      | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 would require
      │  │  └─ cudatoolkit >=11.8,<12, which cannot be installed because there are no viable options:
      │  │     └─ cudatoolkit 11.8.0 | 11.8.0 | 11.8.0 | 11.8.0 is excluded because due to strict channel priority not using this option from:
      'https://conda.anaconda.org/conda-forge/'
      │  ├─ mmcv 2.0.0 | 2.0.0 | 2.0.1 | 2.0.1 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.2.0 | 2.2.0 | 2.2.0 would
      constrain
      │  │  └─ pytorch * cpu* , which conflicts with any installable versions previously reported
      │  ├─ mmcv 2.0.0 | 2.0.0 | 2.0.0 | 2.0.0 | 2.0.1 | 2.0.1 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.2.0
      | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 would require
      │  │  └─ python_abi 3.10.* *_cp310, which cannot be installed because there are no viable options:
      │  │     ├─ python_abi 3.10 | 3.10, which conflicts with the versions reported above.
      │  │     └─ python_abi 3.10, which conflicts with the versions reported above.
      │  ├─ mmcv 2.0.0 | 2.0.0 | 2.0.0 | 2.0.0 | 2.0.1 | 2.0.1 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.2.0
      | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 | 2.2.0 would require
      │  │  └─ python_abi 3.9.* *_cp39, which cannot be installed because there are no viable options:
      │  │     ├─ python_abi 3.9 | 3.9, which conflicts with the versions reported above.
      │  │     ├─ python_abi 3.9, which conflicts with the versions reported above.
      │  │     └─ python_abi 3.9 would constrain
      │  │        └─ python_abi * *_cp39 , which conflicts with any installable versions previously reported
      │  └─ mmcv 2.0.0 | 2.0.0 | 2.0.0 | 2.0.0 | 2.0.1 | 2.0.1 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.1.0 | 2.2.0
      | 2.2.0 | 2.2.0 | 2.2.0 would require
      │     └─ python_abi 3.8.* *_cp38, which cannot be installed because there are no viable options:
      │        ├─ python_abi 3.8 | 3.8, which conflicts with the versions reported above.
      │        ├─ python_abi 3.8, which conflicts with the versions reported above.
      │        └─ python_abi 3.8 | 3.8 would constrain
      │           └─ python_abi * *_cp38 , which conflicts with any installable versions previously reported
      ├─ rerun-sdk >=0.17.0,<0.18 cannot be installed because there are no viable options:
      │  ├─ rerun-sdk 0.17.0 would require
      │  │  └─ python_abi 3.12.* *_cp312, which cannot be installed because there are no viable options:
      │  │     └─ python_abi 3.12, which conflicts with the versions reported above.
      │  ├─ rerun-sdk 0.17.0 | 0.17.0 would require
      │  │  └─ python_abi 3.11.* *_cp311, which cannot be installed because there are no viable options:
      │  │     ├─ python_abi 3.11 would constrain
      │  │     │  └─ python 3.11.* *_cpython , which conflicts with any installable versions previously reported
      │  │     ├─ python_abi 3.11 | 3.11, which conflicts with the versions reported above.
      │  │     └─ python_abi 3.11, which conflicts with the versions reported above.
      │  ├─ rerun-sdk 0.17.0 would require
      │  │  └─ python_abi 3.10.* *_cp310, which cannot be installed because there are no viable options:
      │  │     └─ python_abi 3.10, which conflicts with the versions reported above.
      │  ├─ rerun-sdk 0.17.0 would require
      │  │  └─ python_abi 3.9.* *_cp39, which cannot be installed because there are no viable options:
      │  │     ├─ python_abi 3.9, which conflicts with the versions reported above.
      │  │     └─ python_abi 3.9 would constrain
      │  │        └─ python_abi * *_cp39 , which conflicts with any installable versions previously reported
      │  └─ rerun-sdk 0.17.0 would require
      │     └─ python_abi 3.8.* *_cp38, which cannot be installed because there are no viable options:
      │        ├─ python_abi 3.8, which conflicts with the versions reported above.
      └─ python 3.11.* cannot be installed because there are no viable options:
         └─ python 3.11.0 | 3.11.0 | 3.11.0 | 3.11.0 | 3.11.1 | 3.11.2 | 3.11.3 | 3.11.4 | 3.11.5 | 3.11.6 | 3.11.7 | 3.11.7 | 3.11.8 | 3.11.9
      | 3.11.9 would constrain
            └─ python_abi 3.11.* *_cp311 , which conflicts with any installable versions previously reported

https://prefix.dev/channels/conda-forge/packages/mmcv

for this pixi.toml

[project]
authors = [{ name = "pablovela5620", email = "pablovela5620@gmail.com" }]
dependencies = []
description = "Add a short description here"
name = "sam2-dust3r"
requires-python = ">= 3.11"
version = "0.1.0"

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]

[tool.pixi.project]
channels = [
    "https://prefix.dev/ai-demos",
    "nvidia",
    "nvidia/label/cuda-12.1.0",
    "conda-forge",
    "pytorch",
]
conda-pypi-map = { "pytorch" = "torch_mapping.json" }
platforms = ["linux-64"]

[tool.pixi.system-requirements]
cuda = "12"
libc = { family = "glibc", version = "2.31" }


[tool.pixi.pypi-dependencies]
sam-2 = { git = "https://github.com/facebookresearch/segment-anything-2.git", rev = "0db838b11726893f151fa5826ecfa744e1a7760f", extras = [
    "demo",
] }
sam2-dust3r = { path = ".", editable = true }

[tool.pixi.dependencies]
python = "3.11.*"
pytorch = { version = "2.3.1", channel = "pytorch" }
torchvision = { version = "0.18.1", channel = "pytorch" }
cuda = { version = "*", channel = "nvidia/label/cuda-12.1.0" }
pytorch-cuda = { version = "12.1.*", channel = "pytorch" }
huggingface_hub = ">=0.24.5,<0.25"
rerun-sdk = ">=0.17.0,<0.18"
simplecv = { version = ">=0.0.2,<0.0.3", channel = "https://prefix.dev/ai-demos" }

[tool.pixi.tasks]

@ruben-arts
Copy link
Contributor

Hi all, this is a unfortunately still hard. I discussed it in this discussion: #1812

TL;DR: try if it solves with channel-priority = "disabled" in the [project] table. If that works please turn it back of and setup your dependencies to come from the right channel as was solved by pixi.

Warning

channel-priority = "disabled" can be very dangerous as there is no guarantee that the same package is getting installed from the same channel every time. This is a possible security risk as someone can name drop to overwrite a package with something you didn't expect.


The original issue from @slobodaapl is something that will not work. pixi global can only install tools at the moment of writing this. So not libraries like you requested. Making a project should work for this e.g.:

[project]
channels = ["conda-forge", "pytorch"]
name = "pytorch"
platforms = ["linux-64"]

[dependencies]
pytorch = ">=2.3.1,<3"
torchvision = ">=0.18.0,<0.20"
torchaudio = ">=2.3.1,<3"

# Important to tell pixi that `cuda` is available and which version it can expect on all machines the project needs to run on.
[system-requirements]
cuda = "12.1"

[tasks]
cuda-available = "python -c 'import torch; print(torch.cuda.is_available())'"
---


@pablovela5620
Copy link
Contributor

Thank you for the context, this is helpful. One last question I have is, once I've turned off channel-priority channel-priority = "disabled" and I have things working. Is the idea that I would basically look for the packages I have defined and then specify each channel for each package? I think that's where I'm still a bit confused.

Also there's some confusion on my end for the system-requirements. Does setting cuda ="12.1" mean exactly that version of cuda, or 12.1 and above? Lastly does getting cuda from the nvidia channel like this also satisfy the requirement?

cuda = { version = "*", channel = "nvidia/label/cuda-12.1.0" }

@ruben-arts
Copy link
Contributor

Hi @pablovela5620,
Disabling channel-priority can really easily result in future breaking setups when you for instance run pixi update as these channels can "do what they want". So best practice would be to check if it "can" work with the priority disabled but then use that result to manually set it up in your toml to make sure it would not break in the future.

TL;DR: System-requirements define minimal requirements.
In theory it's the description of the machine you solve for. This means you tell pixi. Solve as if you're running on a machine with cuda = "12.1" but because (i believe) all virtual package dependencies are written as cuda >= 12.1 this would result in the minimal requirement.

The cuda dependencies from the labeled channel will indeed get you cuda-12.1.0 because it can only fetch from this channel: https://anaconda.org/nvidia/cuda/files?channel=cuda-12.1.0.

@timlacroix
Copy link

No idea if this helps, but an FYI, the solution posted here doesn't seem to work for me and I still end up with a cpu pytorch :

[project]
channels = ["conda-forge", "pytorch"]
platforms = ["linux-64"]
name = "test"

[system-requirements]
cuda = "12.3"

[dependencies]
pytorch = ">=2.3.1,<3"
python = "3.10.*"

and

pixi list | grep pytorch
pytorch            2.4.0       cpu_generic_py310h6ad04bf_1  30.6 MiB   conda  pytorch-2.4.0-cpu_generic_py310h6ad04bf_1.conda

@baszalmstra
Copy link
Contributor

baszalmstra commented Sep 10, 2024

This should be fixed once #2025 is merged.

TDLR:
With that change, the solver will always prioritize maximizing the versions of explicit requirements over transitive dependencies.

Explanation:
The problem with the reproducer from @timlacroix is that the solver first decides on a version for python and then decides on an openmp version required by Python. It will decide on libgomp, which is incompatible with pytorch CUDA. So once the solver tries to decide on pytorch, the CUDA version is no longer selectable.

With the changes to the solver, the solver will not first decide on the transitive dependency for openmp but it will first decide on a version of pytorch instead. This will then select the best available pytorch version which is the CUDA version. Only then will it decide on a compatible openmp version.

Before the above PR:

> pixi list | grep pytorch
pytorch            2.4.0       cpu_generic_py310h6ad04bf_1  30.6 MiB   conda  pytorch-2.4.0-cpu_generic_py310h6ad04bf_1.conda

With the above PR:

> pixi list | grep pytorch
pytorch               2.4.0       cuda120_py310h2c91c31_301  30.6 MiB   conda  pytorch-2.4.0-cuda120_py310h2c91c31_301.conda

Btw: You dont need the pytorch channel in the example at all!

@tdejager
Copy link
Contributor

Are you also saying that maybe another version of libgomp should be prioritized?

@baszalmstra
Copy link
Contributor

Are you also saying that maybe another version of libgomp should be prioritized?

Im not entirely sure which one should be prioritized over others. But I think it being a transitive dependency should not restrict the version we select for a direct dependency.

@tdejager
Copy link
Contributor

Okay!

With #2025 now merged this yields:

[project]
authors = ["Tim de Jager <tim@prefix.dev>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "torch"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[system-requirements]
cuda = "12.3"


[dependencies]
pytorch = "*"
$ pixi list --platform linux-64 | grep pytorch
                                                                                                                                           
pytorch               2.4.0       cuda120_py312h26b3cf7_301  24.6 MiB   conda  pytorch-2.4.0-cuda120_py312h26b3cf7_301.conda

@slobodaapl
Copy link
Author

Okay!

With #2025 now merged this yields:

[project]
authors = ["Tim de Jager <tim@prefix.dev>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "torch"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[system-requirements]
cuda = "12.3"


[dependencies]
pytorch = "*"
$ pixi list --platform linux-64 | grep pytorch
                                                                                                                                           
pytorch               2.4.0       cuda120_py312h26b3cf7_301  24.6 MiB   conda  pytorch-2.4.0-cuda120_py312h26b3cf7_301.conda

Will this transparently also translate to a pixi global install?.. Or rather, are either of these possible:

a) Specify all packages in pixi toml to be installed as global
b) Use the pixi global install command from CLI with specifying the cuda as system requirement
c) Any other mechanism to globally install torch with cuda using pixi

Thank you

@ruben-arts
Copy link
Contributor

@slobodaapl We're not planning to support global installation of libraries like torch. pixi global creates a virtual environment per "tool" and if you do this for libraries you can easily get conflicting libraries between the different virtual environments.

A) The next version of pixi global will contain the ability to create some more interesting virtual environments that can be activated globally.
B) The pixi global install doesn't need system-requirements as it uses the once from your own system directly, not the default set we define for the projects. This is because you don't need to be able to share the global installation.
C) torch as a library no, but an environment that contains torch and other tools that depend on it would be possible with the next version (answer A).

@slobodaapl
Copy link
Author

@ruben-arts

That's a valid point, I don't know the mechanics behind pixi enough, I had a misunderstanding of what the global install does.

What I'm looking to do, which is something that even Poetry lacks right now without workarounds, is to install global site packages to use as a "base" environment, and which are considered "concrete".

The goal is to develop a development environment with some predetermined fixed requirements that should be inherited and not touched, with the specific versions they're at, and then create virtual environments in projects that inherit the global packages, without pointlessly re-downloading different versions of them and reinstalling them in the .venv

Is there currently any reasonable way to achieve this?

@ruben-arts
Copy link
Contributor

The main purpose of pixi is to help avoid workflows like this 😅. Same applies to tools like uv, poetry, and others in ecosystems like cargo, npm, etc.

That said, it’s doable. The most flexible approach is to create a pixi project and activate it within your shell’s initialization.

  1. Create a pixi project:

    pixi init global-env
    cd global-env
    # Add your system requirements manually
    pixi add pytorch  # Add any other tools/packages needed
  2. Add this line to your shell’s rc file (e.g. .bashrc):

    eval "$(pixi shell-hook --manifest-path /path/to/global-env/pixi.toml)"
  3. Reload your terminal.

Important

After setting this up, all pixi commands will be tied to that project, essentially disabling other workflows.

Warning

If you use pip install after enabling the shell hook, it will install into the pixi project, which will break things.

P.S. If this approach seems too cumbersome, I’ve proposed another idea that leverages caching, which could solve your issue differently. Check out the discussion here: #1354.

@slobodaapl
Copy link
Author

The main purpose of pixi is to help avoid workflows like this 😅. Same applies to tools like uv, poetry, and others in ecosystems like cargo, npm, etc.

That said, it’s doable. The most flexible approach is to create a pixi project and activate it within your shell’s initialization.

  1. Create a pixi project:
    pixi init global-env
    cd global-env
    # Add your system requirements manually
    pixi add pytorch  # Add any other tools/packages needed
  2. Add this line to your shell’s rc file (e.g. .bashrc):
    eval "$(pixi shell-hook --manifest-path /path/to/global-env/pixi.toml)"
  3. Reload your terminal.

Important

After setting this up, all pixi commands will be tied to that project, essentially disabling other workflows.

Warning

If you use pip install after enabling the shell hook, it will install into the pixi project, which will break things.

P.S. If this approach seems too cumbersome, I’ve proposed another idea that leverages caching, which could solve your issue differently. Check out the discussion here: #1354.

Hmm is it the point to avoid those workflows though?.. Conda by default uses hardlinks when cloning a 'base' environment to avoid taking up more space and using up more disk, or to avoid taking a long time to set up.

The neat thing about Poetry, Pixi, etc., is the streamlined TOML files which allow a lot of flexibility and configuration in one place, plus various other benefits.. but it seems to actually lack that ability, in both cases.

However the cache approach you mention on #1354 does actually seem like a very reasonable approach that gives us best of both worlds, saving on disk space, setup times, but also avoiding potentially breaking combination of versions.

At any rate, #2025 fixes the issue of installing pytorch, so I'll close this issue.

@ruben-arts
Copy link
Contributor

Conda by default uses hardlinks when cloning a 'base' environment to avoid taking up more space and using up more disk, or to avoid taking a long time to set up.

pixi does the linking too, even uses a reflink when possible which makes it safe too. But it doesn't reuse the cache yet.

Hmm is it the point to avoid those workflows though?

I believe that your goal is not to install globally but just to avoid the double download.
The workflow I proposed can easily take up more time in debugging and fixing problems when you accidentally do a pip install for instance. To avoid that most modern tools work with dedicated file-system based virtual environments. To avoid mixing all projects and their environments, also known as the kitchen sink.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants