python3Packages.{torch,torch-bin}: 2.1.1 -> 2.2.0 #285249

GaetanLepage · 2024-01-31T10:39:20Z

Description of changes

Update the torch ecosystem:

torch[-bin]: 2.1.1 -> 2.2.0 https://github.com/pytorch/pytorch/releases/tag/v2.2.0
torchaudio[-bin]: 2.1.1 -> 2.2.0 https://github.com/pytorch/audio/releases/tag/v2.2.0
torchvision[-bin]: 0.16 -> 0.17 https://github.com/pytorch/vision/releases/tag/v0.17.0

cc @teh @thoughtpolice @tscholak

Things done

Add a 👍 reaction to pull requests you find important.

pkgs/development/python-modules/torchaudio/default.nix

GaetanLepage · 2024-01-31T12:15:31Z

It fails with

* Building wheel...
Building wheel torch-2.2.0
-- Building version 2.2.0
cmake --build . --target install --config Release -- -j 48
[3/4] Generating ATen sourcesns_yaml
[545/6025] Generating src/x86_64-fma/2d-fourier-8x8.py.omicrokernels-all.dir/src/f32-vbinary/gen/f32-vmaxc-scalar-x2.c.o.otor.cc.oar-rr2-p5-x4.c.o.oK.o
FAILED: confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o /build/source/build/confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o 
cd /build/source/build/confu-deps/NNPACK && PYTHONPATH=/build/source/build/confu-srcs/six:/build/source/third_party/python-peachpy /nix/store/w4fvvhkzb0ssv0fw5j34pw09f0qw84w8-python>
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/build/source/third_party/python-peachpy/peachpy/__init__.py", line 39, in <module>
    from peachpy.literal import Constant
  File "/build/source/third_party/python-peachpy/peachpy/literal.py", line 1, in <module>
    import six
ModuleNotFoundError: No module named 'six'
[592/6025] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/descriptor.cc.oator.cc.o[Kc.oc.o
ninja: build stopped: subcommand failed.

Although, six is indeed in the propagatedBuildInputs.

junjihashimoto · 2024-02-01T08:24:56Z

Thx! As you may have noticed, torchvision, torchvision-bin, and torchaudio-bin also need to be updated.

GaetanLepage · 2024-02-01T10:13:09Z

Thx! As you may have noticed, torchvision, torchvision-bin, and torchaudio-bin also need to be updated.

Absolutely ! But for this, I first need to fix torch and torch-bin which both fails for weird reasons...

daniel-fahey · 2024-02-08T09:12:50Z

@GaetanLepage I got torch to build without NNPACK, see GaetanLepage#1 commit 186bbe1 I also tried adding six to the nativeBuildInputs (didn't help)

Same problem happened for @jkachmar #283343

GaetanLepage · 2024-02-08T10:44:40Z

Thanks to @daniel-fahey, all the source packages build fine.

Now the issue with torch-bin remains.

GaetanLepage · 2024-02-08T10:46:17Z

Now the issue with torch-bin remains.

auto-patchelf: 20 dependencies could not be satisfied
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_python.so
error: auto-patchelf could not satisfy dependency libcudnn.so.8 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_python.so
warn: auto-patchelf ignoring missing libcuda.so.1 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libcaffe2_nvrtc.so
error: auto-patchelf could not satisfy dependency libnvrtc.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libcaffe2_nvrtc.so
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcusparse.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcufft.so.11 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcurand.so.10 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcublas.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcublasLt.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcudnn.so.8 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libnccl.so.2 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
error: auto-patchelf could not satisfy dependency libcusolver.so.11 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
error: auto-patchelf could not satisfy dependency libcublas.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
error: auto-patchelf could not satisfy dependency libcusparse.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
error: auto-patchelf could not satisfy dependency libcupti.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libc10_cuda.so
error: auto-patchelf could not satisfy dependency libcudart.so.12 wanted by /nix/store/gsk74ydbfm25vda3wr27n513p3xck7cs-python3.11-torch-2.2.0/lib/python3.11/site-packages/torch/lib/libtorch_global_deps.so
auto-patchelf failed to find all the required dependencies.

pkgs/development/python-modules/torch/bin.nix

pkgs/development/python-modules/torch/default.nix

pkgs/development/python-modules/torch/bin.nix

daniel-fahey · 2024-02-09T17:30:46Z

I'm getting a new failure when building torch with

config = {
  allowUnfree = true;
  cudaSupport = true;
};

Looks like the patch you made @GaetanLepage doesn't work any more:

$ nix log /nix/store/3h86msbim7ziklba01gwf1gn5aizw71s-python3.11-torch-2.2.0.drv^*
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pypa-build-hook
Using pypaBuildPhase
Sourcing python-runtime-deps-check-hook
Using pythonRuntimeDepsCheckHook
Sourcing pypa-install-hook
Using pypaInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing auto-add-opengl-runpath-hook
Using autoAddOpenGLRunpathPhase
Sourcing setup-cuda-hook
@nix { "action": "setPhase", "phase": "unpackPhase" }
Running phase: unpackPhase
unpacking source archive /nix/store/chpsb0nf5ik31gnrhyiy5g0dsh3ydjcm-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/version.txt
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: patchPhase
applying patch /nix/store/mv20hpf1x27afyzn4hlfz4bbx4dna8ln-fix-cmake-cuda-toolkit.patch
patching file CMakeLists.txt
Hunk #1 succeeded at 1160 (offset 8 lines).
patching file cmake/public/cuda.cmake
Hunk #1 FAILED at 62.
1 out of 1 hunk FAILED -- saving rejects to file cmake/public/cuda.cmake.rej
patching file tools/setup_helpers/cmake.py

I also tried directly applying on my copy of the pytorch repo:

$ git apply ../nixpkgs/pkgs/development/python-modules/torch/fix-cmake-cuda-toolkit.patch
error: patch failed: cmake/public/cuda.cmake:62
error: cmake/public/cuda.cmake: patch does not apply

I did however get the patch to apply on commit a6b452dfdcb484d5dfdbb577b74cecbd7021df2e:

[daniel@laptop:~/Source/pytorch]$ git checkout a6b452dfdcb484d5dfdbb577b74cecbd7021df2e
Previous HEAD position was 93cea394dee CMake: Loosen CUDA consistency check (#113174)
HEAD is now at a6b452dfdcb [2/N] Enable Wunused-result, Wunused-variable and Wmissing-braces in torch targets (#110836)

[daniel@laptop:~/Source/pytorch]$ git apply ../nixpkgs/pkgs/development/python-modules/torch/fix-cmake-cuda-toolkit.patch
[daniel@laptop:~/Source/pytorch]$ echo $?
0

Have a great weekend anyway I probably won't get back to this until Monday,

Changelog: https://github.com/pytorch/pytorch/releases/tag/v2.2.0 Co-authored-by: Daniel Fahey <daniel.fahey@pm.me>

GaetanLepage · 2024-02-10T11:04:01Z

@daniel-fahey I updated the patch and it now applies fine :)

Changelog: https://github.com/pytorch/pytorch/releases/tag/v2.2.0

Changelog: https://github.com/pytorch/audio/releases/tag/v2.2.0

Changelog: https://github.com/pytorch/vision/releases/tag/v0.17.0

junjihashimoto

Thank you!

SomeoneSerge

Merging tomorrow noon (UTC) unless there are objections

GaetanLepage requested review from ConnorBaker and Madouura January 31, 2024 10:39

github-actions bot added the 6.topic: python label Jan 31, 2024

GaetanLepage requested review from samuela and SomeoneSerge January 31, 2024 10:39

leona-ya requested changes Jan 31, 2024

View reviewed changes

pkgs/development/python-modules/torchaudio/default.nix Outdated Show resolved Hide resolved

GaetanLepage force-pushed the torch branch from 8700d0b to 6476d79 Compare January 31, 2024 12:18

ofborg bot requested review from junjihashimoto, teh, thoughtpolice and tscholak January 31, 2024 12:41

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Jan 31, 2024

samuela mentioned this pull request Jan 31, 2024

openai-triton crashes trying to use ldconfig -p #285307

Open

GaetanLepage force-pushed the torch branch from fa16adb to 9902a83 Compare February 8, 2024 10:12

SomeoneSerge reviewed Feb 8, 2024

View reviewed changes

pkgs/development/python-modules/torch/bin.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 8, 2024

View reviewed changes

pkgs/development/python-modules/torch/bin.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 8, 2024

View reviewed changes

pkgs/development/python-modules/torch/default.nix Show resolved Hide resolved

SomeoneSerge reviewed Feb 8, 2024

View reviewed changes

pkgs/development/python-modules/torch/bin.nix Show resolved Hide resolved

GaetanLepage force-pushed the torch branch from 61b6174 to b35c135 Compare February 8, 2024 15:21

GaetanLepage force-pushed the torch branch from b35c135 to 8db7a81 Compare February 9, 2024 22:31

python311Packages.torch: 2.1.2 -> 2.2.0

6bcf3b8

Changelog: https://github.com/pytorch/pytorch/releases/tag/v2.2.0 Co-authored-by: Daniel Fahey <daniel.fahey@pm.me>

GaetanLepage force-pushed the torch branch from 8db7a81 to c214a12 Compare February 10, 2024 11:03

GaetanLepage added 5 commits February 11, 2024 00:27

python311Packages.torch-bin: 2.1.2 -> 2.2.0

c62f7c2

Changelog: https://github.com/pytorch/pytorch/releases/tag/v2.2.0

python311Packages.torchaudio: 2.1.2 -> 2.2.0

1c0b570

Changelog: https://github.com/pytorch/audio/releases/tag/v2.2.0

python311Packages.torchaudio-bin: 2.1.2 -> 2.2.0

283343b

Changelog: https://github.com/pytorch/audio/releases/tag/v2.2.0

python311Packages.torchvision: 0.16.2 -> 0.17.0

ea27754

Changelog: https://github.com/pytorch/vision/releases/tag/v0.17.0

python311Packages.torchvision-bin: 0.16.2 -> 0.17.0

fac77e6

Changelog: https://github.com/pytorch/vision/releases/tag/v0.17.0

GaetanLepage force-pushed the torch branch from c214a12 to fac77e6 Compare February 10, 2024 23:28

junjihashimoto approved these changes Feb 11, 2024

View reviewed changes

SomeoneSerge mentioned this pull request Feb 11, 2024

python3Packages.torch: package NNPACK separately #288140

Open

SomeoneSerge approved these changes Feb 11, 2024

View reviewed changes

SomeoneSerge merged commit db19b28 into NixOS:master Feb 12, 2024
25 checks passed

GaetanLepage deleted the torch branch February 12, 2024 15:00

fabaff mentioned this pull request Feb 14, 2024

python312Packages.catalogue: 2.0.8 -> 2.0.10 #288777

Merged

13 tasks

CertainLach mentioned this pull request Feb 17, 2024

vllm: init at v0.3.1 with rocm support #283842

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3Packages.{torch,torch-bin}: 2.1.1 -> 2.2.0 #285249

python3Packages.{torch,torch-bin}: 2.1.1 -> 2.2.0 #285249

GaetanLepage commented Jan 31, 2024 •

edited

Loading

GaetanLepage commented Jan 31, 2024

junjihashimoto commented Feb 1, 2024

GaetanLepage commented Feb 1, 2024

daniel-fahey commented Feb 8, 2024 •

edited

Loading

GaetanLepage commented Feb 8, 2024

GaetanLepage commented Feb 8, 2024

daniel-fahey commented Feb 9, 2024 •

edited

Loading

GaetanLepage commented Feb 10, 2024

junjihashimoto left a comment

SomeoneSerge left a comment

python3Packages.{torch,torch-bin}: 2.1.1 -> 2.2.0 #285249

python3Packages.{torch,torch-bin}: 2.1.1 -> 2.2.0 #285249

Conversation

GaetanLepage commented Jan 31, 2024 • edited Loading

Description of changes

Things done

GaetanLepage commented Jan 31, 2024

junjihashimoto commented Feb 1, 2024

GaetanLepage commented Feb 1, 2024

daniel-fahey commented Feb 8, 2024 • edited Loading

GaetanLepage commented Feb 8, 2024

GaetanLepage commented Feb 8, 2024

daniel-fahey commented Feb 9, 2024 • edited Loading

GaetanLepage commented Feb 10, 2024

junjihashimoto left a comment

Choose a reason for hiding this comment

SomeoneSerge left a comment

Choose a reason for hiding this comment

GaetanLepage commented Jan 31, 2024 •

edited

Loading

daniel-fahey commented Feb 8, 2024 •

edited

Loading

daniel-fahey commented Feb 9, 2024 •

edited

Loading