[BE]: Update cudnn to 9.1.0.70 #123475

eqy · 2024-04-05T21:31:39Z

cuDNN has managed to upload cu11 and cu12 wheels for ~~9.0.0.312~~ 9.1.0.70, so trying this out...

cc @csarofeen @ptrblck @xwang233 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

pytorch-bot · 2024-04-05T21:31:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/123475

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 27 New Failures, 7 Unrelated Failures

As of commit 8d85d49 with merge base 0de6d24 ():

NEW FAILURES - The following jobs have failed:

Docathon Labels Sync / check-labels (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_10-cuda11_8-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_10-cuda12_1-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_10-cuda12_4-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_11-cuda11_8-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_11-cuda12_1-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_11-cuda12_4-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_12-cuda11_8-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_12-cuda12_1-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_12-cuda12_4-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_9-cuda11_8-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_9-cuda12_1-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_9-cuda12_4-test / test (gh)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
'test/test_quantization.py::TestQuantizedLinear::test_qlinear_cudnn'
windows-binary-conda / conda-py3_10-cuda11_8-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_10-cuda12_1-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_11-cuda12_1-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_11-cuda12_4-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_12-cuda11_8-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_12-cuda12_1-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_12-cuda12_4-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_8-cuda11_8-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_8-cuda12_1-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_8-cuda12_4-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_9-cuda11_8-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_9-cuda12_1-test (gh)
Process completed with exit code 1.
windows-binary-conda / conda-py3_9-cuda12_4-test (gh)
Process completed with exit code 1.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

linux-binary-manywheel / manywheel-py3_8-cuda11_8-test / test (gh) (similar failure)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'test/test_quantization.py::TestQuantizedLinear::test_qlinear_cudnn'
windows-binary-conda / conda-py3_10-cuda12_4-test (gh) (similar failure)
Process completed with exit code 1.
windows-binary-conda / conda-py3_11-cuda11_8-test (gh) (similar failure)
Process completed with exit code 1.

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (inductor_torchbench_cpu_smoketest_perf, 1, 1, linux.24xl.spr-metal, unstable) (gh) (#126993)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_8-cuda12_1-test / test (gh) (#127288)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
linux-binary-manywheel / manywheel-py3_8-cuda12_4-test / test (gh) (#127289)
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Skylion007

Looks good assuming you fix the Caffe2 issue.

eqy · 2024-04-08T18:44:21Z

@Skylion007 do we anticipate #122527 landing in the near-future? caffe2 fixes might be involved and look something like #115719 which might be unnecessary if caffe2 is being removed soon

r-barnes · 2024-04-16T18:04:56Z

@eqy - I'm still working on eliminating caffe2, but having difficulty isolating all the issues - I haven't worked deeply with PyTorch's build before.

johnnynunez · 2024-05-06T13:41:35Z

9.1.1 is out

eqy · 2024-05-10T20:19:27Z

@pytorchmergebot rebase (checking to see if caffe2 issue has been resolved)

pytorch-bot · 2024-05-10T20:19:30Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: (checking to see if caffe2 issue has been resolved)

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try @pytorchbot --help for more info.

eqy · 2024-05-10T20:19:38Z

@pytorchmergebot rebase

pytorchmergebot · 2024-05-10T20:21:03Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-05-10T20:21:08Z

Successfully rebased updatecudnn9 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout updatecudnn9 && git pull --rebase)

Skylion007 · 2024-05-12T18:43:33Z

@eqy Looks like the issues were fixed. Let's merge.

eqy · 2024-05-13T21:06:34Z

@pytorchmergebot merge

pytorchmergebot · 2024-05-13T21:09:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-13T21:09:23Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / workflow-checks / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

eqy · 2024-05-13T23:59:53Z

@pytorchmergebot rebase

Run ./github/regenerate.sh after modifying scripts/generate_binary_build_matrix.py

Testing cudnn v9 changes with pinned docker image and builder PR Arm CUDA will unfortunately still fail due to pinned buggy(for arm only, x86_64 ok) docker image

pytorch/builder

This reverts commit 86172c5.

atalman · 2024-06-06T18:43:35Z

@pytorchmergebot merge -f "lint is green"

pytorchmergebot · 2024-06-06T18:45:07Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

@Skylion007

cuDNN has managed to upload cu11 and cu12 wheels for ~~9.0.0.312~~ 9.1.0.70, so trying this out... CC @Skylion007 @malfet Co-authored-by: Wei Wang <weiwan@nvidia.com> Co-authored-by: atalman <atalman@fb.com> Pull Request resolved: pytorch#123475 Approved by: https://github.com/Skylion007, https://github.com/malfet, https://github.com/nWEIdia, https://github.com/atalman

eqy added module: cudnn Related to torch.backends.cudnn, and CuDNN support open source topic: not user facing topic category labels Apr 5, 2024

eqy requested review from Skylion007 and malfet April 5, 2024 21:31

eqy requested review from a team and jeffdaily as code owners April 5, 2024 21:31

Skylion007 approved these changes Apr 8, 2024

View reviewed changes

eqy mentioned this pull request Apr 23, 2024

Caffe2 usage of cuDNN RNNv6 API blocks upgrade to cuDNN v9+ #124790

Open

eqy changed the title ~~[BE]: Update cudnn to 9.0.0.312~~ [BE]: Update cudnn to 9.1.0.70 May 2, 2024

pytorchmergebot force-pushed the updatecudnn9 branch from 82402a0 to e202627 Compare May 10, 2024 20:21

malfet approved these changes May 10, 2024

View reviewed changes

Skylion007 mentioned this pull request May 12, 2024

[BE]: Update cudnn to 8.9.7.29 #120642

Closed

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 13, 2024

pytorchmergebot added the merging label May 13, 2024

pytorchmergebot removed the merging label May 13, 2024

eqy and others added 21 commits June 6, 2024 09:39

try using CUDNN_VERSION==8 specialized name

af219af

more version fixes

432c8f6

sed

d3ed156

Update CUDA_ARCHES_CUDNN_VERSION

fe671eb

oops remove hardcoded cudnn8 condition

12a2ccf

oops

51f1e42

in case

231752b

Fix test failures caused by downloading wrong cudnn version.

f92cdac

Run ./github/regenerate.sh after modifying scripts/generate_binary_build_matrix.py

Fix errors in #127589 (libtorch build)

84efa34

Temporary change to be removed before merging

454449c

revert image specific change

2a66f12

update expected failures

876c243

undo temp changes

b26a515

fix

9df474c

Adjust cu121 beit accuracy expectation.

50361ec

Testing cudnn v9 changes with pinned docker image and builder PR Arm CUDA will unfortunately still fail due to pinned buggy(for arm only, x86_64 ok) docker image

Conda docker images were generated using a different main commit of

8ddfb07

pytorch/builder

Sorry, Forgot to run ./regenerate.sh

48dc42a

Change cudnn8 -> cudnn9 after cuda 12.4 move

4bca52d

Fix libtorch cxx11 docker image pin

1f5d9e6

Revert "Update test_efficient_conv_bn_eval.py"

f8ca901

This reverts commit 86172c5.

Roll back temp changes and cherrypicks

8d85d49

nWEIdia force-pushed the updatecudnn9 branch from 8817579 to 8d85d49 Compare June 6, 2024 16:52

atalman approved these changes Jun 6, 2024

View reviewed changes

pytorchmergebot added the merging label Jun 6, 2024

pytorchmergebot closed this in de4f8b9 Jun 6, 2024

pytorchmergebot removed the merging label Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BE]: Update cudnn to 9.1.0.70 #123475

[BE]: Update cudnn to 9.1.0.70 #123475

eqy commented Apr 5, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Apr 5, 2024 •

edited

Loading

Skylion007 left a comment

eqy commented Apr 8, 2024 •

edited

Loading

r-barnes commented Apr 16, 2024

johnnynunez commented May 6, 2024

eqy commented May 10, 2024

pytorch-bot bot commented May 10, 2024

eqy commented May 10, 2024

pytorchmergebot commented May 10, 2024

pytorchmergebot commented May 10, 2024

Skylion007 commented May 12, 2024

eqy commented May 13, 2024

pytorchmergebot commented May 13, 2024

pytorchmergebot commented May 13, 2024

eqy commented May 13, 2024

atalman commented Jun 6, 2024

pytorchmergebot commented Jun 6, 2024

[BE]: Update cudnn to 9.1.0.70 #123475

[BE]: Update cudnn to 9.1.0.70 #123475

Conversation

eqy commented Apr 5, 2024 • edited by pytorch-bot bot Loading

pytorch-bot bot commented Apr 5, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/123475

❌ 27 New Failures, 7 Unrelated Failures

Skylion007 left a comment

Choose a reason for hiding this comment

eqy commented Apr 8, 2024 • edited Loading

r-barnes commented Apr 16, 2024

johnnynunez commented May 6, 2024

eqy commented May 10, 2024

pytorch-bot bot commented May 10, 2024

eqy commented May 10, 2024

pytorchmergebot commented May 10, 2024

pytorchmergebot commented May 10, 2024

Skylion007 commented May 12, 2024

eqy commented May 13, 2024

pytorchmergebot commented May 13, 2024

Merge started

pytorchmergebot commented May 13, 2024

Merge failed

eqy commented May 13, 2024

atalman commented Jun 6, 2024

pytorchmergebot commented Jun 6, 2024

Merge started

eqy commented Apr 5, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Apr 5, 2024 •

edited

Loading

eqy commented Apr 8, 2024 •

edited

Loading