CI Sev - pin docker images for A100 workers

[atalman]

Pinning docker images, trying to address SEV : #108862

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/108871

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• A100 runners down: apt-get install nvidia-docker2, Could not get lock /var/lib/dpkg/lock-frontend

❌ 1 New Failure, 7 Unrelated Failures

As of commit 88c8006 with merge base cdf7f3e ():

NEW FAILURE - The following job has failed:

• linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu) (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-jammy-py3.9-clang12-asan / test (default, 6, 6, linux.4xlarge) (gh)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• lintrunner / linux-job (gh)

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

• linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)
• linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[huydhn]

This looks good, but you probably want to cancel the nightly job and force merge it this case because it takes like 8 hours or something to finish that one.

[ezyang]

It seems like... it didn't work?

[atalman]

It seems like... it didn't work?

I think I found the problem why it did not worked. Manual remediation step is required. Worth giving it a shot. Will merge this now, and run manual remediation.

[atalman]

@pytorchbot merge -f "working on CI sev"

[pytorchmergebot]

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here