CI flakiness when using shared Linux runners across PyTorch org

[huydhn]

🐛 Describe the bug

With the recent change to share runners across PyTorch org in the context of project Nova, we are seeing a new group of infra flakiness issues in Linux CI in which jobs fail if they are run on runners with unexpected changes from domain libraries. The underlying problem is that, unlike PyTorch, some domain libraries don't use Docker for containerization. This invalidates our assumption about the runner immutability.

Here is a concrete example:

1. torchrec uses linux.4xlarge.nvidia.gpu runner and has its own way to install NVIDIA driver and CUDA (11.3) from RHEL yum repo
2. By chance, if the same runner picks up a PyTorch CUDA test job, i.e. https://github.com/pytorch/pytorch/actions/runs/3099236524/jobs/5018444060, the job would fail due to a conflict NVIDIA driver installation (our NVIDIA installation on S3 script vs RHEL repo)

We can fix such issues whenever they appear. But it's hard to know where it comes from. Some domain libraries, i.e. torchrec, haven't been integrated with PyTorch bot yet, so there is no information about them on rockset. It's arguably better if we could figure out a way to address the root cause here. Here are some thoughts:

• Make the Linux runner immutable or mark one used by non-PyTorch repo as tainted somehow so that it can be recreated
• Nudge everyone to start using Docker in their CI

Versions

Linux CI

cc @seemethere @malfet @ZainRizvi @pytorch/pytorch-dev-infra

[huydhn]

I suspect that this permission issue when downloading GHA artifacts https://github.com/pytorch/pytorch/actions/runs/3139172100/jobs/5099775201 is yet another example of this

[malfet]

Perhaps one can configure runners to run some sort of a clean up script before picking a new job?

[huydhn]

Here is a more recent example of this issue https://github.com/pytorch/pytorch/actions/runs/3163489795/jobs/5151202666. The installed NVIDIA driver was 510.60.02 instead of the expected 515.57 from S3. The former was probably too old for CUDA 11.7

++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3163489795 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.7, please update your driver to a newer version, or use an earlier cuda container: unknown.
+ container_name=16b28ce9776a4cb57a84b9ed454d9101aa4f819336b6269bc0f13554aa66aa9e

[malfet]

It could be caused by #85435, that would not reinstall the driver if an older one already present (which would be installed if the same host is used for cuda-11.6 and 11.7 testing..
(Though there should be forward CUDA driver compatibility in place, shouldn't it?)

[huydhn]

Yeah, the problem is that reinstalling the driver would need to be handled in a more complicated way because 510.60.02 comes from RHEL yum repo. So just trying to run the installation script again would fails. I'll need to work on a fix for that, i.e. running yum remove nvidia-drivers-smth first

[huydhn]

This can now be resolved as there is a the new GitHub action https://github.com/pytorch/test-infra/tree/main/.github/actions/setup-nvidia to standardize NVIDIA installation across Nova-support PyTorch repos