PyTorch CI Stability Action Tracker

[janeyx99]

This will be a meta-issue/tracker for directly actionable issues regarding stability and reliability in PyTorch CI.

Motivation

CI stability is crucial to developer velocity and user experience. Otherwise, developers may waste time debugging their CI failures that are irrelevant to their changes and the on-call who is in charge of keeping trunk green may be tasked with more work to triage issues. When trunk is red, PyTorch users who pull from source could be impacted as well.

Immediately Actionable Issues

• Modify Dr. CI so it could detect runner disconnection failures #66902
• [CI] Flaky initialization of OMP #79055
• android-tests is often flaky #79785
• Bazel build is very flaky #78100

Windows Specific

none so far!

Rolled into bigger projects (please comment relevant suggestions in their respective issues)

• Flaky Test Detection + Retry Automatically Detect and Remove Flaky Tests #71005
• Removing Network Connectivity Risks In CI CI: Removing Network Connectivity Risks #71003
• Retryable Steps GHA: Use retryable steps #71563

[suo]

Using pretrained weights from torchvision causes a network call to download weights. Example: https://github.com/pytorch/pytorch/blob/master/test/onnx/test_pytorch_onnx_onnxruntime.py#L354

Found from failure: https://circleci.com/gh/pytorch/pytorch/16139241

A cursory search shows quite a few tests where we are doing this.

Fixing is probably as easy as just switching pretrained to False—I highly doubt any of those tests are sensitive to whether the weights are pretrained are not.

[suo]

Looks like CMake binary is not available?

Example: https://github.com/pytorch/pytorch/runs/3677706168

I'm going to chalk this one up as "probably fixed by ephemeral runners"

[suo]

Downloading nvidia stuffs in test failed:
https://github.com/pytorch/pytorch/runs/3681335684

Would be resolved by baking into the AMI

[suo]

sccache on windows has had network failures 25 times in the last week: https://fburl.com/scuba/opensource_ci_jobs/q3nf3g6n

Googling the error message shows an issue that @ezyang commented on: mozilla/sccache#256. Suggested fixes in that issue are 1) reducing parallelism, and 2) increasing the timeout

[janeyx99]

Problems noted by @mruberry:

Run rm -rf "${PYTORCH_FINAL_PACKAGE_DIR}"
rm: cannot remove './build': Device or resource busy

in https://github.com/pytorch/pytorch/runs/3682204176 and

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

in https://github.com/pytorch/pytorch/runs/3683674360 and

Run docker pull "${DOCKER_IMAGE}"
Error response from daemon: Get "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

in https://github.com/pytorch/pytorch/runs/3682134416 and

Sep 23 02:20:31   test_error_in_init (__main__.TestDataLoader) ... ok (0.164s)

Too long with no output (exceeded 1h30m0s): context deadline exceeded

in https://app.circleci.com/pipelines/github/pytorch/pytorch/384098/workflows/c9f78d92-487c-4a85-b97e-10b222e5c3a9/jobs/16158997 (an ASAN test job).

[suo]

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

This is not a real failure, it's a build failure. We should mark this step as "continue-through-error" or whatever because it's not a user-facing failure.

[driazati]

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

This is not a real failure, it's a build failure. We should mark this step as "continue-through-error" or whatever because it's not a user-facing failure.

Still something we can fix though, we should have guards on those steps instead of if: always(), they should check if the build step succeeded

[janeyx99]

Still something we can fix though, we should have guards on those steps instead of if: always(), they should check if the build step succeeded

I'll take this--I've modified the big summary at the top to include this action item and assigned it to me