[Dr.CI] Handle the closing of disabled test issues #5186

huydhn · 2024-05-08T03:59:10Z

This extends Dr.CI to handle disabled test issues in several cases:

The closing of disabled test issues whereas a test could start failing on PR after its associated disabled issue is resolved.
A disabled test fails on PR where it should be skipped.

In both cases, the failure will be marked as skipped and the reason is provided.

Testing

pytorch/pytorch#125656

❌ 1 New Failure, 3 Unrelated Failures

As of commit 3a1e14cc5b5d44fa1b04561bccf14d832f70eb1d with merge base b6bcd0917310c657fc60bc7b20415f6948524eb8 ():

NEW FAILURE - The following job has failed:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu) (gh)
test_foreach.py::TestForeachCUDA::test_parity__foreach_abs_fastpath_inplace_cuda_complex128

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge) (gh) (disabled by #104012)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge) (gh) (disabled by #104012)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface
trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-stable) (gh) (disabled by #104012)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface

pytorch/pytorch#125634

❌ 1 New Failure, 1 Unrelated Failure

As of commit 28cdf5b4af9803379b6206b8ffbdf214949f111b with merge base 22bcfc25ef3043db37e88ad61ba69c85f98571e1 ():

NEW FAILURE - The following job has failed:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu) (gh) (#98259)
test_type_hints.py::TestTypeHints::test_doc_examples

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh) (disabled by #104012)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface

vercel · 2024-05-08T03:59:14Z

@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

vercel · 2024-05-08T17:47:25Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 9, 2024 2:15am

clee2000

I think the logic is sound, some minor concerns about token usage, and a lot of minor comments regarding readability of the tests

torchci/lib/jobUtils.ts

clee2000 · 2024-05-08T18:00:20Z

torchci/pages/api/drci/drci.ts

+      let prBody = "";
+      let prShas: { sha: string; title: string }[] = [];
+      if (octokit) {
+        const prData = await fetchPR(owner, repo, `${prNumber}`, octokit);


should we be worried about rate limits for this?

Per our chat, this will incur the cost of 2 GitHub queries per PR

I guess I will follow up with an alternative to query from Rockset if this turns out to be an issue. I'm not quite sure if the pull_request Rockset table is always up to date. Maybe there is a legit reason we are using GitHub query to get the PR info instead.

torchci/test/drciBot.test.ts

torchci/test/jobUtils.test.ts

torchci/lib/jobUtils.ts

torchci/pages/api/drci/drci.ts

huydhn · 2024-05-09T00:00:24Z

lol, I just found a perfect example of a recently close disabled issue in pytorch/pytorch#122074. The issue was closed by pytorch/pytorch#125706. Both PR were landed close together

✅ You can merge normally! (4 Unrelated Failures)

As of commit 14343544cbbe4d553f4d5a2084e3f83ee8028e82 with merge base b6bcd0917310c657fc60bc7b20415f6948524eb8 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge) (gh) (disabled by #104012 but the issue was closed recently and a rebase is needed to make it pass)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge) (gh) (disabled by #104012 but the issue was closed recently and a rebase is needed to make it pass)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface
trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral) (gh) (disabled by #104012 but the issue was closed recently and a rebase is needed to make it pass)
test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh) (trunk failure)
test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_mlp

After #5186 Link to hud which then redirects to the issue to prevent the PR getting included in the issue's events like the below ![image](https://github.com/pytorch/test-infra/assets/44682903/89023fe4-f649-42d3-bfd6-be7523055fde) Ran on one of my PRs, worked fine  ## 🔗 Helpful Links ### 🧪 See artifacts and rendered test results at [hud.pytorch.org/pr/125799](https://hud.pytorch.org/pr/125799) * 📄 Preview [Python docs built from this PR](https://docs-preview.pytorch.org/pytorch/pytorch/125799/index.html) * 📄 Preview [C++ docs built from this PR](https://docs-preview.pytorch.org/pytorch/pytorch/125799/cppdocs/index.html) * ❓ Need help or want to give feedback on the CI? Visit the [bot commands wiki](https://github.com/pytorch/pytorch/wiki/Bot-commands) or our [office hours](https://github.com/pytorch/pytorch/wiki/Dev-Infra-Office-Hours) Note: Links to docs will display an error until the docs builds have been completed. ## ❌ 1 New Failure, 5 Pending, 3 Unrelated Failures As of commit 852acd87cd5384ac7c61a93d1e0d894f39a6df13 with merge base 8f27c7f181f05ecac5f1948dc070d7781d3da137 (<img alt="image" width=70 src="https://img.shields.io/date/1715204973?label=&color=FFFFFF&style=flat-square">): <details open><summary>NEW FAILURE - The following job has failed:</summary> * [Lint / lintrunner-noclang / linux-job](https://hud.pytorch.org/pr/pytorch/pytorch/125799#24795916616) ([gh](https://github.com/pytorch/pytorch/actions/runs/9023455827/job/24795916616)) `>>> Lint for tools/testing/upload_artifacts.py:` </details> <details ><summary>FLAKY - The following jobs failed but were likely due to flakiness present on trunk:</summary> * [pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge)](https://hud.pytorch.org/pr/pytorch/pytorch/125799#24795855797) ([gh](https://github.com/pytorch/pytorch/actions/runs/9023455802/job/24795855797)) (disabled by [#104012](https://hud.pytorch.org/pytorch/pytorch/issues/104012) but the issue was closed recently and a rebase is needed to make it pass) `test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface` * [pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge)](https://hud.pytorch.org/pr/pytorch/pytorch/125799#24795855164) ([gh](https://github.com/pytorch/pytorch/actions/runs/9023455802/job/24795855164)) (disabled by [#104012](https://hud.pytorch.org/pytorch/pytorch/issues/104012) but the issue was closed recently and a rebase is needed to make it pass) `test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface` * [pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 3, linux.2xlarge)](https://hud.pytorch.org/pr/pytorch/pytorch/125799#24795856213) ([gh](https://github.com/pytorch/pytorch/actions/runs/9023455802/job/24795856213)) (disabled by [#104012](https://hud.pytorch.org/pytorch/pytorch/issues/104012) but the issue was closed recently and a rebase is needed to make it pass) `test_fx.py::TestFXAPIBackwardCompatibility::test_public_api_surface` </details> This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Dr.CI] Handle the closing of disabled tests

b01974b

huydhn requested review from clee2000 and a team May 8, 2024 03:59

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 8, 2024

Minor tweak

cd9906f

huydhn marked this pull request as ready for review May 8, 2024 17:47

vercel bot deployed to Preview May 8, 2024 17:49 View deployment

Fix platforms regex

1c48f8a

vercel bot deployed to Preview May 8, 2024 18:03 View deployment

Fix platforms regex once more time

a0d3b23

vercel bot deployed to Preview May 8, 2024 18:17 View deployment

clee2000 approved these changes May 8, 2024

View reviewed changes

Address review comments

68752b0

vercel bot deployed to Preview May 9, 2024 01:35 View deployment

Reorder Dr.CI rule a bit

cd6a42f

vercel bot deployed to Preview May 9, 2024 02:06 View deployment

Gate new feature to only pytorch

8074d79

vercel bot deployed to Preview May 9, 2024 02:15 View deployment

huydhn merged commit f476189 into pytorch:main May 9, 2024
7 checks passed

clee2000 mentioned this pull request May 9, 2024

[DrCI] Link to hud instead of gh issue for related disabled test #5192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dr.CI] Handle the closing of disabled test issues #5186

[Dr.CI] Handle the closing of disabled test issues #5186

huydhn commented May 8, 2024 •

edited

Loading

vercel bot commented May 8, 2024

vercel bot commented May 8, 2024 •

edited

Loading

clee2000 left a comment

clee2000 May 8, 2024

huydhn May 9, 2024

huydhn May 9, 2024

huydhn commented May 9, 2024

[Dr.CI] Handle the closing of disabled test issues #5186

[Dr.CI] Handle the closing of disabled test issues #5186

Conversation

huydhn commented May 8, 2024 • edited Loading

Testing

❌ 1 New Failure, 3 Unrelated Failures

❌ 1 New Failure, 1 Unrelated Failure

vercel bot commented May 8, 2024

vercel bot commented May 8, 2024 • edited Loading

clee2000 left a comment

Choose a reason for hiding this comment

clee2000 May 8, 2024

Choose a reason for hiding this comment

huydhn May 9, 2024

Choose a reason for hiding this comment

huydhn May 9, 2024

Choose a reason for hiding this comment

huydhn commented May 9, 2024

✅ You can merge normally! (4 Unrelated Failures)

huydhn commented May 8, 2024 •

edited

Loading

vercel bot commented May 8, 2024 •

edited

Loading