Apply isFailureFromPrevMergeCommit logic to all similar failures #5323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
facebook-github-bot
added
the
CLA Signed
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
Spot check some more reverted PRs ⏳ 20 Pending, 2 Unrelated FailuresAs of commit 0b36b0301cc77abf86ab9ad0764f945bc1046f95 with merge base f681e3689b857b8811f19d60d439bfb3fb2dd2d3 ( FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
|
clee2000
approved these changes
Jun 13, 2024
clee2000
reviewed
Jun 13, 2024
| @@ -52,6 +52,7 @@ describe("Test various utils used by Dr.CI", () => { | |||
| await hasSimilarFailures( | |||
| job, | |||
| emptyBaseCommitDate, | |||
| [], | |||
There was a problem hiding this comment.
sorry is there any testing for this? am i just looking in the wrong place
There was a problem hiding this comment.
Let me add a test for this, moving mergeCommits is actually a boon here because I can now easily mock it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #5188
This is to address the mismatch reported in #5188 where some legit failures from the previous merge commit are marked as flaky. We already have the logic to gate against this in
isFailureFromPrevMergeCommit. The bug here is that if a match from a subsequent broken trunk commit was found beforeisFailureFromPrevMergeCommitwas applied, the match was returned right away.There are 2 fixes here:
isFailureFromPrevMergeCommitto all potential matches. Even if one failure shows up from the previous merge commit, the failure will be treated as a new failure.getPRMergeCommitsfunction call out ofhasSimilarFailuresbecause this is the same for all failures in the PR, so we don't need to call it multiple times for each failures.Testing
pytorch/pytorch#124045
BEFORE
❌ 9 New Failures, 3 Unrelated Failures
As of commit 0b8801e63882c96a3d8650d3076ffbc96cad37c6 with merge base c59a2369bec99dbbc088ac5477efe1ab6adc7ec8 (
):
NEW FAILURES - The following jobs have failed:
inductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheFLAKY - The following jobs failed but were likely due to flakiness present on trunk:
inductor/test_max_autotune.py::TestMaxAutotune::test_max_autotune_remote_caching_dynamic_Falseinductor/test_torchinductor.py::CpuTests::test_AllenaiLongformerBase_repro_cpuinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheAFTER
❌ 12 New Failures
As of commit 0b8801e63882c96a3d8650d3076ffbc96cad37c6 with merge base c59a2369bec99dbbc088ac5477efe1ab6adc7ec8 (
):
NEW FAILURES - The following jobs have failed:
inductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_max_autotune.py::TestMaxAutotune::test_max_autotune_remote_caching_dynamic_Falseinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_codecache.py::TestUtils::test_fresh_inductor_cacheinductor/test_torchinductor.py::CpuTests::test_AllenaiLongformerBase_repro_cpuinductor/test_codecache.py::TestUtils::test_fresh_inductor_cachepytorch/pytorch#128464
BEFORE
❌ 1 New Failure, 2 Unrelated Failures
As of commit b245737edbb1a5372b3b63df870aed9746e94958 with merge base 5d8c7f39d46699d8f8e92512309ea3499a29c08a (
):
NEW FAILURE - The following job has failed:
>>> Lint for torch/csrc/autograd/profiler_kineto.cpp:FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
test_all_reduce_no_op_with_one_replicaProcess completed with exit code 1.AFTER
❌ 1 New Failure, 2 Unrelated Failures
As of commit b245737edbb1a5372b3b63df870aed9746e94958 with merge base 5d8c7f39d46699d8f8e92512309ea3499a29c08a (
):
NEW FAILURE - The following job has failed:
>>> Lint for torch/csrc/autograd/profiler_kineto.cpp:FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
test_all_reduce_no_op_with_one_replicaProcess completed with exit code 1.