Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make RunnableSummary.is_intermittent return None when there is not enough information #666

Open
marco-c opened this issue Feb 24, 2022 · 0 comments

Comments

@marco-c
Copy link
Collaborator

marco-c commented Feb 24, 2022

RunnableSummary.is_intermittent should return None when we don't have enough info to tell for sure (a bit like we do for cross-config and config-consistent failures).
The prior regressions detection logic (

mozci/mozci/push.py

Lines 964 to 988 in 944248b

if name in runnable_summaries:
found_in_parent = True
summary = runnable_summaries[name]
# If the failure is not intermittent...
if not failure_summary.is_intermittent:
# ...and it failed permanently in the first parent where it ran, it is a
# prior regression.
# Otherwise, if it passed or was intermittent, it is likely not a prior
# regression.
if (
summary.status != Status.PASS
and not summary.is_intermittent
):
prior_regression = True
break
# If the failure is intermittent and it failed intermittently in a close
# parent too, it is likely a prior regression.
# We need to explore the parent's parents instead if it passed or failed
# consistently in the parent.
elif summary.is_intermittent:
prior_regression = True
break
) should take None into account and handle all possible cases: parent None child None, parent intermittent child intermittent, parent real child real, parent None child intermittent, etc..

E.g. in a case like this:

def test_intermittent_failure_and_consistent_in_parent(create_pushes):
    """
    Tests the scenario where a task failed intermittently in a push, and failed
    consistently in its parent.
    """
    p = create_pushes(3)
    i = 1  # the index of the push we are mainly interested in

    p[i - 1].tasks = [
        create_task(
            id=fake_id(1),
            label="test-prova",
            result="failed",
            classification="not classified",
        ),
    ]
    p[i].tasks = [
        create_task(
            id=fake_id(2),
            label="test-prova",
            result="failed",
            classification="not classified",
        ),
        create_task(
            id=fake_id(3),
            label="test-prova",
            result="passed",
        ),
    ]

How do we know if the task wouldn't have failed intermittently in the parent push too, had it had a chance to run multiple times? We should consider this case as a prior regression (assuming we have no classifications and no backout information here, the situation is different in case we have them).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant