diff --git a/docs/regressions.rst b/docs/regressions.rst new file mode 100644 index 00000000..77d2df65 --- /dev/null +++ b/docs/regressions.rst @@ -0,0 +1,126 @@ +Regressions +=========== + +One of the primary uses of ``mozci`` is to help detect which tasks and/or tests (if any) a push has +regressed. Since we do not run all tasks on every push and because of other factors like +intermittents, this problem is more difficult than it first appears. In fact ``mozci`` can make very +few guarantees and so has to rely on probabilistic guesses. + +This page will help explain how regressions are calculated by introducing concepts one at a time. + + +Runnable +-------- + +There are currently two different vectors of regression that ``mozci`` can check for, *label* and +*group*. + +A label is a task label (e.g ``test-linux1804-64/debug-mochitest-e10s-1``). Whereas a "group" is a +grouping of tests, typically a manifest (e.g ``dom/indexedDB/test/mochitest.ini``). + +A *runnable* is the unique label identifying a set of tasks, or the unique group identifying a +set of tests. + + +Runnable Summary +---------------- + +Thanks to retriggers, each runnable can run multiple times on the same push. The collection of +labels or groups of the same type that ran on a push is called a *runnable summary*. For instance, +if all the runnables on a push passed, then the status of the runnable summary is also PASS. +Likewise if they all failed. If at least one instance of a runnable passes, and at least one +instance of a runnable failed, then the runnable summary is said to be intermittent. + +The :class:`~mozci.task.GroupSummary` class implements this logic for groups and the +:class:`~mozci.task.LabelSummary` implements the logic for labels. Both classes inherit from the +:class:`~mozci.task.RunnableSummary` abstract base class. + +All instances of :class:`~mozci.task.RunnableSummary` have an overall status and an overall classification. + +.. note:: + + Classifications are annotations that sherrifs apply to tasks manually. It is also known as + starring because it puts a little asterisk next to the task in Treeherder. + + +Candidate Regression +-------------------- + +A candidate regression is a runnable which meets the following criteria: + + * At least one instance of this runnable failed on target push (i.e, the status of the runnable + summary is either FAIL or INTERMITTENT) + * The overall classification of the runnable summary is either ``unclassified``, or ``fixed by + commit``. This means runnables classified as a known intermittent are not candidate + regressions. + * For runnables classified ``fixed by commit``, the referenced backout backs out the target push + and not some other one. + + OR + + * The runnable ran on a child push (up to :data:`~mozci.push.MAX_DEPTH` pushes away), and + is classified ``fixed by commit``. + * The classification references a backout that backs out the target push. + + +Candidate regressions are the set of all runnables that could possibly be a regression of this push. +This *does not* mean that they are regressions. Just that they could be. + +The set of candidate regressions can be obtained by calling +:meth:`Push.get_candidate_regressions() `. + + +Regression +---------- + +A *regression* is a candidate regression that additionally satisfies the following criteria: + + * The candidate regression is not marked as a regression of any parent pushes up + to :data:`~mozci.push.MAX_DEPTH` pushes away. + * The condition ``total_distance <= MAX_DEPTH`` is satisfied. This condition is explained in more detail below. + +.. note:: Distance Calculation + + The ``total_distance`` is the number of parent pushes we need to go back to see the runnable plus + the number of child pushes we need to go forward to see the runnable. A ``total_distance`` of 0 + means the runnable ran on the actual target push. + + The ``total_distance`` can be modified in certain scenarios: + + 1. The push was not backed out => total distance is doubled. + 2. The runnable was intermittent => total distance is doubled. + 3. The runnable was marked as ``fixed by commit`` referencing a backout that backs out the + target push => total distance is 0 even if it didn't run on the target push. + + These modifications help us deal with (un)certainty in special easy to detect circumstances. The + first two make a candidate regression less likely to be treated as a regression, while the third + guarantees it. + +Regressions can be obtained by calling :meth:`Push.get_regressions() +`. + + +Likely Regressions +------------------ + +A *likely regression* is a regression whose associated ``total_distance`` is 0. In other words, we +are as sure as we can be that these are regressions. + +Likely regressions can be obtained by calling :meth:`Push.get_likely_regressions() +`. + + +Possible Regressions +-------------------- + +A *possible regression* is a regression whose associated ``total_distance`` is above 0. In other +words, it could be a regression, or it could be regressed from one of its parent pushes. We aren't +sure. The higher the ``total_distance`` the less sure we are. + +Likely regressions can be obtained by calling :meth:`Push.get_possible_regressions() +`. + +.. note:: + + Candidate regressions that aren't also possible regressions could still technically be real + regressions. Mozci just thinks the likelihood is so low they aren't worth counting.