Daily Beta Test Run Results

Tracking tests

This document will explain the process the Iris test case development team should follow to ensure that test failures are properly tracked and addressed in a timely manner. Daily Iris test runs of Firefox Beta results can be found at Irishub.

Process explained

Triaging results

Look at each test reported as Fail or Error in IrisHub to determine if the failure is an Iris issue or potentially a Firefox bug. This may require running the test locally on affected platform(s). In any case, file a NEW issue in GitHub to “Fix (test_case)”. Please do not open previous issues for the same test. In the new issue provide the following information:

• Add a link to the TestRail test associated with the iris test case.
  • (https://testrail.stage.mozaws.net/index.php?/suites/view/ test ID # here)
• A brief description of failure. At minimum, provide which test assert is failing or which helper is throwing an error.
• Label the issue with
  • Priority: High
  • Test: regression
  • OS: All (or each platform the issue affects)
  • Suite: (test group)
  • Target: Firefox
• If this is an Iris issue, add Label:
  • Disabled: Iris
• If the problem is believed to be a Firefox bug, even if not confirmed yet, add the Label:
  • Disabled: Firefox
• Post a link to the TestRail test case associated with this Iris test and give a brief explanation of the failure in #qa-iris-failures in slack. (https://testrail.stage.mozaws.net/index.php?/suites/view/ test ID # here)
• Then create a branch from dev, ie. issue_1234_d. In the affected test, blocked_by issue_1234 for each OS the test affects.
• Close the issue_1234_d branch.
• Leave the GitHub issue open and assign it to Tracy (for admin tasks. see below)

Process for working on open issues

The triage process above will give us two main paths of work to follow:

Firefox bugs

The first path of work is when a test failure is confirmed to be a Firefox bug, In this case, we need to be notified by RelQA with the bug number. When that happens, add the bug number to the previously opened GitHub issue. These issues will remain in this state until we are notified the Firefox bug has been fixed. When that occurs, change the Priority label in the Iris issue to Priority: Critical. These issues become first priority to work on:

• Create a branch from dev based on the open issue.
• In that working branch, remove the blocked_by status from the test case.
• Ensure the re-enabled test case PASSes on all platforms (perhaps additional work is required, In that case, do what is necessary to make the test pass)
• Then get review and land the fix to re-enable the test on dev
• Assign the issue to Tracy (for admin tasks. see below)

Iris test bustage

The second process is for fixing Iris tests failing due to issue(s) that are not Firefox bugs. In GitHub go to Issues > Labels > select ‘Disabled: Iris’ and sort by oldest. Work on the oldest issues first:

• Assign the issue to yourself.
• Create a branch from dev: issue_####
• In that working branch, remove the blocked_by status from the test case
• Investigate and fix the test to ensure it PASSes on all platforms
• Then get review and land fix on dev
• Assign the issue to Tracy (for admin tasks. see below)

Development priority

The above process gives us a basic firefox target development priority of:

1. Triage the daily test run results to ensure every test failure has an open tracking issue.
2. Re-enable tests in Disabled: Firefox issues now un-blocked because of Firefox bug fix(es).
3. Fix issues with Disabled: Iris label. (currently a large backlog, work on oldest first)
4. New test case development.

For admins

We have agreed to a process for communication with RelQA for which tests are actually being run or not in our daily test runs. This process requires us to keep the Automation field in each test case in TestRail, that we have automated, up to date on a daily basis by ensuring the Automation field is accurate with either Completed (it is being run) or Disabled (not run). I (Tracy) have to do this, because our test development team members do not have edit rights in TestRail tests.