Basic Workflow

Iris can be invoked in several ways, depending on your workflow.

One important note:
Running Iris on your primary machine can be very risky, as it takes over keyboard and mouse input while active. In addition, if Iris detects an open instance of Firefox on screen, this may interfere with core Iris logic and cause it to fail. If you develop or deploy Iris, be aware of this and close any active applications.

For all use cases below, follow the instructions in the setup document to enable the virtualenv, i.e.:

pipenv install
pipenv shell
iris [args]

---

Developing Iris

• You can use the Control Center to configure Iris in a visual manner. This is done by default when invoking iris.
• You can use command-line arguments for more low-level control over Iris. This will cause Iris to skip the Control Center and immediately run tests.

---

Deploying Iris

• If this is to be done via a script, cron job, or other mechanism, you will need to adjust the command line arguments to prevent the Control Center from launching. One simple way to do this is to specify the desired build you wish to test, e.g. iris -f beta.
• If you are invoking Iris in a cron job, special consideration will be needed so that the shell commands are properly executed in your script. In this case, replace use of pipenv shell with pipenv run iris [args].

---

Email and Test Rail reporting

• Iris can be made to report results to an email address.
• Test cases in Iris are mapped to Test Rail IDs. To configure Iris to create and update a Test Rail run, edit the config.ini file in the root of the Iris repo and search for the section [Test_rail]. Enter in your own username, password, and URL. Please note that this is your personal, private information, and as such, this file is not intended to be put in a public repo. To use this feature, you must be comfortable storing your Test Rail credentials in plaintext on your computer. Lastly, it is enabled when the config file has been properly set up, and the -r flag has been added to your command-line arguments.

---

Iris and virtualized environments

• Iris may be used in a desktop virtual machine, such as VMware or VirtualBox. However, due to various issues with screen resolution, video drivers, key handling, and font sizes, the behavior can be unpredictable. It is not recommended for this reason. However, we are exploring solutions, and when we establish a standard configuration, we will provide details.
• Iris is not currently able to be used in other types of virtualized environments, such as Docker images or some CI systems, due to issues with virtual screens and keyboards. If you are able to make this work, please get in touch and share your success story.

---

Dealing with test failures

From time to time you may see test failures, this is to be expected. However not all the time will these tests failures be legitimate (we call these false negatives). Please follow this workflow when dealing with test failures.

1. Make note of the tests that failed on the first run through (saving the log is recommended)
2. Trigger another testrun with just the tests that failed (-o flag from the command line)
3. For any tests that still fail, report the issue in Github and run the test manually (the test should be skipped until it can be fixed)
4. For any tests that still fail manually, report the issue in Bugzilla (you've likely found a Firefox bug)

When filing an issue be sure to include:

• a log file containing the output of the testrun
• system information including platform, Firefox version, and locale
• iris command you were running

If you see a massive amount of recurring failures please just file a single issue.

If your issue impacts a specific project (eg. Deploy: Release QA) please make sure you tag that project.

If ever in doubt, please either email us or find us in the #qa-automation channel on mozilla.slack.com