Continuous Integration & Deployment
Bedrock runs a series of automated tests as part of continuous integration workflow and `Deployment Pipeline`_. You can learn more about each of the individual test suites by reading their respective pieces of documentation:
- Python unit tests (see :ref:`run-python-tests`).
- Redirect tests (see :ref:`testing-redirects`).
- Functional tests (see :ref:`testing`).
Tests in the lifecycle of a change
Below is an overview of the tests during the lifecycle of a change to bedrock:
The change is developed locally, and all integration tests can be executed against a locally running instance of the application.
Push to master branch
Whenever a change is pushed to the master branch, the smoke suite of headless (see :ref:`testing-redirects`) and UI tests (see :ref:`smoke-functional-tests`) are run against Firefox on Linux. If successful, the change is pushed to the dev environment, and the full suite of headless and UI tests are then run against Firefox on Windows 10 using Sauce Labs. This is handled by the pipeline, and is subject to change according to the settings in the jenkins.yml file in the repository.
Push to prod branch (tagged)
When a tagged commit is pushed to the prod branch, everything that happens during a untagged push is still run. In addition, the full suite of UI tests is run against Chrome and Internet Explorer on Windows 10, and the sanity suite is run against older versions of Internet Explorer (currently IE6 & IE7). If successful, the change is pushed to staging, tested, and then to production and the same tests are then run against production. As with untagged pushes, this is all handled by the pipeline, and is subject to change according to the settings in the jenkins.yml file in the repository.
Push to prod cheat sheet
Check out the
Make sure the
masterbranch is up to date with
- Check that dev deployment is green:
- View deployment pipeline
and look at
- View deployment pipeline and look at
Tag and push the deployment by running
By default the
tag-release.sh script will push to the
origin git remote. If you'd
like for it to push to a different remote name you can either pass in a
--remote argument, or set the
MOZ_GIT_REMOTE environment variable. So the following
$ bin/tag-release.sh --push -r mozilla $ MOZ_GIT_REMOTE=mozilla bin/tag-release.sh --push
And if you'd like to just tag and not push the tag anywhere, you may omit the
Our Jenkinsfile will run the integration tests based on information in our jenkins.yml file. This file specifies various test names per branch that will cause it to use different parameters, allowing it to be called in many different ways to cover the testing needs. The job executes this script, which then runs this Docker image, and ultimately runs another script. The two scripts can also be executed locally to replicate the way Jenkins operates.
During the Test Images stage, the Test Runner job is called without a
BASE_URL. This means
that a local instance of the application will be started, and the URL of this instance
will be used for testing. The
DRIVER parameter is set to
Remote, which causes a
local instance of Selenium Grid to be started in Docker and used for the browser-based
functional UI tests.
The test scripts above will be run once for each properties name specified in the jenkins.yml file for the branch being built and tested. Pushes to master will run different tests than pushes to prod for example.
Many of the options are configured via environment variables passed from the initial script, to the Docker image and onto the final script. This means that global defaults can be configured in Jenkins. Note that admin access is required to make changes to the global configuration, and there is a known issue that may cause Jenkins to become unresponsive after a configuration change.
There are two components for Selenium, which are independently versioned. The first is
the Python client, and this can be updated via the test dependencies. The other
component is the server, which in the pipeline is either provided by a Docker container
or Sauce Labs. The
SELENIUM_VERSION environment variable controls both of these, and
they should ideally use the same version, however it’s possible that availability of
versions may differ. You can check the Selenium Docker versions available. If needed, the global
default can be set and then can be overridden in the individual job configuration.
Adding test runs
Test runs can be added by creating a new properties section in the integration tests script with the parameters of the new test run. This is simply a bash script and you can duplicate a clause of the case staement. For example, if you wanted to run tests in Firefox on both Windows 10 and OS X, you could create the following clauses:
case $1 in osx-firefox) BROWSER_NAME=firefox PLATFORM="OS X 10.11" ;; win10-firefox) BROWSER_NAME=firefox PLATFORM="Windows 10" ;;
You can use Sauce Labs platform configurator to help with the parameter values.
If you have commit rights to our Github repo (mozilla/bedrock) you can simply push
your branch to the branch named
run-integration-tests, and the
app will be deployed and all of the integration tests defined in the
file for that branch will be run. Please announce in our IRC channel (#www on irc.mozilla.org)
that you'll be doing this so that we don't get conflicts.
Known issues in Jenkins
Jenkins stalls after global configuration changes
When using the IRC plugin for notifications, global configuration changes can cause Jenkins to become unresponsive. To make such changes it can be necessary to first restart Jenkins, as this issue only appears some time after Jenkins has been started. A bug for the IRC plugin has been raised.