Firefox-CI Configuration

This repository contains configuration for the Firefox-CI Taskcluster instance, the CI for Firefox and other Mozilla repositories.

Specifically, this configuration does not "ride the trains". Instead, the head of the default branch of this repository applies to all Gecko projects and products. Previous revisions exist for historical context, but have no relevance to production.

Configuration in this repository includes:

• Information about the trains themselves -- Mercurial repositories, access levels, etc.
• Information about external resources -- URLs, pinned fingerprints, etc.
• Settings that should apply to all branches at once -- for example, proportional allocation of work across workerTypes

This repository was originally proposed in Taskcluster RFC#91.

Structure

Data is stored in distinct YAML files in the root of this repository. Each file begins with a lengthy comment describing

• The purpose of the file
• The structure of the data in the file

Code to implement this configuration is in src/ciadmin. The implementation of fxci is in src/fxci.

Access

Data in this repository is a "source of truth" about Gecko's CI automation. It can be accessed from anywhere, including

• Decision tasks, action tasks, and cron tasks
• Hooks
• Utility scripts and other integrations with the automation

Typically such access is either by cloning the repository or by simply fetching a single file using the raw HTTP API method.

Deprecation

Files in this directory are likely to live "forever". It's difficult to determine whether any branch or product still refers to a file, so deleting a file always carries some risk of breakage. Furthermore, regression bisection might build an old revision that refers to a file no longer referred to in the head commit.

Managing CI Configuration

This repository introduces a management tool, tc-admin, for the management of taskcluster resources, such as roles and hooks. It can download existing resources and compare them to a stored configuration. A collection of resources also specifies the set of managed resources: this allows controlled deletion of resources that are no longer expected. It is based on tc-admin, the standard Taskcluster administrative tool; see that library's documentation for more details than are provided here.

Initial Setup

1. Create and activate a new python virtualenv
2. pip install -e .
3. pip install -r requirements/local.txt
4. If you will be applying changes, ensure you have a way of generating taskcluster credentials, such as taskcluster-cli

Starting Concepts

This tool examines the contents of the ci-configuration repository, as well as examining and applying changes to the running taskcluster configuration.

The environment describes the cluster being affected, such as firefoxci or staging. There is also a community environment which is managed separately.

You will usually want to check the changes that will be applied using tc-admin diff and then apply them using tc-admin apply

You can supply --grep to both 'diff' and 'apply' options to limit the effects to specific changes.

Making Config Changes

1. Make changes in a local clone of this repository

2. Determine which taskcluster environment is relevant, such as firefoxci; the options are in environments.yml.

3. Use the tc-admin diff and tc-admin check to ensure the changes are what you expect, passing the appropriate --environment.

   Examples:

   • tc-admin diff --environment=firefoxci
   • tc-admin diff --environment=firefoxci --ids-only - only show the id's of the resources to be modified (much shorter!)
   • tc-admin check --environment=firefoxci

4. Submit changes to Phabricator for review. On landing, the changes will be applied automaticallyi.

To apply changes locally (not recommended):

1. Generate some taskcluster credentials, such as taskcluster signin.

2. Apply the generated configuration using either

   • tc-admin apply --environment=firefoxci to apply all of the generated configuration or
   • tc-admin apply --environment=firefoxci --grep my-changes to apply only the selected areas of new configuration.

   Which you choose will depend on the current state of the repository and whether there are multiple changes waiting to be applied at a later time.

   You will be shown a summary of the changes that have been applied.

More Information

• tc-admin diff --environment=firefoxci

  Generate a diff of the currently running taskcluster configuration, and the one generated from the tip of the ci-configuration repository.

• tc-admin diff --environment=firefoxci --grep somestring

  The grep option will return full configuration entries relating to the provided string. For example, if you have added a new action called my-action then --grep my-action will show only those entries.

• tc-admin generate

  Generates the expected CI configuration. Use --json to get JSON output.

• tc-admin current

  Produces the currently running CI configuration. This also understands --json.

  generate and current are two steps run automatically when using tc-admin diff

• tc-admin <sub-command> --help

  Each command should have helpful text here. These commands are defined in tc-admin; see that tool for more information and to report bugs.

• ci-admin <sub-command> ..

  For backward compatibility, the ci-admin command behaves exactly the same as tc-admin.

Development

To update dependencies, make changes to requirements/*.in, then install pip-compile-multi from PyPI and run pip-compile-multi -s -g requirements/base.in.