Skip to content

Latest commit

 

History

History
228 lines (175 loc) · 11.5 KB

development-process.md

File metadata and controls

228 lines (175 loc) · 11.5 KB
Raw

Development Process

Welcome to Taskcluster development!

Taskcluster is composed of a bunch of microservices, some libraries, the React user interface, some client libraries, and a bunch of infrastructure to glue it all together. You will probably be working on only one of these pieces, so read carefully below to see what you need to do.

Setup

Node

You will need Node version 12.16.1 installed. We recommend using https://github.com/nvm-sh/nvm to support installing multiple Node versions.

Go

Go version go1.13.7 is required for some development tasks, in particular to run yarn generate. For new contributors not familiar with Go, it's probably safe to skip installing Go for now -- you will see a helpful error if and when it is needed. We recommend using https://github.com/moovweb/gvm to support installing multiple Go versions.

Dependency Installation

To set up the repository, run yarn in the root directory. This will install all required dependencies from the Yarn registry.

For some of the commands below, such as mocha, you will need to make sure that node_modules/.bin is in your PATH. To set this up, in the root directory, run

export PATH=$PWD/node_modules/.bin:$PATH

Hacking on the UI

Taskcluster requires a Linux-like environment for development. If you are developing on a Windows system, you will need to either

  • install WSL
  • use a virtual machine for development or
  • install Linux as a second OS and boot into that.

The files comprising the Taskcluster UI are under ui/. It relies on a microservice called web-server, which you will need to run in a different terminal window. To run the Taskcluster UI:

  • In a shell window for taskcluster-web-server:

    • Set TASKCLUSTER_ROOT_URL to point to a Taskcluster deployment you wish to represent in the UI. For example:

      export TASKCLUSTER_ROOT_URL=https://community-tc.services.mozilla.com

    • Change to the services/web-server directory and run yarn start. This will start a web server on port 3050. Note that it will warn "No Pulse namespace defined". Unless you are working on parts of the UI that require Pulse support (and most do not), this is OK.

  • In another shell window for taskcluster-ui:

    • Change to the ui/ directory.
    • Run yarn to install the user interface's dependencies. You will only need to do this when the dependencies change.
    • Run yarn start to start the development server. It will output a localhost URL on which you can see the site.

Hacking on Services and Libraries

To run all of the tests for a service, change into the directory containing the service (for example, cd services/queue) and run yarn test. Unless you provide additional credentials, lots of tests will be skipped. Many of these are duplicate runs of tests, labeled "real" and "mock".

For most changes, the mock tests are sufficient to detect errors. To be sure, follow the TDD practice of writing your tests first, ensuring that they fail before you fix the bug or add the feature. If your new test is skipped, then consult with one of the Taskcluster team members to see how you can get some credentials.

To run a specific test file, you can use mocha directly:

# run just the `api_test` file
mocha test/api_test.js
# run the tests in that file until the first failure
mocha -b test/api_test.js

All of the server-side code runs in Node, and uses the debug module to log debugging information. You can see all debugging output with

export DEBUG=*

and can filter out specific debugging output with other values, as described in the module documentation.

Generating Code

If you change the API for a service (even changing documentation), you will need to regenerate the API references. Happily, this is easy. In the root directory of the repository, run yarn generate. It will change a few files, and you should include those changes in your Git commit.

Running Services Locally

We generally depend on tests to ensure that services are behaving correctly. It is rare to run a service directly. However, it's possible!

Look in procs.yml in the service's directory to see what processes you can run. For example, you might see:

expireArtifacts:
  type: cron
  schedule: '0 0 * * *'
  deadline: 86400
  command: node src/main expire-artifacts

To run this process locally:

NODE_ENV=development node src/main expire-artifacts

You may need to provide additional configuration, either as environment variables or in user-config.yml. You can find a template for user-config.yml in user-config-example.yml: just copy the latter to the former and edit it.

Running Services in a Development Cluster

You will first need to have

  • a running kubernetes cluster (at the moment this has to be a gke cluster from google)
  • a rabbitmq cluster
  • an azure account
  • an aws account and an IAM user in that account
  • helm installed (either 2 or 3 should Just Work)
  • latest version of kubectl installed

Once those are all set up, you will need:

  • Configure CLI access for your AWS user; this iam user must be able to configure s3/iam resources
  • Think of a hostname for which you control the DNS; this will be your rootUrl. (hint: .taskcluster-dev.net - this domain managed in Route53 in the taskcluster-aws-staging AWS account. You'll have to create a TXT recordset named _acme-challenge..taskcluster-dev.net with the secret that certbot gave you, and after completing the certbot step - IPv4 recordset with your ingress-ip)
  • Run gcloud container clusters get-credentials for your k8s cluster

Now follow along:

  1. Set up an IP address: gcloud compute addresses create <yourname>-ingress-ip --global. You can find the assigned IP in gcloud compute addresses list, and put it into DNS as an A record.
  2. Create a certificate: certbot certonly --manual --preferred-challenges dns. This will ask you to add a TXT record to the DNS. Note that certbot is installed with brew install letsencrypt on macOS.
  3. Upload the certificate: gcloud compute ssl-certificates create <yourname>-ingress --certificate <path-to-fullchain.pem> --private-key <path-to-key>
  4. yarn dev:init will ask you a bunch of questions and fill out your local config for you (most of it anyway). Once it has done this, your dev-config.yml is filled with secrets so don't leak it. These are dev-only secrets though so don't be too worried. Soon we may work on getting this to be encrypted at rest.
    • SubscriptionId can be found in the Azure console
    • RabbitMQ account creds are in passwordstore at tc/cloudamqp.com/hip-macaw
  5. Run yarn dev:verify and see if it complains about any missing values in your configuration. Please note that dev-config.yml defines values for environment variables rather than configuration fields directly, so if you ever need to edit the file manually, instead of entering the names of the config fields from services' config.yml enter the names of the corresponding environment variables in lower case.
  6. If you want to deploy local changes, run yarn build --push and add the resulting image id to your config file with the key dockerImage.
  7. yarn dev:apply will use helm+kubectl to apply all of your kubernetes resources to the cluster. Note that this will create a new namespace in the cluster for you and switch your kubectl context to it. If you make changes, just apply again. It should change anything you've changed and remove anything you've removed.
  8. yarn dev:delete will uninstall your deployment.

Troubleshooting:

  • Certbot error [Errno 13] Permission denied: '/var/log/letsencrypt' Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths. - do not run as root, but set the directories instead.
  • Dev config creation step: AccessDenied: Access Denied error with a stack trace pointing at aws-sdk library - make sure to have your aws credentials are fetched and stored in environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN.
  • Helm error Error: stat taskcluster: no such file or directory - make sure you have helm3 installed.
  • Kubectl error: Error: unknown flag --current - make sure you run kubectl v1.15.0 or later

Optional configuration

You can add a errorConfig to the top-level of your config containing

errorConfig:
  reporter: SentryReporter
  dsn: ...

in order to report your errors to a sentry project.

Setting up a Taskcluster Github app in your Development Cluster

You will need:

  1. Development cluster up and running (see above)
  2. A github app created and installed for a testing repository.

To set up a taskcluster-github app: 0. In the settings of the github app that you created, at the very bottom of the General tab, you will find Generate Private Key button. Press it to generate the private key.

  1. In your dev-config.yml, in the github section, add github_private_pem - you can copy-paste the contents of the PEM file you have obtained in the previous step. Be careful to remove any newlines from the encrypted part, and the necessary newlines after the header and before the footer should be replaced with \n, so the whole thing is a one-line string like this: -----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEblahblahblah==\n-----END RSA PRIVATE KEY-----
  2. Populate the github_app_id in the dev-config.yml (it's the numerical App ID you will find in the app settings, near the top of the General tab)
  3. If you have webhook_secret in your dev-config.yml, remove it.
  4. In the app settings, on that same General tab, find the Webhook URL field. Enter the api URL in there (should be something like https://<YOUR_ROOT_URL>/api/github/v1/github).
  5. Leave the Webhook Secret field empty, make sure the app is Active and the SSL verification is enabled. On the Permissions & Events tab, carefully add permissions. Do not give it more permissions than it needs.
  6. Try it on a test repo. If the app doesn't work, go into the app settings, Advanced tab, and look at the webhook deliveries. Logs of tc-github service are also good for a few laughs.
  7. If the app does work but lacks scopes, you can add those by creating a repo:github.com/<TEST_OWNER>/<TEST_REPO>:* role and adding the scopes to that role.
  8. If you need functional workers to go with the app, make sure to set up a provider in the cloud you need and then create a workerpool for that provider.

Hacking on Clients

The clients are in clients/. One, client-web, is designed for use in web browsers, while the other, client, is intended for use in Node environments. In fact, it is used by all of the services and libraries in the repository. Note that taskcluster-ui does not use client-web, since it communicates with web-server via GraphQL.

Both clients contain a great deal of generated code. Use yarn generate, as described above, to ensure that any changes you make are not overwritten by the code-generation system.

Building the Taskcluster Docker Image

A Taskcluster deployment is based on a Docker image that contains all Taskcluster services, including the UI. It's rare to use this image during development, as running the image requires lots of credentials and a special environment.

But you're welcome to build the image! It's easy: yarn build in the root directory.