Kubernetes Test Infrastructure

The test-infra repository contains a collection of tools for testing Kubernetes and displaying Kubernetes tests results. See also CONTRIBUTING.md.

See the architecture diagram for an overview of how the different services interact.

Viewing test results

• The Kubernetes TestGrid shows historical test results
  • Configure your own testgrid dashboard at testgrid/config/config.yaml
  • Gubernator formats the output of each run
• PR Dashboard finds PRs that need your attention
• Prow schedules testing and updates issues
  • Prow responds to GitHub events, timers and manual commands
  • The prow dashboard shows what it is currently testing
  • Configure prow to run new tests at prow/config.yaml
• Triage Dashboard aggregates failures
  • Triage clusters together similar failures
  • Search for test failures across jobs
  • Filter down failures in a specific regex of tests and/or jobs
• Test history is a deprecated tool
  • Use the triage dashboard instead
  • Summarizes the last 24 hours of testing
  • See Kettle and the corresponding bigquery metrics that largely supplement this information

Automated testing

Test anything with the following pattern:

git clone https://github.com/kubernetes/test-infra
test-infra/jenkins/bootstrap.py --job=J --repo=R --service-account=S.json --upload=gs://B

The --job=J flag specifies what test job to run. The --repo=R (or --bare) flag controls what we check out from git.

Anyone can reconfigure our CI system with a test-infra PR that updates the appropriate files. Detailed instructions follow:

E2E Testing

Our e2e testing uses kubetest to build/deploy/test kubernetes clusters on various providers. Please see those documents for additional details about this tool as well as e2e testing generally.

Create a new job

Create a PR in this repo to add/update/remove a job or suite. Specifically you'll need to do the following:

• Create an entry in jobs/config.json for the job
  • If this is a kubetest job create the corresponding jobs/env/FOO.env file
  • Ensure the PROJECT=blah in the jobs/env/FOO.env has the right IAM grants
• Add the job name to the test_groups list in testgrid/config/config.yaml
  • Also the group to at least one dashboard_tab
• Add the job to the appropriate section in prow/config.yaml
  • Presubmit jobs run on unmerged code in PRs
  • Postsubmit jobs run after merging code
  • Periodic job run on a timed basis
• (Deprecated!) Some old jobs still run on jenkins
  • Please do not add new jobs to jenkins
  • Jenkins configuration is defined at jenkins/job-configs
  • More deprecated details at jenkins/README.md

Please test the job on your local workstation before creating a PR:

mkdir /tmp/whatever && cd /tmp/whatever
$GOPATH/src/k8s.io/test-infra/jenkins/bootstrap.py \
  --job=J \  # aka your new job
  --repo=R1 --repo=R2 \  # what repos to check out
  --service-account ~/S.json  # the service account to use to launch GCE/GKE clusters
# Note: create a service account at the cloud console for the project J uses

Presubmit will tell you if you forget to do any of this correctly.

Merge your PR and the job will be ready to go as soon as test-infra oncall pushes the prow changes (make -C prow update-config).

Update an existing job

Largely similar to creating a new job, except you can just modify the existing entries rather than adding new ones.

Update what a job does by editing its definition in jobs/config.json. For the kubetest jobs this typically means editing the jobs/FOO.env files it uses.

Update when a job runs by changing its definition in prow/config.yaml. The test-infra oncall must push prow changes (make -C prow update-config).

Update where the job appears on testgrid by changing testgrid/config/config.yaml.

Delete a job

The reverse of creating a new job: delete the appropriate entries in jobs/config.json, prow/config.yaml and testgrid/config/config.yaml.

The test-infra oncall must push prow changes (make -C prow update-config).

Building and testing the test-infra

We use Bazel to build and test the code in this repo. The commands bazel build //... and bazel test //... should be all you need for most cases. If you modify Go code, run ./verify/update-bazel.sh to keep BUILD files up-to-date.

Federated Testing

The Kubernetes project encourages organizations to contribute execution of e2e test jobs for a variety of platforms (e.g., Azure, rktnetes). The test-history scripts gather e2e results from these federated jobs. For information about how to contribute test results, see Federated Testing.