Test infrastructure for the Kubernetes project.
Go Python JavaScript Shell HTML Makefile Other
Switch branches/tags
Nothing to show
Clone or download
k8s-ci-robot Merge pull request #8748 from Random-Liu/enable-prepull-for-containerd
Enable prepull for k8s 1.11+ containerd test.
Latest commit 0e4d5ea Jul 20, 2018
Failed to load latest commit information.
autogo Add autogo/README.md May 8, 2018
boskos Retire GCI jobs and the project pool Jul 14, 2018
charts/mungegithub Remove kubemark-5 test jobs from head Jul 5, 2018
config better handle missing scenario args Jul 20, 2018
def Create a k8s_configmap() rule, use in testgrid Apr 3, 2018
dind Fix dind lint errors Jun 18, 2018
docs Clarify that TestGrid doesn't require editing /kettle/buckets.yaml. Jun 26, 2018
experiment Merge pull request #8699 from BenTheElder/shipit_again Jul 17, 2018
gcsweb Run hack/update-bazel.sh Feb 26, 2018
ghclient Run hack/update-bazel.sh Feb 26, 2018
ghproxy Switch ghproxy base image from distoless to alpine. May 24, 2018
greenhouse reduce minimum space reserved on the greenhouse cache disk Jul 16, 2018
gubernator Remove kubemark-5 presubmit+canary - update with newer scalability pr… Jun 18, 2018
hack fix jobs dir in hack/update-config.sh Jul 3, 2018
images Move the image used for cluster-api testing to have an appropriate name. Jul 20, 2018
jenkins retry service account activation instead of immediately failing Jul 2, 2018
jobs Merge pull request #8748 from Random-Liu/enable-prepull-for-containerd Jul 21, 2018
kettle Fix /kettle/buckets.yaml path for Docker container. Jun 26, 2018
kubetest remove deprecated fields Jul 2, 2018
label_sync Make milestone maintainer team configurable per repo. Jul 9, 2018
logexporter Pass 2: k8s GCR vanity URL Apr 6, 2018
maintenance aws-janitor: chunk deletions into batches of 1000 Jul 20, 2018
metrics Istio flakiness metrics by week Jun 30, 2018
mungegithub Remove kubemark-5 test jobs from head Jul 5, 2018
planter fix id -F does not exist on linux Jul 2, 2018
prow Merge pull request #8770 from fejta/over Jul 21, 2018
queue_health Makefiles: avoid repeated shell evaluations Jun 3, 2018
robots Make robots pass lint Jun 14, 2018
scenarios move KUBE_CUSTOM_NETD_YAML to prow/config.yaml Jun 21, 2018
testgrid Configuration for running per-pull tests for kubernetes-sigs/cluster-… Jul 20, 2018
third_party/moby Start docker with systemd. Allow dind as well as dindind. Mar 27, 2018
triage Add "file bug" links to each cluster. Jul 4, 2018
velodrome Fix velodrome lint errors, fixes #8355 Jun 19, 2018
vendor vendor go-zglob Jun 26, 2018
.bazelrc Move print-workspace-status.sh to hack/ directory Apr 11, 2018
.gitignore Use vendored gazelle and kazel in hack/ scripts Mar 26, 2018
.kazelcfg.json Use gazelle and kazel together to manage BUILD files Aug 11, 2017
.project squash May 9, 2017
BUILD.bazel refactor fixconfig, rename to genjobs and move into config dir Jun 28, 2018
CODEOWNERS Test required_reviewers with prow config Jul 19, 2018
Gopkg.lock update deps Jul 11, 2018
Gopkg.toml Ensure golint is vendored Jun 14, 2018
LICENSE Update boilerplate to remove "All rights reserved" (#458) Aug 26, 2016
OWNERS Add myself to OWNERS for vendor/, etc May 9, 2018
README.md update docs on how to create new jobs Jul 19, 2018
WORKSPACE Bump git image for clonerefs. May 10, 2018
code-of-conduct.md Add code-of-conduct.md Dec 20, 2017
commands.md Tombstone commands.md files Feb 5, 2018
image.bzl Move print-workspace-status.sh to hack/ directory Apr 11, 2018
pylintrc Address nits. Jul 14, 2017
test_infra.bzl Require boilerplate on Dockerfiles, Makefiles, and Bazel Skylark sources Feb 16, 2018


Kubernetes Test Infrastructure

Build Status Go Report Card GoDoc

The test-infra repository contains a collection of tools for testing Kubernetes and displaying Kubernetes tests results. See also CONTRIBUTING.md.

See the architecture diagram for an overview of how the different services interact.

Viewing test results

Automated testing

Test anything with the following pattern:

git clone https://github.com/kubernetes/test-infra
test-infra/jenkins/bootstrap.py --job=J --repo=R --service-account=S.json --upload=gs://B

The --job=J flag specifies what test job to run. The --repo=R (or --bare) flag controls what we check out from git.

Anyone can reconfigure our CI system with a test-infra PR that updates the appropriate files. Detailed instructions follow:

E2E Testing

Our e2e testing uses kubetest to build/deploy/test kubernetes clusters on various providers. Please see those documents for additional details about this tool as well as e2e testing generally.

Create a new job

Create a PR in this repo to add/update/remove a job or suite. Specifically you'll need to do the following:

  • Add the job to the appropriate section in config/jobs

    • Directory Structure:
      • In general for jobs for github.com/org/repo use config/jobs/org/repo/filename.yaml
      • For Kubernetes repos we also allow config/jobs/kubernetes/sig-foo/filename.yaml
      • We use basename of the config name as a key in the prow configmap, so the name of your config file need to be unique across the config subdir
    • Type of jobs:
      • Presubmit jobs run on unmerged code in PRs
      • Postsubmit jobs run after merging code
      • Periodic job run on a timed basis
      • You can find more prowjob definitions at how-to-add-new-jobs
    • Scenario args: (if you are using bootstrap.py instead of podutils)
      • Scenarios are python wrappers used by our entry point script bootstrap.py.
      • You can append scenario/kubetest args inline in your prowjob definition, example:
        - name: foo-repo-test
          interval: 1h
          agent: kubernetes
            - image: gcr.io/k8s-testimages/kubekins-e2e:latest-master
              - --repo=github.com/org/repo
              - --timeout=90
              - --scenario=execute
              - --
              - make
              - test
  • Add the job name to the test_groups list in testgrid/config.yaml

    • Also the group to at least one dashboard_tab

The configs need to be sorted and kubernetes must be in sync with the security repo, or else presubmit will fail. You can run the script below to keep them valid:


NOTE: kubernetes/kubernetes and kubernetes-security/kubernetes must have matching presubmits.

Please test the job on your local workstation before creating a PR:

mkdir /tmp/whatever && cd /tmp/whatever
$GOPATH/src/k8s.io/test-infra/jenkins/bootstrap.py \
  --job=J \  # aka your new job
  --repo=R1 --repo=R2 \  # what repos to check out
  --service-account ~/S.json  # the service account to use to launch GCE/GKE clusters
# Note: create a service account at the cloud console for the project J uses

Release branch jobs & Image validation jobs

Release branch jobs and image validation jobs are defined in test_config.yaml. We test different master/node image versions against multiple k8s branches on different features.

Those jobs are using channel based versions, current supported testing map is:

  • k8s-dev : master
  • k8s-beta : release-1.11
  • k8s-stable1 : release-1.10
  • k8s-stable2 : release-1.9
  • k8s-stable3 : release-1.8

Our build job will generate a ci/(channel-name) file pointer in gcs.

After you update test_config.yaml, please run

bazel run //experiment:generate_tests -- --yaml-config-path=experiment/test_config.yaml

to regenerate the job configs.

We are moving towards making more jobs to fit into the generated config.

Presubmit will tell you if you forget to do any of this correctly.

Merge your PR and @k8s-ci-robot will deploy your change automatically.

Update an existing job

Largely similar to creating a new job, except you can just modify the existing entries rather than adding new ones.

Update what a job does by editing its definition in config/jobs.

Update where the job appears on testgrid by changing testgrid/config.yaml.

Delete a job

The reverse of creating a new job: delete the appropriate entries in config/jobs and testgrid/config.yaml.

Merge your PR and @k8s-ci-robot will deploy your change automatically.

Building and testing the test-infra

We use Bazel to build and test the code in this repo. The commands bazel build //... and bazel test //... should be all you need for most cases. If you modify Go code, run ./hack/update-bazel.sh to keep BUILD.bazel files up-to-date.

Contributing Test Results

The Kubernetes project encourages organizations to contribute execution of e2e test jobs for a variety of platforms (e.g., Azure, rktnetes). For information about how to contribute test results, see Contributing Test Results.

Other Docs