Skip to content
Branch: master
Find file History
justaugustus sig-release: Use kubekins for debs/rpms build jobs
Signed-off-by: Stephen Augustus <>
Latest commit abf005e Aug 17, 2019

Kubernetes Prow Job Configs

All prow job configs for live here.

They are tested and validated by tests in config/tests

Job Cookbook

This document attempts to be a step-by-step, copy-pastable guide to the use of prow jobs for the Kubernetes project. It may fall out of date. For more info, it was sourced from the following:

Job Types

There are three types of prow jobs:

  • Presubumits run against code in PRs
  • Postsubmits run after merging code
  • Periodics run on a periodic basis

Please see ProwJob docs for more info

Job Images

Where possible, we prefer that jobs use images that are pinned to a specific version, containing only what is needed.

Some good examples include:

Many jobs use images that are built from source in images/. Some of these have evolved organically, with way more dependencies than needed, and will be periodically bumped by PRs. These are sources of technical debt that are often not very well maintained. Use at your own risk, eg:

  • pull-community-verify uses to run make verify, which ends up invoking some bash scripts, which use commands like git and go. The gcloud dependency is not needed at all.
  • periodic-kubernetes-e2e-packages-pushed uses to run ./tests/e2e/packages/ which ends up running apt-get and yum commands. Perhaps a debian image would be better.

Job Presets

Prow supports Presets to define and patch in common env vars and volumes used for credentials or common job config. Some are defined centrally in [prow/config.yaml], while others can be defined in files here. eg:

Job Examples

A presubmit job named "pull-community-verify" that will run against all PRs to kubernetes/community's master branch. It will run make verify in a checkout of kubernetes/community at the PR's HEAD. It will report back to the PR via a status context named pull-kubernetes-community. Its logs and results are going to end up in GCS under kubernetes-jenkins/pr-logs/pull/community. Historical results will display in testgrid on the sig-contribex-community dashboard under the pull-verify tab

  - name: pull-community-verify  # convention: (job type)-(repo name)-(suite name)
      testgrid-dashboards: sig-contribex-community
      testgrid-tab-name: pull-verify
    - master
    decorate: true
    always_run: true
      - image: golang:1.12.5
        - /bin/bash
        - -c
        # Add GOPATH/bin back to PATH to workaround #9469
        - "export PATH=$GOPATH/bin:$PATH && make verify"

A periodic job named "periodic-cluster-api-provider-aws-test-creds" that will run every 4 hours against kubernetes-sigs/cluster-api-provider-aws's master branch. It will run ./scripts/ in a checkout of the repo located at The presets it's using will ensure it has aws credentials and aws ssh keys in well known locations. Its logs and results are going to end up in GCS under kubernetes-jenkins/logs/periodic-cluster-api-provider-aws-test-creds. Historical results will display in testgrid on the sig-cluster-lifecycle-cluster-api-provider-aws dashboard under the test-creds tab

It's using the kubekins-e2e image which isn't recommended, but works for now.

- name: periodic-cluster-api-provider-aws-test-creds
    testgrid-dashboards: sig-cluster-lifecycle-cluster-api-provider-aws
    testgrid-tab-name: test-creds
  decorate: true
  interval: 4h
    preset-service-account: "true"
    preset-aws-ssh: "true"
    preset-aws-credential: "true"
  - org: kubernetes-sigs
    repo: cluster-api-provider-aws
    base_ref: master
    path_alias: ""
    - image:
      - "./scripts/"

Adding or Updating Jobs

  1. Find or create the prowjob config file in this directory
    • In general jobs for use org/repo/filename.yaml
    • For kubernetes/kubernetes we prefer kubernetes/sig-foo/filename.yaml
    • Ensure filename.yaml is unique across the config subdir; prow uses this as a key in its configmap
  2. Ensure an OWNERS file exists in the directory for job, and has appropriate approvers/reviewers
  3. Write or edit the job config (please see how-to-add-new-jobs)
  4. Ensure the job is configured to to display its results in
  5. Ensure any necessary formatting or config generation happens
  6. Open a PR with the changes; when it merges @k8s-ci-robot will deploy the changes automatically

Deleting Jobs

  1. Find the prowjob config file in this directory
  2. Remove the entry for your job; if that was the last job in the file, remove the file
  3. If the job had no testgrid annotations, ensure its [testgrid/config.yaml] entries are gone
  4. Ensure any necessary formatting or config generation happens
  5. Open a PR with the changes; when it merges @k8s-ci-robot will deploy the changes automatically

Testing Jobs Locally

Please try using phaino, it will interactively help you run a docker command that approximates the pod that would be scheduled on behalf of an actual prow job.

Testing Jobs Remotely

This requires a running instance of prow. In general we discourage the use of as a testbed for job development, and recommend the use of your own instance of prow for faster iteration. That said, an approach that people have used in the past with mostly-there jobs is to iterate via PRs; just recognize this is going to depend on review latency.

For more details, please refer to How to Test a ProwJob

Generated Jobs

There are some sets of jobs that are generated and should not be edited by hand. These specific instructions should probably just live adjacent to the jobs rather than in this central README, but here we are for now.

kubernetes-security jobs

These are automatically generated by hack/ and should require no human intervention

image-validation jobs

These test different master/node image versions against multiple k8s branches. If you want to change these, update experiment/test_config.yaml and then run

bazel run //experiment:generate_tests -- --yaml-config-path=experiment/test_config.yaml

release-branch jobs

When a release branch of kubernetes is first cut, the current set of master jobs must be forked to use the new release branch. Use experiment/config-forker to accomplish this, eg:

# from test-infra root
bazel run //experiment/config-forker -- \
  --job-config $(pwd)/config/jobs \
  --version 1.15 \
  --output $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.15.yaml
You can’t perform that action at this time.