Add a 'local' pod mode to the pod-utils and mkpod to generate decorated pods suitable for local testing. #13561

cjwagner · 2019-07-23T23:11:56Z

This PR is broken into 3 commits:

Allow mkpod to autogenerate buildID when 'snowflake' is specified.
Add local output mode to pod utilities.
- In this mode the pod utilities don't require or mount a GCS secret and don't upload to GCS. Instead a hostPath output volume is mounted and files that would have been uploaded to GCS are copied there.
Add support for local pods to mkpod and include a script for running local pods with kind.
- When mkpod is run with --local it configures the pod utilities to output locally and prompts for each volume that is not a hostPath or emptyDir to be replaced by an emptyDir, hostPath, or kept the same (this assumes the volume will be available in the cluster where the pod will run).
- The included local-kind.sh script is a simple wrapper around mkpj, mkpod, and kind. It creates a kind cluster named mkpod if one doesn't exist and uses mkpj and mkpod to create a pod that is then applied to the kind cluster and watched.
  - I'm not set on this being a bash script, but I wrote it this way initially, because it makes it easy to modify for different workflows (e.g. any k8s resources the job depends on could be added to the script). I'm happy to change this.

Example usage after installing docker and kind:

./local-kind.sh pull-test-infra-verify-file-perms

Testing a job from a different Prow instance in a different repo (istio/test-infra):

CONFIG_PATH=$GOPATH/src/istio.io/test-infra/prow/config.yaml JOB_CONFIG_PATH=$GOPATH/src/istio.io/test-infra/prow/cluster/jobs ./local-kind.sh pull-test-infra-prow-checkconfig

I still need to add tests and documention, but I wanted to get some feedback on the direction first.
/assign @stevekuznetsov @fejta
fixes #13475

prow/cmd/mkpod/local-kind-config.yaml

prow/cmd/mkpod/local-kind.sh

stevekuznetsov · 2019-07-24T16:44:59Z

prow/cmd/mkpod/local-kind.sh

+echo config=${config}
+echo job_config=${job_config}
+
+if [[ -n ${job_config} ]]


you enforce that this is set above with :- so no need to check

I used - not :- so I think the check is necessary.
Allowing JOB_CONFIG_PATH="" is necessary to override the default if you are using this script with a different Prow instance that doesn't have separate job config.

This seems like gross UX to me though. I'm open to suggestions.

prow/cmd/mkpod/main.go

prow/gcsupload/options.go

cjwagner · 2019-07-26T22:57:07Z

Changed to having a general script and a prow.k8s.io specific one that defaults the config locations. We can create similar scripts for other prow instances.
PTAL.

fejta · 2019-07-26T23:02:18Z

config/pj-on-kind.sh

+
+export CONFIG_PATH="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/../prow/config.yaml)"
+export JOB_CONFIG_PATH="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/jobs)"
+bash <(curl -s https://raw.githubusercontent.com/cjwagner/test-infra/local-pod-go/prow/pj-on-kind.sh) "$@"


That was pointed at my fork for testing. Fixed.

stevekuznetsov · 2019-07-26T23:19:02Z

config/pj-on-kind.sh

+
+export CONFIG_PATH="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/../prow/config.yaml)"
+export JOB_CONFIG_PATH="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/jobs)"
+bash <(curl -s https://raw.githubusercontent.com/kubernetes/test-infra/master/prow/pj-on-kind.sh) "$@"


Why a curl instead of relative path?

With curl, this script can be copied into any other repo with Prow config (e.g. istio/test-infra) and the only change needed to make everything work is to adjust the config envvars accordingly. Then, jobs for that Prow instance can be run with ./pj-on-kind.sh my-istio-job. Fetching the k/t-i/prow/pj-on-kind.sh file each time ensures that the latest version is always used.

In this case a relative path is equivalent, but I want a copy-pasteable example.

I changed to use a relative path, but left the commented out the curl version.

Might want to just package it in a container image for re-use

prow/cmd/mkpod/local-kind-config.yaml

prow/cmd/mkpod/main.go

stevekuznetsov · 2019-07-26T23:22:31Z

prow/pj-on-kind.sh

+job="${1:-""}"
+config="${CONFIG_PATH:-""}"
+job_config="${JOB_CONFIG_PATH:-""}"
+out_dir="${OUT_DIR:-/tmp/prowjob-out}"


nit: if you're going to quote the empty strings for defaults above you might as well quote this default, too :)

stevekuznetsov · 2019-07-26T23:22:40Z

prow/pj-on-kind.sh

+echo JOB_CONFIG_PATH=${job_config}
+echo OUT_DIR=${out_dir} "(May be different when reusing an existing kind cluster.)"
+
+if [[ -z ${job} ]]


quote your expansions

I think I got them all.

stevekuznetsov · 2019-07-26T23:22:52Z

prow/pj-on-kind.sh

+out_dir="${OUT_DIR:-/tmp/prowjob-out}"
+
+echo job=${job}
+echo CONFIG_PATH=${config}


quote strings with embedded expansions

stevekuznetsov · 2019-07-26T23:24:29Z

prow/pj-on-kind.sh

+fi
+
+# Install kind and set up cluster if not already done.
+if [[ -z $(which kind) ]]


which will output something like /usr/bin/which: no kind in (${PATH}) on failure so this does not do what you want.

use:

if ! which kind >/dev/null 2>&1; then

and i think there was some weirdness with macos on this one, so you might have to use command -v

Switched to command -v based on this SO post: https://stackoverflow.com/a/677212

stevekuznetsov · 2019-07-26T23:25:24Z

prow/pj-on-kind.sh

+    found="true"
+  fi
+done
+if [[ ${found} == "false" ]]


nit: could just inline this into the if where you find the match instead of doing it after the loop`

We only want to do this if we don't find a match though.

stevekuznetsov · 2019-07-26T23:25:56Z

prow/pj-on-kind.sh

+fi
+export KUBECONFIG="$(kind get kubeconfig-path --name="mkpod")"
+
+# Install mkpj and mkpod if not already done.


maybe prefer running the published containers?

I wanted to avoid the volume mounting and figured this workflow makes it easier to tweak mkpj and mkpod when needed.
I'll add a docker mode in a follow up and we can even make that the default if that seems preferable, but this mode is easier to start with while I'm developing.

stevekuznetsov · 2019-07-26T23:28:05Z

GCSupload/ go code LGTM

cjwagner

This should be ready for review now. I tried to clean up the bash and I moved the LocalOutputDir field into the prowapi.GCSOptions type so that it could be specified in ProwJob config as well.

I'll keep improving this in follow up PRs, but I'd like to get this merged so that you don't have to update the prow config to use my forked versions of the pod utils to use this.

cjwagner · 2019-07-27T03:36:44Z

prow/pj-on-kind.sh

+job="${1:-""}"
+config="${CONFIG_PATH:-""}"
+job_config="${JOB_CONFIG_PATH:-""}"
+out_dir="${OUT_DIR:-/tmp/prowjob-out}"


cjwagner · 2019-07-27T03:36:53Z

prow/pj-on-kind.sh

+out_dir="${OUT_DIR:-/tmp/prowjob-out}"
+
+echo job=${job}
+echo CONFIG_PATH=${config}


cjwagner · 2019-07-27T03:37:02Z

prow/pj-on-kind.sh

+echo JOB_CONFIG_PATH=${job_config}
+echo OUT_DIR=${out_dir} "(May be different when reusing an existing kind cluster.)"
+
+if [[ -z ${job} ]]


I think I got them all.

cjwagner · 2019-07-27T03:39:49Z

prow/pj-on-kind.sh

+fi
+
+# Install kind and set up cluster if not already done.
+if [[ -z $(which kind) ]]


Switched to command -v based on this SO post: https://stackoverflow.com/a/677212

cjwagner · 2019-07-27T03:41:32Z

prow/pj-on-kind.sh

+    found="true"
+  fi
+done
+if [[ ${found} == "false" ]]


We only want to do this if we don't find a match though.

cjwagner · 2019-07-27T04:04:41Z

prow/pj-on-kind.sh

+fi
+export KUBECONFIG="$(kind get kubeconfig-path --name="mkpod")"
+
+# Install mkpj and mkpod if not already done.


I wanted to avoid the volume mounting and figured this workflow makes it easier to tweak mkpj and mkpod when needed.
I'll add a docker mode in a follow up and we can even make that the default if that seems preferable, but this mode is easier to start with while I'm developing.

prow/cmd/mkpod/main.go

…local pods with kind.

stevekuznetsov

One bash if nit otherwise good

stevekuznetsov · 2019-07-30T03:26:58Z

prow/pj-on-kind.sh

+# Ensures installation of prow tools, kind, and a kind cluster named "mkpod".
+function ensureInstall() {
+  # Install mkpj and mkpod if not already done.
+  if [[ $(command -v mkpj &>/dev/null) ]]; then


If you're checking the result of a call just do

if ! command -v mkpj >/dev/null 2>&1; then

No need to test if the output string is or is not empty

Also switches to building mkpj and mkpod with go instead of bazel.

stevekuznetsov · 2019-07-30T17:43:07Z

/lgtm
/approve

k8s-ci-robot · 2019-07-30T17:43:09Z

LGTM label has been added.

Git tree hash: f367fcefb3c4bf5f2a3dd4a1b73ff72a1aa3d68b

k8s-ci-robot · 2019-07-30T17:43:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cjwagner, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [cjwagner,stevekuznetsov]
~~prow/OWNERS~~ [cjwagner,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Allow mkpod to autogenerate buildID when 'snowflake' is specified.

53b6b0a

k8s-ci-robot assigned fejta and stevekuznetsov Jul 23, 2019

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 23, 2019

k8s-ci-robot requested review from alvaroaleman and matthyx July 23, 2019 23:12

cjwagner force-pushed the local-pod branch from ee2c6fd to b63f12a Compare July 24, 2019 00:57

cjwagner mentioned this pull request Jul 24, 2019

Better local testing of decorated ProwJobs using Phaino. #13475

Closed

stevekuznetsov reviewed Jul 24, 2019

View reviewed changes

k8s-ci-robot added area/config Issues or PRs related to code in /config size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 26, 2019

cjwagner force-pushed the local-pod branch from b53591b to 6889f8a Compare July 26, 2019 22:55

fejta reviewed Jul 26, 2019

View reviewed changes

cjwagner force-pushed the local-pod branch from 6889f8a to 8ff53c8 Compare July 26, 2019 23:04

stevekuznetsov reviewed Jul 26, 2019

View reviewed changes

stevekuznetsov mentioned this pull request Jul 29, 2019

Use clonerefs without initupload or sidecar #10946

Closed

cjwagner mentioned this pull request Jul 30, 2019

Phaino should respect Refs and ExtraRefs properly. #13452

Closed

cjwagner force-pushed the local-pod branch from 8ff53c8 to 99d9bf2 Compare July 30, 2019 01:02

cjwagner commented Jul 30, 2019

View reviewed changes

cjwagner marked this pull request as ready for review July 30, 2019 01:06

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2019

cjwagner added 2 commits July 29, 2019 18:11

Add local output mode to pod utilities.

8aa5063

Add support for local pods to mkpod and include a script for running …

1a9b974

…local pods with kind.

cjwagner force-pushed the local-pod branch from 99d9bf2 to 9181831 Compare July 30, 2019 01:12

stevekuznetsov reviewed Jul 30, 2019

View reviewed changes

Make general and k8s specific versions of pj-on-kind.sh.

e57d667

Also switches to building mkpj and mkpod with go instead of bazel.

cjwagner force-pushed the local-pod branch from 9181831 to e57d667 Compare July 30, 2019 17:40

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 30, 2019

k8s-ci-robot merged commit 9330dba into kubernetes:master Jul 30, 2019

k8s-ci-robot added this to the v1.16 milestone Jul 30, 2019

cjwagner deleted the local-pod branch July 31, 2019 18:27

cjwagner mentioned this pull request Aug 8, 2019

Run a prowjob with podutils locally #6590

Closed

Add a 'local' pod mode to the pod-utils and mkpod to generate decorated pods suitable for local testing. #13561

Add a 'local' pod mode to the pod-utils and mkpod to generate decorated pods suitable for local testing. #13561

Conversation

cjwagner commented Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjwagner commented Jul 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov commented Jul 26, 2019

cjwagner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov commented Jul 30, 2019

k8s-ci-robot commented Jul 30, 2019

k8s-ci-robot commented Jul 30, 2019

cjwagner commented Jul 23, 2019 •

edited

Loading