Add support for running Kubernetes builds via Google Cloud Build #900

justaugustus · 2019-10-17T09:37:46Z

This PR abstracts the steps of the kubernetes_build scenario in k/test-infra into GCB build steps and a set of build variants which can be triggered without having to dissect the job configs in test-infra.

This is the k/release counterpart to the following PRs: kubernetes/test-infra#14747, kubernetes/test-infra#14773, kubernetes/test-infra#14788

As a note, I plan on cleaning up the Dockerfile in a future PR. I just want to minimize the changes in this PR, so I can start getting signal on these build runs.

Update the k8s-cloud-builder Dockerfile

Here we instead base the image on the most recent version of
k8s.gcr.io/kube-cross (v1.12.12-1), which is an image that contains
all of the relevant dependencies to build Kubernetes. We also add
build steps for the GCP CLI tools (to be able to access GCP resources
within the image) and install docker (to build and push container
images to GCR).
Add gcb/build.yaml

This is a new GCB config file which encapsulates the build steps of
the kubernetes_build scenario, which we use in CI to build
Kubernetes. Specifically, the following build steps are happening:
- git clone kubernetes/kubernetes
- git clone kubernetes/release
- (k/k) make clean
- (k/k) make release-in-a-container
- (k/r) push-build.sh
Add gcb/variants.yaml

variants.yaml files store a set of build types, which contain a set of
GCB substitutions, which are interpreted by
k/test-infra/images/builder before submitting a build or set of builds
to GCB. This pattern is used in several places in k/test-infra to
define build variations (primarily for image building via GCB).

We adopt it here to circumvent creating an additional dependent of
gcbmgr/anago.

Signed-off-by: Stephen Augustus saugustus@vmware.com

k8s-ci-robot · 2019-10-17T09:37:47Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

gcb/build.yaml

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

justaugustus · 2019-10-20T20:53:35Z

/retest

justaugustus · 2019-10-21T20:46:44Z

/assign @tpepper @calebamiles
cc: @kubernetes/release-engineering

listx · 2019-10-21T20:55:50Z

gcb/build.yaml

+timeout: 7200s
+steps:
+- name: 'gcr.io/cloud-builders/git'
+  dir: 'go/src/k8s.io'


Shouldn't this be /workspace/go/src/k8s.io ?

@listx -- either would work, I think.

dir is set relative to /workspace.
ref: https://cloud.google.com/cloud-build/docs/build-config

tpepper · 2019-10-21T21:56:57Z

images/k8s-cloud-builder/Dockerfile

+    && apt-get -y install git \
+    # Setup Google Cloud SDK (latest)
+    && curl -sSL https://sdk.cloud.google.com > /tmp/install.sh \
+    && bash /tmp/install.sh --install-dir=/opt/google --disable-prompts \


A nit, but could this be gotten via apt-get also?

@tpepper -- yep, it can. This Dockerfile is Frankenstein work from a few strewn across other repos (that are known to work). I'm going to squash the layers (as you can see some repetition of commands as well) in a follow-up PR.

tpepper · 2019-10-21T22:04:42Z

images/k8s-cloud-builder/Dockerfile

+        python-dev \
+        python-setuptools \
+        wget \
+    && rm -rf /var/lib/apt/lists/* \


My thinking on getting the Google Cloud SDK via apt was around being able to inventory the container contents, which is useful if nothing else for runtime debug, but likely other reasons too. Being able to docker run and dpkg -L or apt list --installed and see which gcc, which python-dev, which Google Cloud SDK are in a given binary instance of the container described by this Dockerfile is really important versus "it's whatever was available on the internet at build time".

tpepper · 2019-10-21T22:06:24Z

/hold
please squash and put a meaningful description on the commit but otherwise

/lgtm
to merge...ping after you squash for a re-lgtm since this will get cleared.

- Update the k8s-cloud-builder Dockerfile Here we instead base the image on the most recent version of k8s.gcr.io/kube-cross (v1.12.12-1), which is an image that contains all of the relevant dependencies to build Kubernetes. We also add build steps for the GCP CLI tools (to be able to access GCP resources within the image) and install docker (to build and push container images to GCR). - Add gcb/build.yaml This is a new GCB config file which encapsulates the build steps of the `kubernetes_build` scenario, which we use in CI to build Kubernetes. Specifically, the following build steps are happening: - git clone kubernetes/kubernetes - git clone kubernetes/release - (k/k) make clean - (k/k) make release-in-a-container - (k/r) push-build.sh - Add gcb/variants.yaml variants.yaml files store a set of build types, which contain a set of GCB substitutions, which are interpreted by k/test-infra/images/builder before submitting a build or set of builds to GCB. This pattern is used in several places in k/test-infra to define build variations (primarily for image building via GCB). We adopt it here to circumvent creating an additional dependent of gcbmgr/anago. Signed-off-by: Stephen Augustus <saugustus@vmware.com>

k8s-ci-robot · 2019-10-21T22:44:46Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

justaugustus · 2019-10-21T22:47:27Z

This is ready to go. Squashed with an updated commit message and PR description.
/hold cancel

tpepper · 2019-10-22T00:21:30Z

/lgtm

k8s-ci-robot · 2019-10-22T00:21:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, tpepper

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justaugustus,tpepper]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from Bubblemelon and listx October 17, 2019 09:38

justaugustus force-pushed the k8s-build branch 4 times, most recently from 302651e to 59d32ce Compare October 19, 2019 10:25

This was referenced Oct 19, 2019

images/builder: Allow GCB builds from arbitrary build directories kubernetes/test-infra#14747

Merged

sig-release: Add ci-kubernetes-build-canary to canary release tooling kubernetes/test-infra#13340

Closed

justaugustus force-pushed the k8s-build branch 2 times, most recently from 4b5de9b to d6e5cc6 Compare October 19, 2019 17:07

justaugustus changed the title ~~[WIP] gcb: Update k8s-cloud-builder image to support Kubernetes builds~~ [WIP] Add support for running Kubernetes builds via Google Cloud Build Oct 19, 2019

justaugustus added this to the v1.17 milestone Oct 19, 2019

justaugustus force-pushed the k8s-build branch 4 times, most recently from 5ef8e05 to ccf7743 Compare October 21, 2019 19:00

justaugustus mentioned this pull request Oct 21, 2019

releng: Configure kubernetes-build prototype jobs to use the GCB builder kubernetes/test-infra#14788

Merged

justaugustus force-pushed the k8s-build branch from ccf7743 to 393c166 Compare October 21, 2019 20:41

justaugustus changed the title ~~[WIP] Add support for running Kubernetes builds via Google Cloud Build~~ Add support for running Kubernetes builds via Google Cloud Build Oct 21, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2019

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Oct 21, 2019

k8s-ci-robot assigned calebamiles and tpepper Oct 21, 2019

listx reviewed Oct 21, 2019

View reviewed changes

tpepper reviewed Oct 21, 2019

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Oct 21, 2019

justaugustus force-pushed the k8s-build branch from 393c166 to 312ae43 Compare October 21, 2019 22:44

k8s-ci-robot added do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Oct 21, 2019

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Oct 21, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 21, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 22, 2019

k8s-ci-robot merged commit c2a9d36 into kubernetes:master Oct 22, 2019

justaugustus mentioned this pull request Oct 28, 2019

Establish Kubernetes build/stage/release process on k8s-infra #911

Open

23 tasks

Uh oh!

Add support for running Kubernetes builds via Google Cloud Build #900

Add support for running Kubernetes builds via Google Cloud Build #900

Uh oh!

Conversation

justaugustus commented Oct 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 17, 2019

Reasons for blocking this PR:

Uh oh!

justaugustus commented Oct 20, 2019

Uh oh!

justaugustus commented Oct 21, 2019

Uh oh!

listx Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

justaugustus Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

tpepper Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

justaugustus Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

tpepper Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

tpepper commented Oct 21, 2019

Uh oh!

k8s-ci-robot commented Oct 21, 2019

Reasons for blocking this PR:

Uh oh!

justaugustus commented Oct 21, 2019

Uh oh!

tpepper commented Oct 22, 2019

Uh oh!

k8s-ci-robot commented Oct 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

justaugustus commented Oct 17, 2019 •

edited

Loading