Build release tars using bazel #39898

ixdy · 2017-01-13T22:59:41Z

What this PR does / why we need it: builds equivalents of the various kubernetes release tarballs, solely using bazel.

For example, you can now do

$ make bazel-release
$ hack/e2e.go -v -up -test -down

Special notes for your reviewer: this is currently dependent on ixdy/bazel@3b29803, which I have yet to turn into a pull request, since I'm still trying to figure out if this is the best approach.

Basically, the issue comes up with the way we generate the various server docker image tarfiles and load them on nodes:

we md5sum the binary being encapsulated (e.g. kube-proxy) and save that to $binary.docker_tag in the server tarball
we then build the docker image and tag using that md5sum (e.g. gcr.io/google_containers/kube-proxy:$MD5SUM)
we docker save this image, which embeds the full tag in the $binary.tar file.
on cluster startup, we docker load these tarballs, which are loaded with the tag that we'd created at build time. the nodes then use the $binary.docker_tag file to find the right image.

With the current bazel docker_build rule, the tag isn't saved in the docker image tar, so the node is unable to find the image after docker loading it.

My changes to the rule save the tag in the docker image tar, though I don't know if there are subtle issues with it. (Maybe we want to only tag when --stamp is given?)

Also, the docker images produced by bazel have the timestamp set to the unix epoch, which is not great for debugging. Might be another thing to change with a --stamp.

Long story short, we probably need to follow up with bazel folks on the best way to solve this problem.

Release note:

NONE

k8s-reviewable · 2017-01-13T22:59:48Z

This change is

k8s-ci-robot · 2017-01-14T00:08:19Z

Jenkins kops AWS e2e failed for commit 386b1bcf1ae7959924a34401f63d92ef5a122e94. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot kops aws e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:08:49Z

Jenkins verification failed for commit 386b1bcf1ae7959924a34401f63d92ef5a122e94. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot verify test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:10:55Z

Jenkins GCE etcd3 e2e failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot gce etcd3 e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:11:25Z

Jenkins GCI GCE e2e failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot gci gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:11:25Z

Jenkins unit/integration failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:11:25Z

Jenkins CRI GCE Node e2e failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot cri node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:11:26Z

Jenkins GCE e2e failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot cvm gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-14T00:12:25Z

Jenkins Kubemark GCE e2e failed for commit fff44e7fd6da44c85369da83fe24b12b17485af6. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot kubemark e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ixdy · 2017-01-14T00:14:29Z

making a few changes/fixes, sorry for the noise.

I think this should be working now.

ixdy · 2017-01-14T00:18:24Z

I think this should be working now.

I lied. But maybe now.

k8s-ci-robot · 2017-01-14T01:23:12Z

Jenkins GCE Node e2e failed for commit 1ecca07. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

spxtr · 2017-01-14T05:41:40Z

@k8s-bot node e2e test this
#35983

spxtr · 2017-01-17T18:24:33Z

cluster/saltbase/BUILD

+    tags = ["automanaged"],
+)
+
+# TODO(#3579): This is a temporary hack. It gathers up the yaml,


How temporary?

I copied the comment from here. I'm still not sure if it's necessary or not.

Ah, "temporary"

spxtr · 2017-01-17T18:28:36Z

build/release-tars/BUILD

+
+genrule(
+    name = "dummy",
+    outs = [".dummy"],


spxtr · 2017-01-17T18:35:18Z

Overall this looks really spiffy. Have you reached out to the bazel people about the docker rule updates?

spxtr · 2017-01-17T19:44:24Z

I'm not too happy the build speeds, but we can optimize later. It's still significantly faster than make quick-release for both the initial and incremental builds.

mikedanese · 2017-01-18T18:00:37Z

build/release-tars/BUILD

+    cmd = "touch $@",
+)
+
+# Some of the startup scripts fail if there isn't an addons/ directory in the server tarball.


TODOD: delete this

yeah, this part makes me sad. I'm not sure how many places assume server/addons/ exists, but that can hopefully be addressed in a follow-up.

mikedanese · 2017-01-18T18:01:18Z

This is so beautiful i cried

mikedanese · 2017-01-18T18:01:28Z

/lgtm

k8s-ci-robot · 2017-01-18T18:57:06Z

Jenkins GCI GKE smoke e2e failed for commit 1ecca07. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2017-01-18T18:59:35Z

Jenkins GKE smoke e2e failed for commit 1ecca07. Full PR test history. cc @ixdy

The magic incantation to run this job again is @k8s-bot cvm gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ixdy · 2017-01-18T20:43:55Z

@k8s-bot gke e2e test this

k8s-github-robot · 2017-01-18T21:37:45Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-01-18T22:24:48Z

Automatic merge from submit-queue

ixdy added 4 commits January 13, 2017 14:14

Use custom io_bazel repo for docker_build changes

23fac9d

Reorder package_kube_manifests_tarball a bit to make it more readable

c0e972d

Refactor docker bazel rules and tag docker images

9beb13a

Add genrule to produce e2e_node.test binary artifact

14dd0d3

ixdy added the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Jan 13, 2017

ixdy assigned mikedanese and spxtr Jan 13, 2017

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 13, 2017

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. labels Jan 13, 2017

ixdy force-pushed the bazel-release-tars branch 3 times, most recently from 386b1bc to fff44e7 Compare January 14, 2017 00:05

ixdy force-pushed the bazel-release-tars branch from fff44e7 to c53bc50 Compare January 14, 2017 00:09

ixdy added 2 commits January 13, 2017 16:17

Build release tarballs in bazel and add make bazel-release rule

bc4b6ac

Update scripts to look for binary artifacts in bazel-bin/

b9e060a

ixdy force-pushed the bazel-release-tars branch from c53bc50 to b9e060a Compare January 14, 2017 00:17

Add release-tars to ci-artifacts

1ecca07

spxtr reviewed Jan 17, 2017

View reviewed changes

build/release-tars/BUILD

genrule(

name = "dummy",

outs = [".dummy"],

Copy link

Contributor

spxtr Jan 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😕

mikedanese reviewed Jan 18, 2017

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 18, 2017

ixdy removed the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Jan 18, 2017

k8s-github-robot merged commit b29d9cd into kubernetes:master Jan 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build release tars using bazel #39898

Build release tars using bazel #39898

ixdy commented Jan 13, 2017

k8s-reviewable commented Jan 13, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

ixdy commented Jan 14, 2017

ixdy commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

spxtr commented Jan 14, 2017

spxtr Jan 17, 2017

ixdy Jan 18, 2017

spxtr Jan 18, 2017

spxtr Jan 17, 2017

spxtr commented Jan 17, 2017

spxtr commented Jan 17, 2017

mikedanese Jan 18, 2017

ixdy Jan 18, 2017

mikedanese commented Jan 18, 2017

mikedanese commented Jan 18, 2017

k8s-ci-robot commented Jan 18, 2017

k8s-ci-robot commented Jan 18, 2017

ixdy commented Jan 18, 2017

k8s-github-robot commented Jan 18, 2017

k8s-github-robot commented Jan 18, 2017

Build release tars using bazel #39898

Build release tars using bazel #39898

Conversation

ixdy commented Jan 13, 2017

k8s-reviewable commented Jan 13, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

ixdy commented Jan 14, 2017

ixdy commented Jan 14, 2017

k8s-ci-robot commented Jan 14, 2017

spxtr commented Jan 14, 2017

spxtr Jan 17, 2017

Choose a reason for hiding this comment

ixdy Jan 18, 2017

Choose a reason for hiding this comment

spxtr Jan 18, 2017

Choose a reason for hiding this comment

spxtr Jan 17, 2017

Choose a reason for hiding this comment

spxtr commented Jan 17, 2017

spxtr commented Jan 17, 2017

mikedanese Jan 18, 2017

Choose a reason for hiding this comment

ixdy Jan 18, 2017

Choose a reason for hiding this comment

mikedanese commented Jan 18, 2017

mikedanese commented Jan 18, 2017

k8s-ci-robot commented Jan 18, 2017

k8s-ci-robot commented Jan 18, 2017

ixdy commented Jan 18, 2017

k8s-github-robot commented Jan 18, 2017

k8s-github-robot commented Jan 18, 2017