Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Umbrella] Implement improvements to protect CI and releases from changes to release tooling #816

Open
justaugustus opened this issue Jul 4, 2019 · 14 comments

Comments

@justaugustus
Copy link
Member

commented Jul 4, 2019

(Also sent to k-dev, SIG Release MLs, and SIG Testing: https://groups.google.com/d/topic/kubernetes-dev/OuLMzqZkdtw/discussion)

Following an attempt to improve the semantics of the release tooling via shellcheck (#726), we found that we were unable to stage releases.

Multiple fixes were merged in an attempt to bring us to a usable state.

An unintended and unexpected side effect of this was a cascading failure of multiple release-blocking jobs. A few for example:

Ultimately, it was decided that the right course of action was to revert back to a known good state in the repo (#814) to stop the bleeding.

This implies that, in our current state, it is inadvisable to make any changes to the tooling in this repo.

As such, I'm advising the following course of action (h/t to @nikhita, @liggitt, and @BenTheElder for being a sounding board):

  • (kubernetes/test-infra#13328) Add a blockade for files that have the potential to impact releasing and CI signal
    (this will require repo admins to explicitly approve and override the blockade to merge changes to critical tooling)
  • (in progress below) Examine and document exactly why these release-blocking jobs failed
    (they are using something in k/release; we need to figure out what those somethings are)
  • Tag the repo after executing a successful release of Kubernetes
    (this locks in a known good state of k/release that doesn't need to be master)
  • (kubernetes/test-infra#13340) Setup a periodic/presubmit job that can emulate one of the existing jobs that broke recently
  • Refactor release tooling/jobs that depend on tooling to accept pulling a tag of k/release instead of master

At this point, we will have gotten to a place where we can safely make changes to k/release without impacting CI. We will then:

For longer term goals, we should seek to:

  • Write go tooling (and tests!) to replace the shell libraries (lib/{common,gitlib,releaselib}) and call these new tools in the existing release tooling
    (this allows us to get some immediate benefit of a more robust language w/o having to completely refactor)
  • Full refactor of existing tools (shell --> go)

(Some historical references: kubernetes/kubernetes#28922, kubernetes/kubernetes#16529, kubernetes/kubernetes#15560, kubernetes/kubernetes#8686)

Please take this an initial assessment of the situation and feel free to provide feedback. :)

/assign
/milestone v1.16
/area release-eng
/sig release
/kind bug
/priority critical-urgent

cc @kubernetes/sig-release-admins @kubernetes/release-engineering @dims @neolit123 @pswica

@hoegaarden

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

FWIW, what you can do today is set RELEASE_TOOL_REPO and/or RELEASE_TOOL_BRANCH when submitting the cloud builder job via gcbmgr, e.g.:

RELEASE_TOOL_REPO='https://github.com/other-org/other-release-repo'  RELEASE_TOOL_BRANCH='feature-foo' \
  ./gcbmgr stage release-1.12 --official --build-at-head

This & mock releases should be safe enough and a way forward to start backfilling tests, refactoring, ...

Just to be clear: I am not advocating to keep using this and a mock stage & release as "the test". We need way faster feedback. But we can start with that.

@neolit123

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

Examine and document exactly why these release-blocking jobs failed
(they are using something in k/release; we need to figure out what those somethings are)

on the side of kubeadm release-master-informing jobs the failures stared happening because the kubeadm jobs could not find the CI artifacts they needs such as:

https://storage.googleapis.com/kubernetes-release-dev/ci/<CI-TAG>/bin/linux/amd64/kubelet
https://storage.googleapis.com/kubernetes-release-dev/ci/<CI-TAG>/bin/linux/amd64/kubeadm
...

examining the bucket:

gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/extra/
# ^ the bin directory is missing

as @dims outlined, the prow job ci-kubernetes-build even if passing/green, had errors:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-build/1146386381002510337/

W0703 12:54:56.715] tar: Substituting `.' for empty member name
W0703 12:54:56.715] tar: : Cannot stat: No such file or directory
W0703 12:54:56.715] tar: Exiting with failure status due to previous errors

this is the first kubeadm job failure we saw:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-master-on-stable/1145990882009288707

i couldn't not map the time to a problematic commit in k/release.

@neolit123

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

Setup a presubmit job that can emulate one of the existing jobs that broke recently

i think this is a great idea and such an e2e can compensate for the lack of unit tests.

the problems i saw - e.g.

gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/

indicate issues in the staging process.
so such an e2e has to stage somewhere and verify if all the artifacts are up.

@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 4, 2019

@neolit123 -- that's super useful; thanks for the data!

FWIW, what you can do today is set RELEASE_TOOL_REPO and/or RELEASE_TOOL_BRANCH when submitting the cloud builder job via gcbmgr, e.g.:

@hoegaarden -- I didn't know you could do that. Do we have it documented somewhere?

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

There is one such presubmit now (pull-release-cluster-up) but that doesn't cover everything used by EG the kubeadm job.

@dims

This comment has been minimized.

Copy link
Member

commented Jul 5, 2019

@hoegaarden @justaugustus that gcbmgr script needs access to release-test-dev bucket too

dims@rex:~/go/src/k8s.io/release$ RELEASE_TOOL_REPO='https://github.com/kubernetes/release' RELEASE_TOOL_BRANCH=master ./gcbmgr stage master --build-at-head
gcbmgr: BEGIN main on rex Fri Jul  5 02:17:21 UTC 2019

Checking required system packages: OK
Checking/setting cloud tools: OK

FAILED: Job was not submitted.  Details:
ERROR: (gcloud.builds.submit) User [davanum@gmail.com] does not have permission to access project [kubernetes-release-test] (or it may not exist): Caller
does not have required permission to use project project:kubernetes-release-test. Grant the caller the Owner or Editor role, or a custom role with the
serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=project:kubernetes-release-test
and then retry (propagation of new permission may take a few minutes).  - '@type': type.googleapis.com/google.rpc.Help
  links: - description: Google developers console
    url: https://console.developers.google.com
- '@type': type.googleapis.com/google.rpc.Help
  links: - description: Google developer console IAM admin
    url: https://console.developers.google.com/iam-admin/iam/project?project=project:kubernetes-release-test
@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2019

@dims -- Could you clarify what you mean by access to release-test-dev?

That snippet only shows that you don't have access to the kubernetes-release-test project on GCP:

augustus@auggievmw01:~/go/src/k8s.io/release$ RELEASE_TOOL_REPO='https://github.com/kubernetes/release' RELEASE_TOOL_BRANCH=master ./gcbmgr stage master --build-at-head
gcbmgr: BEGIN main on auggievmw01 Thu Jul  4 22:45:51 EDT 2019

Checking required system packages: OK
Checking /home/augustus/go/src/k8s.io/release state: OK
Checking/setting cloud tools: OK

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
STAGE 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c submitted successfully.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

To view last build:
$ gcbmgr tail

To view this specific build:
$ gcbmgr tail 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c
-OR-
$ /usr/bin/gcloud --project kubernetes-release-test builds log --stream 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c
-OR-
https://console.cloud.google.com/gcr/builds/92f24742-f1fe-4ca8-b33d-ffbca2e1c43c?project=648026197307

gcbmgr: DONE main on auggievmw01 Thu Jul  4 22:45:55 EDT 2019 in 4s
@hoegaarden

This comment has been minimized.

Copy link
Member

commented Jul 5, 2019

@hoegaarden -- I didn't know you could do that. Do we have it documented somewhere?

It is not documented, it came in somewhat recently with 636269f.

@hoegaarden

This comment has been minimized.

Copy link
Member

commented Jul 5, 2019

@hoegaarden @justaugustus that gcbmgr script needs access to release-test-dev bucket too

You still need access to the kubernetes-release-test -- that is a known issue and yet to be solved. Some parts, IIRC, can be reconfigured to use a different GCP project (even maybe different buckets for individual things), but all in all we are not setup right now to use different projects.

@dims

This comment has been minimized.

Copy link
Member

commented Jul 5, 2019

@justaugustus sorry i typo-ed it :) like @hoegaarden and you pointed out it's access to kubernetes-release-test

@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2019

@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2019

To make sure we capture the various failure modes here, I did a little digging, layered with the diagnoses we've already received from a few on this thread.

Download failures

Extract test failure

W0702 08:57:51.792] 2019/07/02 08:57:51 extract_k8s.go:288: U=https://storage.googleapis.com/kubernetes-release-dev/ci R=v1.16.0-alpha.0.1787+b3c6e2189576c9 get-kube.sh
W0702 08:57:51.792] 2019/07/02 08:57:51 process.go:153: Running: /workspace/get-kube.sh
W0702 08:57:51.813]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
W0702 08:57:51.814]                                  Dload  Upload   Total   Spent    Left  Speed
W0702 08:57:51.926] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
W0702 08:57:51.928] curl: (22) The requested URL returned error: 404 
W0702 08:57:51.931] 2019/07/02 08:57:51 process.go:155: Step '/workspace/get-kube.sh' finished in 139.476343ms
W0702 08:57:51.932] 2019/07/02 08:57:51 extract_k8s.go:298: U=https://storage.googleapis.com/kubernetes-release-dev/ci R=v1.16.0-alpha.0.1787+b3c6e2189576c9 get-kube.sh failed: error during /workspace/get-kube.sh: exit status 22
W0702 08:57:51.932] 2019/07/02 08:57:51 process.go:153: Running: /workspace/get-kube.sh
W0702 08:57:51.951]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
W0702 08:57:51.951]                                  Dload  Upload   Total   Spent    Left  Speed
W0702 08:57:52.024] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
W0702 08:57:52.026] curl: (22) The requested URL returned error: 404 
W0702 08:57:52.029] 2019/07/02 08:57:52 process.go:155: Step '/workspace/get-kube.sh' finished in 96.684177ms
W0702 08:57:52.029] 2019/07/02 08:57:52 extract_k8s.go:298: U=https://storage.googleapis.com/kubernetes-release-dev/ci R=v1.16.0-alpha.0.1787+b3c6e2189576c9 get-kube.sh failed: error during /workspace/get-kube.sh: exit status 22
I0702 08:57:52.130] Downloading kubernetes release v1.16.0-alpha.0.1787+b3c6e2189576c9
I0702 08:57:52.130]   from https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1787+b3c6e2189576c9/kubernetes.tar.gz
I0702 08:57:52.130]   to /go/src/k8s.io/kubernetes/kubernetes.tar.gz
I0702 08:57:52.131] Downloading kubernetes release v1.16.0-alpha.0.1787+b3c6e2189576c9
I0702 08:57:52.131]   from https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1787+b3c6e2189576c9/kubernetes.tar.gz
I0702 08:57:52.131]   to /go/src/k8s.io/kubernetes/kubernetes.tar.gz
W0702 08:57:53.030] 2019/07/02 08:57:53 process.go:153: Running: /workspace/get-kube.sh
W0702 08:57:53.049]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
W0702 08:57:53.050]                                  Dload  Upload   Total   Spent    Left  Speed
W0702 08:57:53.124] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
W0702 08:57:53.126] curl: (22) The requested URL returned error: 404 
W0702 08:57:53.129] 2019/07/02 08:57:53 process.go:155: Step '/workspace/get-kube.sh' finished in 99.041748ms
W0702 08:57:53.130] 2019/07/02 08:57:53 process.go:96: Saved XML output to /workspace/_artifacts/junit_runner.xml.
W0702 08:57:53.130] 2019/07/02 08:57:53 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
I0702 08:57:53.231] Downloading kubernetes release v1.16.0-alpha.0.1787+b3c6e2189576c9
I0702 08:57:53.231]   from https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1787+b3c6e2189576c9/kubernetes.tar.gz
I0702 08:57:53.232]   to /go/src/k8s.io/kubernetes/kubernetes.tar.gz
W0702 08:57:53.481] 2019/07/02 08:57:53 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 350.832651ms
W0702 08:57:58.265] 2019/07/02 08:57:58 main.go:316: Something went wrong: failed to acquire k8s binaries: U=https://storage.googleapis.com/kubernetes-release-dev/ci R=v1.16.0-alpha.0.1787+b3c6e2189576c9 get-kube.sh failed: error during /workspace/get-kube.sh: exit status 22
W0702 08:57:58.270] Traceback (most recent call last):
W0702 08:57:58.271]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 778, in <module>
W0702 08:57:58.271]     main(parse_args())
W0702 08:57:58.271]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 626, in main
W0702 08:57:58.272]     mode.start(runner_args)
W0702 08:57:58.272]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start
W0702 08:57:58.272]     check_env(env, self.command, *args)
W0702 08:57:58.272]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0702 08:57:58.272]     subprocess.check_call(cmd, env=env)
W0702 08:57:58.272]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W0702 08:57:58.272]     raise CalledProcessError(retcode, cmd)
W0702 08:57:58.273] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--provider=gce', '--cluster=e2e-big', '--gcp-network=e2e-big', '--check-leaked-resources', '--extract=ci/latest', '--gcp-node-image=gci', '--gcp-nodes=100', '--gcp-project-type=scalability-project', '--gcp-zone=us-east1-b', '--test-cmd=/go/src/k8s.io/perf-tests/run-e2e.sh', '--test-cmd-args=cluster-loader2', '--test-cmd-args=--nodes=100', '--test-cmd-args=--provider=gce', '--test-cmd-args=--report-dir=/workspace/_artifacts', '--test-cmd-args=--testconfig=testing/density/config.yaml', '--test-cmd-args=--testconfig=testing/load/config.yaml', '--test-cmd-args=--testoverrides=./testing/density/100_nodes/override.yaml', '--test-cmd-args=--testoverrides=./testing/prometheus/scrape-etcd.yaml', '--test-cmd-args=--testoverrides=./testing/prometheus/scrape-kube-proxy.yaml', '--test-cmd-name=ClusterLoaderV2', '--timeout=120m', '--logexporter-gcs-path=gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1145979554884489221/artifacts')' returned non-zero exit status 1
E0702 08:57:58.277] Command failed
I0702 08:57:58.278] process 326 exited with code 1 after 0.2m
E0702 08:57:58.278] FAIL: ci-kubernetes-e2e-gci-gce-scalability
I0702 08:57:58.278] Call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json
W0702 08:57:58.785] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com]
I0702 08:57:58.844] process 430 exited with code 0 after 0.0m
I0702 08:57:58.844] Call:  gcloud config get-value account
I0702 08:57:59.163] process 442 exited with code 0 after 0.0m
I0702 08:57:59.163] Will upload results to gs://kubernetes-jenkins/logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com
I0702 08:57:59.163] Upload result and artifacts...
I0702 08:57:59.164] Gubernator results at https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1145979554884489221
I0702 08:57:59.164] Call:  gsutil ls gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1145979554884489221/artifacts
W0702 08:58:00.227] CommandException: One or more URLs matched no objects.
E0702 08:58:00.356] Command failed

ref:

As an aside, all of these jobs should be migrated to podutils.

In short, ./get-kube.sh is failing to get https://storage.googleapis.com/kubernetes-release-dev/ci/<version>/kubernetes.tar.gz

We'll come back to this failure.

kubeadm/kinder failures

# task-01-add-kubernetes-versions
kinder build node-image-variant --base-image=kindest/base:v20190403-1ebf15f --image=kindest/node:test --with-init-artifacts=v1.15.0 --with-upgrade-artifacts=v1.16.0-alpha.0.1789+a807cb625b08c7 --loglevel=debug
 time="09:36:48" level=info msg="Altering node image in: /tmp/kinder-alter-image547261655"
time="09:36:48" level=info msg="Starting populate bits ..."
time="09:36:48" level=info msg="version file created"
time="09:36:48" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubelet\n"
time="09:36:49" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl\n"
time="09:36:50" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubeadm\n"
time="09:36:51" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kube-apiserver.tar\n"
time="09:36:53" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kube-controller-manager.tar\n"
time="09:36:55" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kube-scheduler.tar\n"
time="09:36:56" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kube-proxy.tar\n"
time="09:36:56" level=info msg="Downloaded files saved into /tmp/kinder-alter-image547261655/bits/init"
time="09:36:56" level=info msg="version file created"
time="09:36:56" level=info msg="Downloading https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/bin/linux/amd64/kubelet\n"
Error: error altering node image: failed to copy alter bits: failed to copy https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/bin/linux/amd64/kubelet to /tmp/kinder-alter-image547261655/bits/upgrade/v1.16.0-alpha.0.1789+a807cb625b08c7/kubelet: error getting reader for https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/bin/linux/amd64/kubelet: HTTP GET https://storage.googleapis.com/kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/bin/linux/amd64/kubelet failed: 404 Not Found
 exit status 1
 # task-02-create-cluster
kinder create cluster --name=kinder-upgrade --image=kindest/node:test --control-plane-nodes=3 --worker-nodes=2
  skipping because a predecessor task failed
 # task-03-init
kinder do kubeadm-init --name=kinder-upgrade --automatic-copy-certs
  skipping because a predecessor task failed
 # task-04-join
kinder do kubeadm-join --name=kinder-upgrade --automatic-copy-certs
  skipping because a predecessor task failed
 # task-05-cluster-info-before
kinder do cluster-info --name=kinder-upgrade
  skipping because a predecessor task failed
 # task-06-upgrade
kinder do kubeadm-upgrade --upgrade-version=v1.16.0-alpha.0.1789+a807cb625b08c7 --name=kinder-upgrade
  skipping because a predecessor task failed
 # task-07-e2e-kubeadm-after
kinder test e2e-kubeadm --test-flags=--report-dir=/logs/artifacts --report-prefix=e2e-kubeadm-after-upgrade --name=kinder-upgrade
  skipping because a predecessor task failed
 # task-08-cluster-info-after
kinder do cluster-info --name=kinder-upgrade
  skipping because a predecessor task failed
 # task-09-e2e-after
kinder test e2e --test-flags=--report-dir=/logs/artifacts --report-prefix=e2e-after-upgrade --parallel --name=kinder-upgrade
  skipping because a predecessor task failed
 # task-10-get-logs
kinder export logs --loglevel=debug --name=kinder-upgrade /logs/artifacts
 time="09:36:57" level=debug msg="Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\\t{{.Label \"io.k8s.sigs.kind.cluster\"}}]"
Error: unknown cluster "kinder-upgrade"
 completed!
 # task-11-reset
kinder do kubeadm-reset --name=kinder-upgrade
 Error: a cluster with the name "kinder-upgrade" does not exists
 exit status 1
 # task-12-delete
kinder delete cluster --name=kinder-upgrade
 Error: unknown cluster "kinder-upgrade"
 exit status 1
 Ran 5 of 13 tasks in 0.000 seconds
FAIL! -- 2 tasks Passed | 3 Failed | 8 Skipped

ref:

Failing green (ci-kubernetes-*)

This is the insiduous one, and actually the cause of the symptoms above.

I'll only focus on ci-kubernetes-build here.

There appear to be two types of successful builds:

Let's examine a failed build (first assumed failure)...

Nothing in the job log looks particularly eventful; in fact, it looks identical to the log of a build that happens after a package push.

But, let's take a look at the GCS bucket:

root:

$ gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-client-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-client-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-client-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-client-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-manifests.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-manifests.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-manifests.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-manifests.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-node-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-node-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-node-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-node-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-server-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-server-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-server-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-server-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-src.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-src.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-src.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-src.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-portable.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-portable.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-portable.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test-portable.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes-test.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/kubernetes.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/bin/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/extra/

bin:

$ gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/bin/linux/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1773+f7f1b2d5f74ebd/bin/linux/amd64/

Comparing to a GCS bucket after a successful run:

root:

$ gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-386.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-386.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-386.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-386.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-darwin-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-386.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-386.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-386.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-386.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-arm64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-ppc64le.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-ppc64le.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-ppc64le.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-ppc64le.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-s390x.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-s390x.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-s390x.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-linux-s390x.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-386.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-386.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-386.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-386.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-client-windows-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-manifests.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-manifests.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-manifests.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-manifests.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-arm64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-ppc64le.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-ppc64le.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-ppc64le.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-ppc64le.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-s390x.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-s390x.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-s390x.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-linux-s390x.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-windows-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-windows-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-windows-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-node-windows-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-arm64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-ppc64le.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-ppc64le.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-ppc64le.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-ppc64le.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-s390x.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-s390x.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-s390x.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-server-linux-s390x.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-src.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-src.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-src.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-src.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-darwin-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-darwin-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-darwin-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-darwin-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-arm64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-ppc64le.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-ppc64le.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-ppc64le.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-ppc64le.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-s390x.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-s390x.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-s390x.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-linux-s390x.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-portable.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-portable.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-portable.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-portable.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-windows-amd64.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-windows-amd64.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-windows-amd64.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test-windows-amd64.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes-test.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes.tar.gz
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes.tar.gz.md5
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes.tar.gz.sha1
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/kubernetes.tar.gz.sha512
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/extra/

bin:

$ gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/386/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/amd64/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/arm/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/arm64/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/ppc64le/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1771+38e8b8a9f65e66/bin/linux/s390x/

That's odd; we're missing stuff.
Namely, binaries and tars for multiple arches:

$ diff bad good
4a5,16
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-386.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-386.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-386.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-386.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-amd64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-amd64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-amd64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-darwin-amd64.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-386.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-386.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-386.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-386.tar.gz.sha512
8a21,44
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-arm64.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-ppc64le.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-ppc64le.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-ppc64le.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-ppc64le.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-s390x.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-s390x.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-s390x.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-linux-s390x.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-386.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-386.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-386.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-386.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-amd64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-amd64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-amd64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-client-windows-amd64.tar.gz.sha512
16a53,72
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-arm64.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-ppc64le.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-ppc64le.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-ppc64le.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-ppc64le.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-s390x.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-s390x.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-s390x.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-linux-s390x.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-windows-amd64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-windows-amd64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-windows-amd64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-node-windows-amd64.tar.gz.sha512
20a77,92
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-arm64.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-ppc64le.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-ppc64le.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-ppc64le.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-ppc64le.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-s390x.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-s390x.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-s390x.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-server-linux-s390x.tar.gz.sha512
24a97,100
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-darwin-amd64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-darwin-amd64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-darwin-amd64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-darwin-amd64.tar.gz.sha512
28a105,120
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-arm64.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-ppc64le.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-ppc64le.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-ppc64le.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-ppc64le.tar.gz.sha512
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-s390x.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-s390x.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-s390x.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-linux-s390x.tar.gz.sha512
32a125,128
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-windows-amd64.tar.gz
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-windows-amd64.tar.gz.md5
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-windows-amd64.tar.gz.sha1
> gs://kubernetes-release-dev/ci/<version>/kubernetes-test-windows-amd64.tar.gz.sha512
48a145
> gs://kubernetes-release-dev/ci/<version>/bin/linux/386/
49a147,150
> gs://kubernetes-release-dev/ci/<version>/bin/linux/arm/
> gs://kubernetes-release-dev/ci/<version>/bin/linux/arm64/
> gs://kubernetes-release-dev/ci/<version>/bin/linux/ppc64le/
> gs://kubernetes-release-dev/ci/<version>/bin/linux/s390x/

I've got more investigation to do, but I at least want to post this in the meantime since the comment is starting to get pretty long :)

@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

Before we dive in, it's helpful to understand what the ci-kubernetes-build actually does.

Here's its' job config:

periodics:
<snip>
- interval: 1h
  name: ci-kubernetes-build
  labels:
    preset-service-account: "true"
    preset-dind-enabled: "true"
  annotations:
    fork-per-release: "true"
    fork-per-release-replacements: "--extra-publish-file=k8s-master -> --extra-publish-file=k8s-beta"
    fork-per-release-generic-suffix: "true"
    testgrid-dashboards: sig-release-master-blocking
    testgrid-tab-name: build-master
    testgrid-alert-email: "kubernetes-release-team@googlegroups.com"
  spec:
    containers:
      - image: gcr.io/k8s-testimages/bootstrap:v20190703-1f4d616
        args:
          - --repo=k8s.io/kubernetes
          - --repo=k8s.io/release
          - --root=/go/src
          - --timeout=180
          - --scenario=kubernetes_build
          - --
          - --allow-dup
          - --extra-publish-file=k8s-master
          - --hyperkube
          - --registry=gcr.io/kubernetes-ci-images
        # docker-in-docker needs privileged mode
        securityContext:
          privileged: true
        resources:
          requests:
            cpu: 4
            memory: "8Gi"

Focusing on the containers block:

image

We use the bootstrap image for this test.

FROM debian:stretch
<snip>
RUN mkdir /workspace/scenarios
COPY ["./scenarios", "/workspace/scenarios"]

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

Looking at the entrypoint, we execute:

<snip>
/usr/local/bin/runner.sh \
    ./test-infra/jenkins/bootstrap.py \
        --job="${JOB_NAME}" \
        --service-account="${GOOGLE_APPLICATION_CREDENTIALS}" \
        --upload='gs://kubernetes-jenkins/logs' \
        "$@"

We now know that a script called bootstrap.py is run.

args

Bootstrap takes the following args:

          - --repo=k8s.io/kubernetes
          - --repo=k8s.io/release
          - --root=/go/src
          - --timeout=180
          - --scenario=kubernetes_build
          - --
          - --allow-dup
          - --extra-publish-file=k8s-master
          - --hyperkube
          - --registry=gcr.io/kubernetes-ci-images

Passing the second set into a scenario (python script which executes a specific test scenario) called kubernetes_build.py.

def main(args):
    <snip>
    push_build_args = ['--nomock', '--verbose', '--ci']
    <snip>
    check('make', 'clean')
    if args.fast:
        check('make', 'quick-release')
    else:
        check('make', 'release')
    check(args.push_build_script, *push_build_args)

So ultimately, kubernetes_build.py runs:

  • make clean (from kubernetes/kubernetes)
  • make quick-release || make release (from kubernetes/kubernetes)
  • push-build.sh (from kubernetes/release)

Let's now take a look at a failure when ci-kubernetes-build has to actually push a build...

In this instance, the command executed was:

W0702 00:35:33.049] Run: ('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--release-kind=kubernetes', '--docker-registry=gcr.io/kubernetes-ci-images', '--extra-publish-file=k8s-master', '--allow-dup')

In a successful CI build, we expect to see the following sections run:

  • push-build.sh
  • push-build.sh::main()
  • push-build.sh::release::gcs::check_release_bucket()
  • push-build.sh::release::gcs::locally_stage_release_artifacts()
  • push-build.sh::release::gcs::stage_and_hash()
  • push-build.sh::release::gcs::push_release_artifacts()
  • (MISSING) push-build.sh::release::docker::release()
  • push-build.sh::release::gcs::publish_version()
  • push-build.sh::release::gcs::publish()

Walking the individual failures:

1

W0702 09:02:40.500] /go/src/k8s.io/release/lib/gitlib.sh: line 137: export: `PROGSTEP[gitlib::github_acls]': not a valid identifier

Checking the diff between failure and reverted state (git diff cee9fe13fdb4507f0e54c6d6387c8f5d0e3fdbaf..aa5d55925f060febb18df86a8f5b4c815ed560d9 lib/releaselib.sh):

@@ -136,7 +132,9 @@ gitlib::github_api_token () {
 ##############################################################################
 # Checks github ACLs
 # returns 1 on failure
-PROGSTEP[gitlib::github_acls]="CHECK GITHUB AUTH"
+# Disable shellcheck for dynamically defined variable
+# shellcheck disable=SC2154
+export PROGSTEP[gitlib::github_acls]="CHECK GITHUB AUTH"
 gitlib::github_acls () {
 
   gitlib::github_api_token || return 1
@@ -161,7 +159,6 @@ gitlib::git_config_for_gcb () {

Explanation of PROGSTEP:

# Print PROGSTEPs as bolded headers within scripts.
# PROGSTEP is a globally defined dictionary (associative array) that can
# take a function name or integer as its key
# The function indexes the dictionary in the order the items are added (by
# calling the function) so that progress can be shown during script execution
# (1/4, 2/4...4/4)
# If a PROGSTEP dictionary is empty, common::stepheader() will just show the
# text passed in.

Fix

None required, post-revert

2

I0702 09:02:50.192] push-build.sh::release::gcs::push_release_artifacts(): /google-cloud-sdk/bin/gsutil -qm cp -rc /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.16.0-alpha.0.1787+b3c6e2189576c9/* gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1787+b3c6e2189576c9/
W0702 09:02:50.292] tar: *: Cannot stat: No such file or directory
W0702 09:02:50.293] tar: Exiting with failure status due to previous errors
W0702 09:02:50.293] tar: /go/src/k8s.io/kubernetes/_output/release-stage/client/-- darwin-386 darwin-amd64 linux-386 linux-amd64 linux-arm linux-arm64 linux-ppc64le linux-s390x windows-386 windows-amd64/kubernetes/client/bin: Cannot open: No such file or directory
W0702 09:02:50.294] tar: Error is not recoverable: exiting now
W0702 09:02:50.294] tar: This does not look like a tar archive
W0702 09:02:50.294] tar: Exiting with failure status due to previous errors
I0702 09:02:52.032] OK

Checking the diff between failure and reverted state (git diff cee9fe13fdb4507f0e54c6d6387c8f5d0e3fdbaf..aa5d55925f060febb18df86a8f5b4c815ed560d9 lib/releaselib.sh):

@@ -597,12 +611,12 @@ release::gcs::locally_stage_release_artifacts() {
 
   logecho "Locally stage release artifacts..."
 
-  logrun rm -rf $gcs_stage || return 1
-  logrun mkdir -p $gcs_stage || return 1
+  logrun rm -rf "$gcs_stage" || return 1
+  logrun mkdir -p "$gcs_stage" || return 1
 
   # Stage everything in release directory
   logecho "- Staging locally to ${gcs_stage##$build_output/}..."
-  release::gcs::stage_and_hash $gcs_stage $release_tars/* . || return 1
+  release::gcs::stage_and_hash "$gcs_stage" "$release_tars"/* . || return 1
 
   if [[ "$release_kind" == "kubernetes" ]]; then
     local gce_path=$release_stage/full/kubernetes/cluster/gce
@@ -621,20 +635,20 @@ release::gcs::locally_stage_release_artifacts() {
release::gcs::stage_and_hash "$gcs_stage" "$release_tars"/* . || return 1

causes the failure.

It probably should've been:

release::gcs::stage_and_hash "${gcs_stage}" "${release_tars}/*" . || return 1

So here, it's entirely possible that the tars were on the workspace during the CI build but they never made it to GCS.

Fix

  • Fail immediately
  • Add a check for expected GCS artifacts

3

I0702 09:02:58.585] Send docker containers from release-images to gcr.io/kubernetes-ci-images...
I0702 09:02:58.612] FAILED: malformed tag in /go/src/k8s.io/kubernetes/_output/release-images/-- amd64
I0702 09:02:58.612] arm arm64 ppc64le s390x/*.tar:

Checking the diff between failure and reverted state (git diff cee9fe13fdb4507f0e54c6d6387c8f5d0e3fdbaf..aa5d55925f060febb18df86a8f5b4c815ed560d9 lib/releaselib.sh):

@@ -931,14 +947,16 @@ release::docker::release () {
 
   logecho "Send docker containers from release-images to $push_registry..."
 
-  arches=($(cd "$release_images"; echo *))
-  for arch in ${arches[@]}; do
-    for tarfile in $release_images/$arch/*.tar; do
+  mapfile -t arches < <(
+  cd "$release_images" || return 1
+  find . -mindepth 1 -maxdepth 1 -type d | cut -c 3-)
+  for arch in "${arches[@]}"; do
+    for tarfile in "$release_images/$arch"/*.tar; do
       # There may be multiple tags; just get the first
-      orig_tag=$(tar xf $tarfile manifest.json -O  | jq -r '.[0].RepoTags[0]')
+      orig_tag=$(tar xf "$tarfile" manifest.json -O  | jq -r '.[0].RepoTags[0]')
       if [[ ! "$orig_tag" =~ ^.+/(.+):.+$ ]]; then
         logecho "$FAILED: malformed tag in $tarfile:"
-        logecho $orig_tag
+        logecho "$orig_tag"
         return 1
       fi
       binary=${BASH_REMATCH[1]}
@@ -947,14 +965,15 @@ release::docker::release () {

Seems to be another miss on globbing here:

for tarfile in "$release_images/$arch"/*.tar; do

It probably should've been:

for tarfile in "${release_images}/${arch}/*.tar"; do

Fix

  • Fail immediately

More to come!

justaugustus added a commit to justaugustus/test-infra that referenced this issue Jul 6, 2019

sig-release: Add ci-kubernetes-build-canary to canary release tooling
ci-kubernetes-build-canary builds and publishes artifacts to a canary GCS bucket
using the tooling on the master branch of kubernetes/release. The goal here is
to allow Release Engineering to improve the current toolset without impacting CI
for the entire project.

While this job should closely mirror the configuration of ci-kubernetes-build,
it should differ in a few ways:
 - runs on CI ref of kubernetes/kubernetes
 - runs on master branch of kubernetes/release
 - publishes artifacts to a different GCS bucket
 - alerts Release Managers instead of the Release Team

ref: kubernetes/release/issues/816

Signed-off-by: Stephen Augustus <saugustus@vmware.com>
@justaugustus

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

Adding a canary job to discuss here: kubernetes/test-infra#13340

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.