Release 1.11.1 images not pushed to k8s.gcr.io

[foxish]

In the release invocation that succeeded, there is:

Step #1: Send docker containers from release-images to staging-k8s.gcr.io...
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-amd64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-arm:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-arm64:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-ppc64le:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-s390x:v1.11.1: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-s390x:v1.11.1: OK
Step #1: [2018-Jul-17 21:52:37 UTC] release::docker::release in 7m0s
Step #1: 
Step #1: Publish official pointer text files to kubernetes-release...
Step #1: Test v1.11.1 > release/stable.txt (published): OK
Step #1: * v1.11.1 > v1.11.0 (published), updating
Step #1: Making uploaded version file public: OK
Step #1: * Validating uploaded version file at https://dl.k8s.io/release/stable.txt: OK
Step #1: Test v1.11.1 > release/stable-1.txt (published): OK
Step #1: * v1.11.1 > v1.11.0 (published), updating
Step #1: Making uploaded version file public: OK
Step #1: * Validating uploaded version file at https://dl.k8s.io/release/stable-1.txt: OK
Step #1: Test v1.11.1 > release/stable-1.11.txt (published): OK
Step #1: * v1.11.1 > v1.11.0 (published), updating
Step #1: Making uploaded version file public: OK
Step #1: * Validating uploaded version file at https://dl.k8s.io/release/stable-1.11.txt: OK
Step #1: [2018-Jul-17 21:52:50 UTC] release::gcs::publish_version in 13s
Step #1: [2018-Jul-17 21:52:50 UTC] push_all_artifacts in 9m5s
Step #1: 
Step #1: ================================================================================
Step #1: PUSH BINARY RELEASE ARTIFACTS beta (12/15)
Step #1: ================================================================================
Step #1: 
Step #1: Bucket-to-bucket copy gs://kubernetes-release/stage/v1.11.1-beta.0.115+b1b29978270dc2/v1.11.2-beta.0/gcs-stage artifacts to gs://kubernetes-release/release/v1.11.2-beta.0: OK
Step #1: Copy staged kubernetes.tar.gz to /workspace/anago-v1.11.1/src/k8s.io/kubernetes/_output-v1.11.2-beta.0/gcs-stage/v1.11.2-beta.0: OK
Step #1: Copy staged docker images to /workspace/anago-v1.11.1/src/k8s.io/kubernetes/_output-v1.11.2-beta.0/release-images: OK
Step #1: [2018-Jul-17 21:54:42 UTC] copy_staged_from_gcs in 1m52s
Step #1: Send docker containers from release-images to staging-k8s.gcr.io...
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-amd64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-arm:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-arm64:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-ppc64le:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/cloud-controller-manager-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/hyperkube-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-aggregator-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-apiserver-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-controller-manager-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-proxy-s390x:v1.11.2-beta.0: OK
Step #1: Pushing staging-k8s.gcr.io/kube-scheduler-s390x:v1.11.2-beta.0: OK

However, at no point does it attempt to push to k8s.gcr.io.
Logs in https://console.cloud.google.com/gcr/builds/b79ba696-2a04-48b8-a92c-c5f9166925ba?project=kubernetes-release-test

cc @neolit123 @jpbetz @david-mcmahon @ixdy @listx

[foxish]

For now, someone with superpowers may need to re-push those images from staging into k8s.gcr.io.

[foxish]

The command seems to have picked up the registry correctly at invocation time. I see:

Step #1: ================================================================================
Step #1: SESSION VALUES
Step #1: ================================================================================
Step #1: 
Step #1: WORKDIR: /workspace/anago-v1.11.1
Step #1: TREE_ROOT: $WORKDIR/src/k8s.io/kubernetes
Step #1: JENKINS_BUILD_VERSION: v1.11.1-beta.0.115+b1b29978270dc2
Step #1: RELEASE_VERSION_PRIME: v1.11.1
Step #1: RELEASE_VERSION[official]: v1.11.1
Step #1: RELEASE_VERSION[beta]: v1.11.2-beta.0
Step #1: RELEASE_BRANCH: release-1.11
Step #1: GCRIO_PATH: k8s.gcr.io
Step #1: RELEASE_BUCKET: kubernetes-release
Step #1: BUCKET_TYPE: release
Step #1: CHANGELOG_FILE: CHANGELOG-1.11.md
Step #1: FLAGS_nomock: 1
Step #1: FLAGS_rc: 0
Step #1: FLAGS_official: 1
Step #1: LOGFILE: /workspace/tmp/anago.log

[neolit123]

/cc @roberthbailey

[BenTheElder]

I sent out a proposed fix internally on the Google side... 🤞

[Multiply]

Is there a way to pull images from staging so I can just rename them manually for now?

[dims]

@Multiply i believe they are here right now - http://gcsweb.k8s.io/gcs/kubernetes-release/stage/v1.11.1-beta.0.115+b1b29978270dc2/v1.11.1/release-images/amd64/

[d11wtq]

Awesome, thanks @dims. I believe this should work for now (I'm using arm... substitute accordingly):

for img in kube-apiserver kube-controller-manager kube-scheduler kube-proxy
do
  curl -sL http://gcsweb.k8s.io/gcs/kubernetes-release/stage/v1.11.1-beta.0.115+b1b29978270dc2/v1.11.1/release-images/arm/$img.tar | docker load
done

[BenTheElder]

Thanks @dims.
Also re-iterating my update from #sig-release slack:

Hi folks, the fix did not go through yet. I got a review at 1:14AM my time that it needed an additionally change 😞 Just made that (was asleep...), waiting for it to go in after that 🤞

[ixdy]

/assign @listx