Makefile: Push a single multi-arch image

[mrueg]

Matching patterns from https://github.com/kubernetes-sigs/metrics-server/blob/master/Makefile

What this PR does / why we need it:
@serathius @lilic This adopts some targets of the Makefile of metrics-server's project to support a multi-arch image via test-infra's image build jobs.

[mrueg]

Linking this issue here as well: #1089

[lilic]

Restarted the job, seems like e2e tests failed.

[lilic]

You are changing the make targets we use in our e2e tests, hence the failures.

These are the make targets we call in our e2e tests and need to continue to function correctly. https://github.com/kubernetes/kube-state-metrics/blob/master/tests/e2e.sh#L108-L114

[mrueg]

You are changing the make targets we use in our e2e tests, hence the failures.

These are the make targets we call in our e2e tests and need to continue to function correctly. https://github.com/kubernetes/kube-state-metrics/blob/master/tests/e2e.sh#L108-L114

Thanks for debugging. Good catch! Will look into what needs to be changed there

[mrueg]

@lilic adjusted e2e tests and extended them to support other architectures as well, also figured out that we don't need to build the binary twice in the e2e tests (one regular, one in container) anymore.

[paulfantom]

Looks good to me. I tested it by running REGISTRY=quay.io/paulfantom make push and using resulting images in multi-arch (arm, arm64, amd64) cluster in thaum-xyz/ankhmorpork#39. Everything seems to be working as expected.

[lilic]

Thanks for your PR 🎉

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lilic, mrueg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lilic]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lilic]

Sadly this caused the pipeline build to fail for us:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-kube-state-metrics-push-images/1289119421062713344

Step 6/11 : RUN make build-local
 ---> Running in 187624cf0fd0
make: command: Command not found
fatal: not a git repository (or any of the parent directories): .git
GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -ldflags "-s -w -X k8s.io/kube-state-metrics/pkg/version.Release=v1.9.7 -X k8s.io/kube-state-metrics/pkg/version.Commit= -X k8s.io/kube-state-metrics/pkg/version.BuildDate=2020-07-31T08:51:59Z" -o kube-state-metrics
Removing intermediate container 187624cf0fd0
 ---> 956e164cd568
Step 7/11 : FROM gcr.io/distroless/static:latest
latest: Pulling from distroless/static
Digest: sha256:c6d5981545ce1406d33e61434c61e9452dad93ecd8397c41e89036ef977a88f4
Status: Image is up to date for gcr.io/distroless/static:latest
 ---> e4d2a899b1bf
Step 8/11 : COPY --from=builder /go/src/k8s.io/kube-state-metrics/kube-state-metrics /
 ---> 6d27092e473f
Step 9/11 : USER nobody
 ---> Running in 1b2c7e17d2bd
Removing intermediate container 1b2c7e17d2bd
 ---> 432439a8219f
Step 10/11 : ENTRYPOINT ["/kube-state-metrics", "--port=8080", "--telemetry-port=8081"]
 ---> Running in b594a5de6e97
Removing intermediate container b594a5de6e97
 ---> 30b4cf90b8bc
Step 11/11 : EXPOSE 8080 8081
 ---> Running in 7f3761eec6e6
Removing intermediate container 7f3761eec6e6
 ---> 2d4295fa4415
Successfully built 2d4295fa4415
Successfully tagged gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-s390x:v1.9.7
docker push gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-s390x:v1.9.7
The push refers to repository [gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-s390x]
2f9157df7831: Preparing
0d1435bd79e4: Preparing
0d1435bd79e4: Layer already exists
2f9157df7831: Pushed
v1.9.7: digest: sha256:a996734244d5a9e4446c15e32fd0bfc7856368fd2dc86f80f5b95d56aa4f6084 size: 738
docker manifest create --amend gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-amd64:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm64:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-ppc64le:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-s390x:v1.9.7
docker manifest create is only supported on a Docker cli with experimental cli features enabled
make: *** [Makefile:106: push-multi-arch] Error 1
ERROR
ERROR: build step 0 "gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4" failed: step exited with non-zero status: 2

[lilic]

For push we must use the following as its how it is build in gcr:

push: .push-$(ARCH)
.push-$(ARCH): .container-$(ARCH)
	gcloud docker -- push $(MULTI_ARCH_IMG):$(TAG)
	gcloud docker -- push $(MULTI_ARCH_IMG):latest
ifeq ($(ARCH), amd64)
	gcloud docker -- push $(IMAGE):$(TAG)
	gcloud docker -- push $(IMAGE):latest

Are you willling to make the fixes for this? @mrueg

[mrueg]

Sure will fix.

 docker manifest create --amend gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-amd64:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm64:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-ppc64le:v1.9.7 gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-s390x:v1.9.7
docker manifest create is only supported on a Docker cli with experimental cli features enabled
make: *** [Makefile:106: push-multi-arch] Error 1

needs the experimental docker cli enabled.

[cablespaghetti]

@mrueg @lilic @paulfantom Just looking at this in relation to #1037. Will this work? gcr.io/distroless/static is not multi-arch and so I think the base-image might need changing to get this to work on other architectures.

[cablespaghetti]

Sorry ignore me. Whilst the base image isn't multi-arch it doesn't seem to actually contain anything architecture specific, so this approach does work for metrics-server and should work here. Sorry for the doubt...