use docker buildx to build multi-arch image

[mainred]

Before this change, the image built through our Makefile docker-push/xxx is with the same arch as the host as no arch simulation is introduced, even though ARCH is specified, as shown below I used REGISTRY=qingchuanhao.azurecr.io ARCH=arm64 make docker-push/proxy-server to push an arm64 image to ACR.

➜  apiserver-network-proxy git:(qinhao/docker-buildx) ✗ docker manifest inspect qingchuanhao.azurecr.io/proxy-server-arm64:4e8b12d2cd77db7ae5acecb5b706d5ecb3bf93e3 --verbose
{
        "Ref": "qingchuanhao.azurecr.io/proxy-server-arm64:4e8b12d2cd77db7ae5acecb5b706d5ecb3bf93e3",
        "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:1b7c6cda30a9107b6f3663f14aec7a8d8345db0edfb2d5a27df0be625120639a",
                "size": 529,
                "platform": {
                        "architecture": "amd64",
                        "os": "linux"
                }
        },
        "SchemaV2Manifest": {
                "schemaVersion": 2,
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "config": {
                        "mediaType": "application/vnd.docker.container.image.v1+json",
                        "size": 1626,
                        "digest": "sha256:6c8176319e0266e592a7c196fc59452c4f911264a28de871a6e7f6595b03bfaf"
                },
                "layers": [
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 16844398,
                                "digest": "sha256:ac1c233a6102ecd2569ae1b62251b35c56aab32519e5cb6f85384230515c9374"
                        }
                ]
        }
}

After this change, the same command REGISTRY=qingchuanhao.azurecr.io ARCH=arm64 make docker-push/proxy-server return a arm64-based image

➜  apiserver-network-proxy git:(qinhao/docker-buildx) ✗ docker manifest inspect qingchuanhao.azurecr.io/proxy-server-arm64:4e8b12d2cd77db7ae5acecb5b706d5ecb3bf93e3 --verbose
{
        "Ref": "qingchuanhao.azurecr.io/proxy-server-arm64:4e8b12d2cd77db7ae5acecb5b706d5ecb3bf93e3",
        "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:50a9ac6271ab785d1bbe0dc4c90c691d8864882120042538816291986c386b9e",
                "size": 528,
                "platform": {
                        "architecture": "arm64",
                        "os": "linux"
                }
        },
        "SchemaV2Manifest": {
                "schemaVersion": 2,
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "config": {
                        "mediaType": "application/vnd.docker.container.image.v1+json",
                        "size": 795,
                        "digest": "sha256:553c6675d128f33660bd3a2d6e1c7af342e2253b7728954b2fed860f2cda618d"
                },
                "layers": [
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 16844403,
                                "digest": "sha256:5d361ee5f90647819936c375addaab5996ccfb7d17181e5454c947d25a62cec8"
                        }
                ]
        }
}

[k8s-ci-robot]

Hi @mainred. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cheftako]

/ok-to-test

[jkh52]

Make sure to fix pull-apiserver-network-proxy-docker-build-all.

/retest

[mainred]

The building failure is caused by timeout and potential resource limitation by docker buildx
Building non-amd64 images seem to be too time-consuming and overall time take by make docker-bulid-all is more than 2 hours, for example, building arm-based server binary takes 33 minutes

#14 [builder 10/10] RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -a -ldflags '-extldflags "-static"' -o proxy-server sigs.k8s.io/apiserver-network-proxy/cmd/server
#14 DONE 2027.8s

[Jefftree]

Ok I've verified that the images were indeed built incorrectly and this PR should fix it.

As for the docker-build-all presubmit test failure I think we have a couple options

• Keep the test as blocking but only build for amd64. We only officially publish the amd64 image anyway, I'm not sure if I've seen any cases where an amd64 build would succeed but fail for other architectures. If anything, we would actually need to run the images to test them fully
• Increase the timeout and maybe keep the test as a non-merge blocking job. This prevents us from having to wait 2 hrs for a merge, but also lowers the value of the test
• Separate the docker-build-all presubmit job into one for each arch so they run in parallel. arm64 would probably still be the bottleneck and increase our average build time to 30 mins+

Let me know your thoughts @cheftako @jdnurme @jkh52

[jkh52]

Ok I've verified that the images were indeed built incorrectly and this PR should fix it.

As for the docker-build-all presubmit test failure I think we have a couple options

• Keep the test as blocking but only build for amd64. We only officially publish the amd64 image anyway, I'm not sure if I've seen any cases where an amd64 build would succeed but fail for other architectures. If anything, we would actually need to run the images to test them fully
• Increase the timeout and maybe keep the test as a non-merge blocking job. This prevents us from having to wait 2 hrs for a merge, but also lowers the value of the test
• Separate the docker-build-all presubmit job into one for each arch so they run in parallel. arm64 would probably still be the bottleneck and increase our average build time to 30 mins+

Let me know your thoughts @cheftako @jdnurme @jkh52

I think it's wise to cover at least two cases as blocking (amd64 since it's the published ARCH plus one variant, perhaps arm64). I agree with running in parallel, in any case, if the savings are non-negligible.

[cheftako]

Can we try parallel and see if that's sufficient? Not sure a 30 minute build time is a huge issue given our current PR cadence.

[mainred]

testing each arch docker building before changing the test-infra configuration.

[mainred]

I tested each arch image building, the time consumed by arm is extremely high. We can build images in parallel, but arm image building taking more than 1 hour is too much and brings no failure

We can build arm64 and amd64 in presubmit job suggested by @jkh52 , while making other arch image building available in Makefile, I don't mind also adding other archs expect arm in presubmit job.

Time consumed for each arch:

• arm: 1h28m59s
• amd64: 2m13s
• arm64:16m2s
• ppc64le: 19m17s
• s390x: 14m14s

[mainred]

Created a PR kubernetes/test-infra#24369 in case we want to run theses buildings in parallel

[k8s-ci-robot]

@mainred: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-apiserver-network-proxy-docker-build-all	84bab2f	link	true	/test pull-apiserver-network-proxy-docker-build-all

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[Jefftree]

I tested each arch image building, the time consumed by arm is extremely high. We can build images in parallel, but arm image building taking more than 1 hour is too much and brings no failure

We can build arm64 and amd64 in presubmit job suggested by @jkh52 , while making other arch image building available in Makefile, I don't mind also adding other archs expect arm in presubmit job.

Time consumed for each arch:

• arm: 1h28m59s
• amd64: 2m13s
• arm64:16m2s
• ppc64le: 19m17s
• s390x: 14m14s

Thanks for gathering the data @mainred! LGTM assuming kubernetes/test-infra#24369 is merged.

[mainred]

@cheftako @Jefftree if you agree with this change. I need your help to merge kubernetes/test-infra#24369, then I will retrigger docker build check here and ping you guys to merge this PR when it's green.

[mainred]

/retest

[Jefftree]

/lgtm

[cheftako]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheftako, mainred

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cheftako]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment