Split up xbuild Prow job

[jberkhahn]

currently our prow xbuild job takes nearly an hour to run. It would be nice if we could split this job up into different jobs that built the images for different architectures that run separately. We should already have most of the cruft required for this in our Makefile, so it should mostly be a matter of writing the prow job yaml.

[mszostok]

/dibs

Hope that it also makes those builds more stable.

In the first iteration, we can split prow xbuild job to:

• user-broker-image & test-broker-image
• service-catalog-image
• healthcheck-image
• svcat-all

and after closing that issue #2658 and #2189 then we can even remove the user-broker-image job.

But maybe after that, we should consider if building each image e.g. service-catalog-image for 5 different architectures (amd64 arm arm64 ppc64le s390x) is really necessary? What is the purpose of doing that? Any thoughts @jberkhahn?

[MHBauer]

/assign @mszostok

Do we still have the #dibs in the docs somewhere? We should be using the kubernetes prow commands now.

[mszostok]

@MHBauer tbh I just followed the Jonathan approach from this issue #2189 (comment) :D

and about the #dibs it's still mentioned here:

• https://github.com/kubernetes-sigs/service-catalog/blob/492c01787080a7bc72dfb221007998e9c5abd773/CONTRIBUTING.md#L13-L17
• https://github.com/kubernetes-sigs/service-catalog/blob/492c01787080a7bc72dfb221007998e9c5abd773/docs/devguide.md#L17-L21

[MHBauer]

oh, yea, I forgot that people have to be org member for assignment.

Even so, that's the only thing, and it's helpful for us to use the tooling we have. It's easier to check while browsing, what's been assigned, and who to talk to if you want and update on progress or to take over.

[MHBauer]

if building each image e.g. service-catalog-image for 5 different architectures (amd64 arm arm64 ppc64le s390x) is really necessary?

I don't know about arm or arm64, but IBM requires amd64, ppc64le & s390x.

[mszostok]

could you point me, where you are using that?

I understand that the svcat binary should be built against different platforms because it's directly executed on the client host and each of us can have a different OS arch.

But what about the user-broker-image, test-broker-image, service-catalog-image, healthcheck-image? In production, they are executed as a docker container inside the Kubernetes cluster and they are shipped with docker image with the amd64 arch. (check that job: https://travis-ci.org/kubernetes-sigs/service-catalog/jobs/540981158)

Times

When we split prow xbuild job to:

• user-broker-image & test-broker-image
• service-catalog-image
• healthcheck-image
• svcat-all

and each of them will be built against 5 different arch then each build will take around 25minutes.
When we will execute that only against amd64 then it will take the only 5min.

Errors

The bottom line is that the amd and amd64 never failed with the exec format error.

So leaving only those build will also increase our builds stability.

[mszostok]

Discussed on the SIG meeting. The service catalog binary is executed directly on OS, instead of using the default docker image. Currently, we want to leave support for all of the architecture.

[mszostok]

Description

I've created PR to split the pull-service-catalog-xbuild job.

Instead of having one pipeline for building all images I split that by arch into:

• pull-build-all-images-for-amd64
• pull-build-all-images-for-arm
• pull-build-all-images-for-arm64
• pull-build-all-images-for-ppc64le
• pull-build-all-images-for-s390x
• pull-build-all-images-for-svcat (cli for darwin, linux, windows)

pros:

• reduce pipeline time from 1h to around 7min
• based on investigation Split up xbuild Prow job #2663 (comment), the most common random failures are connected with architecture so splitting by that type will help for restarting only failed builds

cons:

• have 5 additional pipelines

I've tested that solution by executing those build on our test-infra clusters, results:

• pull-build-all-images-for-svcat duration: 4m31s
• pull-build-all-images-for-amd64 duration: 4m24s
• pull-build-all-images-for-arm duration: 7m2s
• pull-build-all-images-for-arm64 duration: 5m27s
• pull-build-all-images-for-ppc64le duration: 6m49s
• pull-build-all-images-for-s390x duration: 5m36s

Sum duration: 43m55s

Based on the pros IMO is worth to have those additional 5 pipelines. What's more, they are split by arch so it's a fixed number of pipeline regardless of the number of domains (brokers, apiservers, healtchecks etc.)

Outdated solution

My first approach was to split build by domain into:

• brokers-build-image-all-arch (user-broker & test-broker)
• service-catalog-build-image-all-arch
• healthcheck-build-image-all-arch
• svcat-build-image-all-arch

pros:

• reduce time to merge from 1h to 25min
• have only 3 additional pipelines

cons:

• based on investigation Split up xbuild Prow job #2663 (comment), the most common failures are connected with architecture so splitting by domain does not help too much
• requires changes in makefile: mszostok/service-catalog@master...mszostok:change-mf

[mszostok]

Resolved by: kubernetes/test-infra#13855

(merged&tested)

[MHBauer]

wow, nice!