Consider building with Cloud Build

[steren]

Today, we build locally in Cloud Shell and then push to GCR.

It is not clear if this is fastest than building remotely in Cloud Build, have you performed benchmarks?
It also seems that the push stage takes a lot of time in Cloud Shell, while it is likely faster in Cloud Build.

Maybe it's a good opportunity to implement both, collect metrics and A/B test.

Note that Cloud Build can be used for Docker builds, but also for buildpack based builds with a custom cloudbuild.yaml

[jamesward]

Yeah, this would be great. Also related to #14

[steren]

Seeing how slow builds are on Cloud Shell, I really wonder if they wouldn't be faster on Cloud Build

[ahmetb]

I don't think we've benchmarked this, but I have a gut feeling like images that require more CPU and/or network bandwidth (to pull/push the img) will most likely perform better on Cloud Build.

If someone does that, we can switch to Cloud Build, should be fairly easy. However, right now it's one fewer API we need to enable.

[jamesward]

There is a few factors to this to think about:

• What is the first time build duration compared to a subsequent one that can utilize a warm cache? Cloud Build caching is not great today. Cloud Shell doesn't do any persistence of the docker daemon today (which could really make subsequent builds faster).
• Could we make boosted the default for Cloud Run Button?
• Should we use a larger-than-default Cloud Build instance?

Now for some numbers for https://github.com/jamesward/kotlin-cloud-android with buildpacks:
Cloud Build:
time gcloud alpha builds submit --pack=image=gcr.io/jw-demo/kotlin-cloud-android = 4m52s

Non-Boosted Cloud Shell:
time pack build gcr.io/jw-demo/kotlin-cloud-android --builder gcr.io/buildpacks/builder:v1 = 17m55s

Boosted Cloud Shell:
time pack build gcr.io/jw-demo/kotlin-cloud-android --builder gcr.io/buildpacks/builder:v1 = 4m22s

Boosted Cloud Shell 2nd run:
time pack build gcr.io/jw-demo/kotlin-cloud-android --builder gcr.io/buildpacks/builder:v1 = 1m14s

Local machine:
time pack build kotlin-cloud-android --builder gcr.io/buildpacks/builder:v1 = 2m3s

Local machine 2nd run:
time pack build kotlin-cloud-android --builder gcr.io/buildpacks/builder:v1 = 0m21s

[ahmetb]

@jamesward Great insights. Thanks for measuring. I'd say most of the users probably won't use pack, so we should ideally measure docker build + docker push latencies without caching while keeping "Non-boosted cloud shell" constant.

[jamesward]

Here is another one with a Dockerfile: https://github.com/jamesward/hello-kotlin-ktor

Cloud Build:
time gcloud builds submit --tag gcr.io/jw-demo/hello-kotlin-ktor . = 2m41s

Non-Boosted Cloud Shell:
time docker build -t gcr.io/jw-demo/hello-kotlin-ktor . = 8m51s

Boosted Cloud Shell:
time docker build -t gcr.io/jw-demo/hello-kotlin-ktor . = 2m37s

[ahmetb]

Thanks for testing again. Makes sense. It's a consistent n1-standard-1 VM that the build runs in after all.

• Enabling API: This is probably fine, however the users who are "driving by" will end up with a new "service account" and a "storage bucket" (GCB creates to store logs and build contexts). Not super ideal.

• gcloud dependency: We've spent a good time trying to cut our dependency to gcloud so we can separate our fate from it. However, in this case reimplementing gcloud builds submit will be non-trivial:

  • tar/zip the build context
  • create a gcs bucket
  • upload the context archive to the bucket
  • create a build to point it to the gcs object
  • read operation status (see the file that streams the logs)
  • poll for operation status
  • if fails, download the logs from gcs bucket and print

Any other solution like asking Cloud Shell team to boost the shell for our image will likely result in longer loading times.

[jamesward]

Yeah, I'm kinda leaning towards trying to improve things with Cloud Shell, and work on #75 for helping the user setup Cloud Build as a CI/CD next step perhaps with Cloud Run's CI/CD stuff. I also realized that our Jib support is not easily portable to Cloud Build. I'll reach out to the Cloud Shell team to see what we can do.