Python/Go binary to bulk replace images in kustomization.yaml files (was Port use_gcr_for_all_images.sh to work with kustomize)

[jlewi]

Port https://github.com/kubeflow/kubeflow/blob/master/scripts/gke/use_gcr_for_all_images.sh
to kustomize.

This script is used for switching the images to a private registry. We will need to update the script
to work with the new kustomize manifests.

[issue-label-bot]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.93. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

I might suggest rewriting this as a go program and baking it into kfctl.

It might be nice if the tool supported some flexible way to match and replace images; e.g. using regexes. This way we could easily replace all images hosted in quay.io with another registry.

This would go a long way to making it easy to mirror all the required images to a different registry and then pull the images from it.

[swiftdiaries]

/help-wanted

[swiftdiaries]

Yeah, we're slated for 0.7 but this hasn't been picked yet

[ashahba]

To backup your point @jlewi I think we should start with only supporting internal registries that have the same repo structure that we propose, otherwise users need to update image urls manually.
This is mainly to simplify the private registry support.

[jlewi]

If someone wants to pick this up; I think a great first step would just be to modify
https://github.com/kubeflow/kubeflow/blob/master/scripts/gke/use_gcr_for_all_images.sh

To replace the ks param commands with kustomize edit set commands.

If we wanted to be a bit more forward looking we could rewrite that bash script as a go program but still shell out to kustomize and hard code the list of images to replace.

[xaniasd]

I'll take a stab at this if noone else is doing that already!

[jlewi]

Thanks @xaniasd !

After #4222 is merged can we update the docs on kubeflow/website before we close this issue?

A follow on issue would be to provide tooling to mirror of the Kubeflow images to a different repository.

I think one way we might be able to do that in a generic way would be to spit out a Tekton workflow that would pull the images from one repo, retag them and push them. (I'm hoping that's doable with Tekton).

@xaniasd Any interest in tackling that next (after the docs?)?

[xaniasd]

@jlewi I can add the documentation (I'll look into the setup and make a PR), no worries. About the follow-up; sure thing. I'm not familiar with Tekton and how we want to use it in this case, if you could give some background information I can give it a try. That said, a simple way to do this would be to just implement it in kfctl and fork docker pull/push commands for each image. Then we don't assume a running tekton instance (?). Along the same lines, an even simpler option would be to use kfctl to just spit out a list of images and build a bash command to do the pulling and pushing.

[yanniszark]

@xaniasd I actually have a small PR ready for exactly this thing, spitting out a changelist in stdout.
The user can use it to mirror images with any tool they want.
Will make a PR tomorrow.

[jlewi]

There's a couple reasons why on GCP we'd like to leverage Google Cloud Build or possibly Tekton to replicate the images

1. The number of images for Kubeflow is large O(10)'s if not O(100')s so we want to do the retagging in parallel
2. Given the large number of images we'd like to do the image pulling/pushing in the cloud to avoid network transfer to lower machines
3. With private GKE and Secure VPC we want to be able to use service accounts to talk to GCR

We have a sample jsonnet workflow here.

kubeflow/scripts/gke/gcb_copy_images.jsonnet

Line 28 in 0a7155c

steps+: [

I think for GCP what we'd want is a command or separate go binary to traverse the kustomize directory and generate a YAML spec for the workflow to replicate all images.

I think this should be separate from the set-image-name command so we can build a replication workflow without modifying the kustomization.yaml files.

The code should look at kustomization.yaml to get the old image name and tag and the new image name and tag.

Generating a Tekton workflow would be a cloud neutral solution to this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlewi]

/lifecycle frozen

It looks like kustomize supports composition; see kubernetes-sigs/kustomize#171

So if we define a kustomize manifest that composes all of the individual applications in Kubeflow (i.e. lists each kubeflow application as a base); then we should just be able to override all the images by writing a kustomization.yaml file in that package that specifies the image we want to replace and the new image.

Related: Get rid of kustomize magic in kfctl kubeflow/manifests#774
Relevant thread: https://groups.google.com/forum/m/#!topic/kubeflow-discuss/uWp39i2nNuQ

[kunmingg]

kubeflow/kfctl#226

[jlewi]

@kunmingg what is the remaining work to close out this issue?

It looks like we need to update the documentation page; e.g https://www.kubeflow.org/docs/gke/private-clusters/#deploy-kubeflow-with-private-gke

to describe how to use the new functionality. Is there anything else that needs to be done?

[jlewi]

@kunmingg thoughts? Any more immediate work to be done?

[jlewi]

@kunmingg ping?