Precache container images

[mumoshu]

If we do it in custom AMIs, boot times of nodes will be shortened considerably.
For example, in my region ap-northeast-1, each worker node takes several minutes to download the hyperkube image required to run kubelet from quay.io.

[mumoshu]

rkt is considering to support on-disk sub ACI repos rkt/rkt#695

[mumoshu]

If we'd want to bake AMIs, ensure /etc/machine-id is removed
https://groups.google.com/forum/#!topic/coreos-user/_wmOxOfMsEY

[mumoshu]

A container linux doc says that users should use /usr/share/oem/cloud-config.yaml for customizing machine images but that doesn't apply in this case?

https://coreos.com/os/docs/latest/notes-for-distributors.html#image-customization

[redbaron]

I’ve been thinking about speeding up instance boot, but I was thinking more along the lines of republishing images into local ECR registry and then optionally retagging them into their original names. It looks easier to pull off and less intrusive than rebuilding AMIs

[cknowles]

If we did bake, I think we should go with Packer. Baking it should in theory be less prone to some of the intermittent startup issues we've had. Since those have mostly been fixed already, as @redbaron says just having the images in ECR may help.

[mumoshu]

@redbaron Sounds reasonable - Have you tried that yourself? I was too lazy to try myself 😉
I'm wondering if we could have some adhoc scripts added under the contrib/ of this project to help users retagging all the images and adding appropriate *Image settings to cluster.yaml.
Would it sound good to you?

[cheungpat]

I maintain a cluster in ap-southeast-1 and stack update randomly fails because cloudformation timed out waiting for the controller to send SUCCESS signal. It takes well over 10 minutes for a controller to pull images and PT15M is not enough.

Use PT20M solves the problem but it would be great if I can speed up the boot process by caching the Docker images.

I think caching Docker images in the AMI is a better approach because it is faster and there is one less Docker registry to maintain. Using ECR is easier but it requires cluster admin to create a repository for each image and the ECR has to keep up to date with upstream registries. We also need to run aws ecr get-login before each pull.

[redbaron]

@cheungpat kube-aws is perfectly capable of creating ECR registries for images it is going to republish, so from user perspective no additional steps would be needed

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten