[kubeadm] specify an alternate location for all images and pre pull them #36759

bulletRush · 2016-11-14T17:45:16Z

What this PR does / why we need it: specify an alternate location for all images and pre pull them when kubeadm init

Which issue this PR fixes (optional, in fixes #<issue number>(, #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
[kubeadm] pre pull images and configurable pod implement

Special notes for your reviewer:

sample command for this pr:

./kubeadm init --image-prefix=barrettwu --use-kubernetes-version=v1.4.1 --pre-pull-images=true

and I will make a PR to kubernetes docs when this is merged to master.

Release note:

If you want to use kubeadm `image-prefix` flag, you may need to set kubelet `pod-infra-container-image` flag at the same time.

This change is

k8s-ci-robot · 2016-11-14T17:45:18Z

Can a kubernetes member verify that this patch is reasonable to test? If so, please reply with "@k8s-bot ok to test" on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands will still work. Regular contributors should join the org to skip this step.

If you have questions or suggestions related to this bot's behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

jbeda

This flag should be documented fully (both in the code and in our main documentation) or moved to an env variable so as not to pollute our help strings.

jbeda · 2016-11-15T22:09:22Z

cmd/kubeadm/app/cmd/init.go

@@ -155,6 +156,10 @@ func NewCmdInit(out io.Writer) *cobra.Command {
 		"Port for JWS discovery service to bind to",
 	)

+	cmd.PersistentFlags().StringVar(
+		&cfg.ImagePrefix, "image-prefix", gcrPrefix,
+		"use specific images instead of gcr",


This help string should be more descriptive. I'd do something like: Specify an alternate location for all images. Currently this includes: etcd, kube-discovery, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, kubedns, kube-dnsmasq, exechealthz, pause

Ideally that list would be generated and we'd have clear instructions on how to create a custom set of images. Or we'd have a way to override just some of them as necessary. As things stand, this is a pretty advanced move. Unless we are ready to document how to use this we should probably make it an env variable instead of a command line flag in order to not pollute our kubeadm --help.

A command line flag is more friendly for kubeadm users.
In fact, I have a more radical branch( pre pull images and configurable pod ) for personal dev and test.

jbeda · 2016-11-15T22:10:42Z

If this is a command line flag, it should probably have a release note.

bulletRush · 2016-11-17T01:19:35Z

need a pre flight check to notify user set kubelet pod-infra-image flag. will do this change tonight. you can review this commit tomorrow.

…hem.

jbeda

@bulletRush -- this looks difficult enough that I don't think it should be a command line flag. It isn't clear or easy to use this/these flags. We've been working hard to keep the number of flags minimal here.

The case for doing the pre-pulling of images (vs just letting the kubelet do it) isn't clear.

If we think that this is something that a large percentage of kubeadm users will be using (>30-50%) then I'm game. If not, we should probably leave it as a more obscure env variable. If/as we document this and it becomes more common we can promote it to a flag.

jbeda · 2016-11-21T19:23:41Z

cmd/kubeadm/app/master/discovery.go

 	"k8s.io/kubernetes/pkg/api"
 	"k8s.io/kubernetes/pkg/apis/extensions"
 	clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset"
 	certutil "k8s.io/kubernetes/pkg/util/cert"
 	"k8s.io/kubernetes/pkg/util/wait"
+	"time"


Why did time move down here?

It's seem like my ide or ./hack/update-gofmt.sh does that. I'm not sure and I will try to revert this.

jbeda · 2016-11-21T19:24:19Z

cmd/kubeadm/app/cmd/init.go

+		"Specify an alternate location for all images. Currently this includes: etcd, kube-discovery, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, kubedns, kube-dnsmasq, exechealthz, pause",
+	)
+
+	cmd.PersistentFlags().BoolVar(


It looks like this defaults to false. What is the scenario here?

It's DefaultImagePrefix (gcr.io/google_containers). see cmd/kubeadm/app/apis/kubeadm/v1alpha/defaults.go

jbeda · 2016-11-21T22:56:14Z

cmd/kubeadm/app/preflight/checks.go

+		return
+	}
+	warnings = []error{
+		fmt.Errorf("custom image prefix(%s) detected, please confirm whether you need to modify the kubelet `pod-infra-container-image` flag", ipc.imagePrefix),


What does this mean? How/what should be modified here? Without some sort of guide this is not actionable for users.

when start up a pod, kubelet will create a pod-infra-container first. Kubelet will download image from gcr.io/google_containers by default. If you specify a custom registry to download your container image, you may also need change where pod-infra-container image download from.

kubelet

--pod-infra-container-image string The image whose network/ipc namespaces containers in each pod will use. (default "gcr.io/google_containers/pause-amd64:3.0")

jbeda · 2016-11-21T22:59:53Z

@kubernetes/sig-cluster-lifecycle

mikedanese · 2016-11-22T03:03:39Z

This is docker specific. We need kubeadm to work with any container runtime that k8s supports.

bulletRush · 2016-11-22T07:08:26Z

@jbeda if you have a bad network or you are just in some special country like China, you can't download container image from gcr.io success, and you will wait for control plane to become ready forever.

when kubeadm is first released, most people complain they stop at "<master/apiclient> created API client, waiting for the control plane to become ready" and don't know what happened.

but if we have pre pulled images and still stop at 'waitting' step after a long time, then we will relize that there must be something else happened. For me, it's invalid hostname, and I have make a PR fixed that.

jbeda · 2016-11-22T18:30:34Z

@bulletRush

If we want to support "mirroring" these images elsewhere, we should think about this from the point of view of the total user experience and make sure that we make that feature usable.

First off -- I'm okay with moving the default image pull policy to IfNotPresent makes sense. It is technically a no-op as we are specifying a version and that is the default for that field. This will make it possible for people to pre-pull images. They can always rename them to the gcr.io name even if they didn't come from there. That should be enough to get folks with a spotty connection or without access going.

Second -- we should start with documenting this stuff. I suspect that we can get away with a combination of docker save and docker load to do this.

The tricky part, actually, is getting the list of images necessary. There are just too many of them. Perhaps the best way to go here would be to implement a new command (kubeadm list-images?) that lists out all of the images.

Then we tell people to run docker save $(kubeadm list-images) | gzip > kube-images.tar.gz to save up images someplace where they have access. Then copy that tar file to each machine and run docker load < kube-images.tar.gz on each machine in the cluster. This would solve the problem without introducing any docker specific stuff to kubeadm or new command line flags.

If we go ahead with kubeadm list-images we could have profiles there for all, node, master.

bulletRush · 2016-11-23T03:35:43Z

Seem like this is an unacceptable PR. please close this it and I'll keep this in my private branch.

luxas · 2016-11-23T17:50:34Z

I think we should document this for sure, but we have to be very cautious with what we merge into kubeadm, otherwise it will do everything but do the dishes for you :)

If you want/can, you could send a PR against kubernetes.github.io and we'll see where we like to put that information. Getting kubeadm smoothly running in China is important for sure 👍

Also, if you want to contribute the parts of this PR that set the imagePullPolicy to IfNotExists and the code that refactored the kube-discovery image path, feel free to open a new PR with those changes, because I think those are good changes.

Thanks

KeithTt · 2017-01-24T06:46:10Z

@jbeda hope this command will be implemented kubeadm list-images.

googlebot added the cla: yes label Nov 14, 2016

k8s-github-robot assigned jbeda Nov 14, 2016

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-label-needed labels Nov 14, 2016

jbeda suggested changes Nov 15, 2016

View reviewed changes

bulletRush force-pushed the feature/custom-images branch from 1e44a9a to 461fb3d Compare November 16, 2016 16:38

bulletRush changed the title ~~use specific images instead of gcr~~ [kubeadm] specify an alternate location for all images and pre pull them Nov 16, 2016

bulletRush force-pushed the feature/custom-images branch from 461fb3d to fb9b508 Compare November 16, 2016 16:44

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 16, 2016

[kubeadm] specify an alternate location for all images and pre pull t…

722bde5

…hem.

bulletRush force-pushed the feature/custom-images branch from fb9b508 to 722bde5 Compare November 17, 2016 04:39

jbeda suggested changes Nov 21, 2016

View reviewed changes

luxas closed this Nov 23, 2016

bulletRush deleted the feature/custom-images branch November 24, 2016 10:07

bulletRush restored the feature/custom-images branch November 24, 2016 10:07

bulletRush mentioned this pull request Nov 26, 2016

Document how to pre-pull images kubernetes/kubeadm#34

Closed

KeithTt mentioned this pull request Jan 27, 2017

how to find all the images when install k8s-1.5? #40336

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kubeadm] specify an alternate location for all images and pre pull them #36759

[kubeadm] specify an alternate location for all images and pre pull them #36759

bulletRush commented Nov 14, 2016 •

edited

k8s-ci-robot commented Nov 14, 2016

jbeda left a comment

jbeda Nov 15, 2016

bulletRush Nov 16, 2016

jbeda commented Nov 15, 2016

bulletRush commented Nov 17, 2016

jbeda left a comment

jbeda Nov 21, 2016

bulletRush Nov 22, 2016

jbeda Nov 21, 2016

bulletRush Nov 22, 2016

jbeda Nov 21, 2016

bulletRush Nov 22, 2016

bulletRush Nov 22, 2016

jbeda commented Nov 21, 2016

mikedanese commented Nov 22, 2016

bulletRush commented Nov 22, 2016 •

edited

jbeda commented Nov 22, 2016

bulletRush commented Nov 23, 2016 •

edited

luxas commented Nov 23, 2016 •

edited

KeithTt commented Jan 24, 2017

[kubeadm] specify an alternate location for all images and pre pull them #36759

[kubeadm] specify an alternate location for all images and pre pull them #36759

Conversation

bulletRush commented Nov 14, 2016 • edited

k8s-ci-robot commented Nov 14, 2016

jbeda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbeda commented Nov 15, 2016

bulletRush commented Nov 17, 2016

jbeda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbeda commented Nov 21, 2016

mikedanese commented Nov 22, 2016

bulletRush commented Nov 22, 2016 • edited

jbeda commented Nov 22, 2016

bulletRush commented Nov 23, 2016 • edited

luxas commented Nov 23, 2016 • edited

KeithTt commented Jan 24, 2017

bulletRush commented Nov 14, 2016 •

edited

bulletRush commented Nov 22, 2016 •

edited

bulletRush commented Nov 23, 2016 •

edited

luxas commented Nov 23, 2016 •

edited