Discuss how kubeadm should deploy critical addons

[mikedanese]

It's important for kubeadm to deploy critical addons because we want users to have a functioning cluster after the process completes. The current implementation is an new addon management solution which does not align with the larger goals for the addon manager.

cc @kubernetes/sig-cluster-lifecycle

[luxas]

Which addons except dns belongs to this group?
dashboard and heapster?

Should the process be to opt in (kubeadm init --addons=dashboard,heapster) or opt-out (kubeadm init --exclude-addon heapster) in your mind?

[bprashanth]

I think the problem is broader than just deployment. I want priority on pods (nice) that kicks in even when they get vertically autoscaled. If there're finite resources on nodes, I want them allocated to the most important pods, even if this means evicting lower priority pods. Does the deployment problem solve itself with such a system?

[mikedanese]

Which addons except dns belongs to this group?

kube-proxy, heapster, maybe dashboard

We should fix addon deployment in general and use that solution rather than inventing yet another.

[colemickens]

Sorry to rehash this since I've had this conversation a couple times in ephemeral Slack contexts... but what are the arguments against kube-cert-manager? It was suggested as an option for our use, but I hesitated because I knew there were reservations... but I don't recall them well enough to argue for/against it's use.

[luxas]

xref my proposal earlier: #23233 (comment)

[justinsb]

Check out channels, which is how I've avoided the problems of kube-addons: https://github.com/kubernetes/kops/tree/master/channels

It depends on your work on getting prune to work in kubectl apply @mikedanese (or will do!), but it also adds a few things for a better UX:

1. a meta-manifest for an addon, listing the various available versions (https://github.com/kubernetes/kops/blob/master/addons/kubernetes-dashboard/addon.yaml)
2. a when you install an addon, we record the version you have installed and the source (currently as an annotation on the namespace)
3. this means if you edit the addon, we won't immediately undo your changes (I'm looking at you, kube-addons), even if run in a cron-job
4. we also can prompt you to upgrade your addons when a new version comes out

I think we should aim to replace the kube-addons container in 1.5 with something like this. I was planning on working on it anyway - want to team up @mikedanese ? I think kubeadm should call into it, but there is much wider appetite for this (and for starting to build a repository of addons). Of course if it's all go code it can be a standalone tool and part of kubeadm.

[errordeveloper]

Current implementation is there only because there was not concrete proposal at the time. There are several ideas floating around, let's make sure we settle on something soon.

[mikedanese]

This issue was moved to kubernetes/kubeadm#62