addon-installer POC

[stealthybox]

#14
This patch proposes an API type and small library to install addons via exec'ing kubectl.

The API type addons.config.k8s.io/AddonInstallerConfiguration is used to list addons as
kustomize packages.

The intent is that installers like kops, kubeadm, eksctl, and etc. can make use of this library and API type.

Included is a commandline tool that reads this type from a configration file to apply these addons
to the currently configured cluster.

Care has been taken to support flag overrides of config values as well as signal passing.

Open work:

• plumb kubectl extraArgs into the type/config ?
• demonstrate unpacking a kustomize addon using docker and/or skopeo+umoci in a separate patch

usage

bin/installer --config demo/dupes.yaml --dry-run
bin/installer --config demo/v1alpha1.yaml

development

# fetch deps + regenerate all API's
make

# build just the binary from existing files
make only-build

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stealthybox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [stealthybox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stealthybox]

This code has been adapted from https://github.com/luxas/sample-config and re-licensed to APL2 with kind permission from @luxas.

[fabriziopandini]

/assign

[fabriziopandini]

@stealthybox
Thanks for this POC!
might be I'm missing something, but I think to a possible integration with kubeadm I can't figure out all the details e.g.

1. Are we expecting the user download addons/patch in advance?
2. How do we think to address the air-gapped scenario?
3. How kubeadm can pass configurations to the addon-installer (e.g. the expected kube-proxy version, the kube-proxy component config)
4. How upgrades are expected to work?

[fabriziopandini]

Additional questions:

• how kubeadm can pass the kubeconfig for the cluster that we are creating
• when dry running, kubeadm currently prints all the resources that will be generated, including also resources for addons. How can we do something similar with the installer?

[stealthybox]

:)

(***) == needs implementation

1. Are we expecting the user download addons/patch in advance?

Not normally. This is handled internally in kubectl by kustomize.

2. How do we think to address the air-gapped scenario?

kustomize supports git repos and local directories for transport.
The user can clone the official repos and copy them to the local filesystem and modify the AddonInstallerConfiguration to use them without a runtime network dependency.
We hope to add an OCI/docker image transport to kustomize which would allow mirroring similar the control-plane images (***).

3. How kubeadm can pass configurations to the addon-installer
   (e.g. the expected kube-proxy version, the kube-proxy component config)

kubeadm can have a default AddonInstallerConfigration.
This can be overridden by adding an AddonInstallerConfiguration to the list of objects in the kubeadm config.
If a user wants to change what's installed, they can omit packages from the addons list or provide their own versions. kustomize allows you to patch and extend the addons.

4. How are upgrades are expected to work?

kubectl apply -k should upgrade the addon additively.
More complex upgrades can be handled by packaging the addon as an operator /w RBAC objects.
I have not looked at using the --prune kubectl flag, but an operator can handle deletes.

5. How can kubeadm control the kubeconfig?

This should probably be added to the installer Runtime struct or AddonInstallerConfiguration API (***).
Should this be a file name or an actual kubeconfig type?

6. When dry running, kubeadm currently prints all the resources that will be generated, including also resources for addons. How can we do something similar with the installer?

kubectl prints the names and whether they were changed.
The output currently looks like this:

$ ./bin/installer --config demo/v1alpha1.yaml --dry-run
...installing ../../kustomize/examples/helloWorld (dry run)
configmap/the-map created (dry run)
service/the-service created (dry run)
deployment.apps/the-deployment created (dry run)

...installing github.com/kubernetes-sigs/kustomize//examples/multibases/dev/?ref=v1.0.6 (dry run)
pod/dev-myapp-pod created (dry run)

The human-readable output is helpful in that it tells you what would happen when applied to any existing objects in the cluster currently.
You can control the Runtime's stdout, so that would allow you to store and parse it.
If we pass kubectl apply -k ... -o json or -o yaml it will be structured output. (a List of the full objects)
I'm hesitant to suggest re-implementing any of this logic since it's already so sophisticated.

[fabriziopandini]

@stealthybox what I'm trying to understand is: what is the value-added the addon installers/the addon project WRT to kustomize? how can the addon installers/the addon project simplify the addon lifecycle when used as a library by kubeadm?

[stealthybox]

@fabriziopandini

• The CoreDNS, kube-dns, kube-proxy implementations are no longer hard-coded into the kubeadm binary.
• These kustomize addons can be shared between multiple installers. (kops + eksctl have been following)
• Users can configure kubeadm to omit certain networking components. (ex: https://kube-router.io cannot deploy /w kube-proxy)
• Users can fork and extend the networking components. (can supply their own Corefile)
• Users can configure their CNI provider to be installed /w kubeadm in a ComponentConfig/GitOps manner. (Calico, Weavenet, etc)
• Opening up this config enables kubeadm to execute differing opinions of cluster bootstrap.
  Core addons are no-longer a requirement -- just a default

[stealthybox]

/assign @timothysc
/assign @neolit123

[stealthybox]

gdoc: Brainstorming on integration between kubeadm and addons project

[jwforres]

@stealthybox FYI there are new requirements around any APIs being added under the k8s.io domain, see kubernetes/enhancements#1111

I think there will be less friction about iterating and experimenting with an add-ons API if it is developed under the x-k8s.io domain

[stealthybox]

I think there will be less friction about iterating and experimenting with an add-ons API if it is developed under the x-k8s.io domain

Thanks for the heads-up @jwforres -- it looks like that's the trend with certain clusterapi providers as well.

I don't think we intend for this API to be served be the APIServer with a CRD at the moment.
It's easy to start with this domain though 👍

[fabriziopandini]

As discussed in the kubeadm planning session for 1.17, as soon as we get a KEP defining all the details/impacts on the users, we are going to start executing for integrating this with kubeadm.
Rif this document for the first brainstorming on use cases to be covered

[dholbach]

The KEP is now up at kubernetes/enhancements#1308

[dholbach]

Can we go ahead and merge this PR after #28 is fixed and then iterate as necessary?

[stealthybox]

I've implemented the library-internal KUBECONFIG feature through a filePath.
Hopefully this is an acceptable interface for now.

The config API is now scoped under x-k8s.io -- thanks for pointing this out @jwforres. (Thanks @johnsonj for opening #28)

The library now has a map per-addon requiring a name and kustomizeRef/manifestRef.
It supports installing a single manifest from a URL or File as well as recursive directories via kubectl.

Lastly, I've added a delete mechanism; this is generally useful, and will be necessary for pruning addons based off of an "oldConfig".

I'd like to get this merged and work on these in followups:

• prune logic
• config validation tests
• inline patches

[dholbach]

Great work, Leigh!

Big ➕ 1️⃣ on landing this and filing followup issues to resolved later.

[dholbach]

We discussed this in the Cluster Addons meeting yesterday and noted a couple of things which can be fixed in the next revision. I'll make sure to file issues for this. (Can we add a [kind/installer] label? I'll check.)

/lgtm

[dholbach]

Can we add a [kind/installer] label? I'll check.

Nevermind... we already have 105 labels to choose from. I think I'm good.

[dholbach]

Thanks @stealthybox for your work on this! This is a great foundation for the Cluster Addons project.