Addons are critical components within a Kubernetes cluster that provide critical services needed by applications like DNS, Ingress, Metrics, Logging, etc. Addon Manager provides a CRD for lifecycle management of such addons using Argo Workflows.
This is Alpha state software, anything can change at any time. Our deprecation and removal policy during Alpha is very aggressive, see below:
- Any changes to the api types will result in an iterative version bump e.g.,
v1alpha1->v1alpha2 - API versions will NOT be backward compatible, all new api versions will immediately replace the older version
- All api kinds within the group
addonmgr.keikoproj.iowill be versioned the same e.g.,v1alphaX
- kubernetes-1.12+
- kubectl-1.14+
To use: kubectl kustomize github.com/keikoproj/addon-manager.git/config/default | kubectl apply -f -
An Addon describes a kubernetes resource-based application that is deployed to a cluster. The Addon CRD defines a spec with some optional and required fields, and a lifecycle where most of the addon may be contained. Internally, Argo Workflows is used for each lifecycle step to specify and submit the resources. The design spec for the Addon CRD is as follows:
apiVersion: addonmgr.keikoproj.io/v1alpha1
kind: Addon
metadata:
name: fluentd-addon
namespace: addon-manager-system
spec:
pkgName: core/fluentd
pkgVersion: v0.0.1
pkgType: composite
pkgDescription: Company fluentd addon.
pkgDeps:
argoproj/workflows: v2.2.1
params:
namespace: mynamespace
clusterContext:
clusterName: "my-test-cluster"
clusterRegion: "us-west-2"
data:
hec_splunk_server: hec.splunk.example.com
selector:
matchLabels:
app.kubernetes.io/name: fluentd
app.kubernetes.io/version: "1.0.0"
lifecycle:
prereqs:
template: |
apiVersion: argoproj.io/v1alpha1
kind: Workflow
...
install:
template: |
apiVersion: argoproj.io/v1alpha1
kind: Workflow
...
delete:
template: |
apiVersion: argoproj.io/v1alpha1
kind: Workflow
...
validate:
template: |
apiVersion: argoproj.io/v1alpha1
kind: Workflow
...To submit: kubectl apply -f addon.yaml
The 4 stages of the addon lifecycle (prereqs, install, validate, delete) have 3 fields, Name, Role, and Template.
The template is specified as an inline Argo Workflow ![alt [^]](https://upload.wikimedia.org/wikipedia/commons/d/d9/VisualEditor_-_Icon_-_External-link.svg){kind=icon}
...
prereqs:
template: |
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: prereqs-minion-manager-
spec:
entrypoint: entry
serviceAccountName: addon-manager-workflow-installer-sa
templates:
- name: entry
steps:
- - name: prereq-resources
template: submit
arguments:
artifacts:
- name: doc
path: /tmp/doc
raw:
data: |
apiVersion: v1
kind: Namespace
metadata:
name: "{{workflow.parameters.namespace}}"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: example-sa
namespace: "{{workflow.parameters.namespace}}"
- name: submit
inputs:
artifacts:
- name: doc
path: /tmp/doc
container:
image: expert360/kubectl-awscli:v1.11.2
command: [sh, -c]
args: ["kubectl apply -f /tmp/doc"]The Addon Manager provides parameter injection to all lifecycle workflows to get rid of the need to input the parameters into each workflow. Before the workflows are run, all key-value pairs in the addon spec.params are made into global workflow parameters. This means their values are accessible like so: "{{workflow.parameters.NAME}}". It's important to make sure when referencing parameters that the templated value is escaped, either with quotes or if it's included in a string literal; if this is not done, you may experience a failed workflow due to the worflow failing to be parsed correctly.
Generally, there are a set of best practices defined that make defining an Addon CR straightforward:
- Each addon (with a few exceptions) should be deployed to its own namespace. This is done by specifying a namespace name
in
spec.params.namespace, and then templating that into each lifecycle workflow where there are namespaced resources, like in the example above. - Best-practice for the prereqs lifecycle step workflow: if the addon has any namespaced resources, the namespace in question should be specified first in the prereqs workflow. Following the namespace, all other non-deployable resources should be specified by priotity in the prereqs workflow as well.
- Best-practice for the install lifecycle step workflow: all deployable resources (deployments, services, statefulsets, replicasets, daemonsets, etc.) should be supplied as part of this workflow.
kubectl get addons -n addon-manager-system
NAME PACKAGE VERSION STATUS AGE
addon-manager-argo-addon addon-argo-workflow v2.2.1 Succeeded 14m
cluster-autoscaler cluster-autoscaler-addon v0.1 Pending 1m
event-router event-router v0.2 Pending 1m
external-dns external-dns v0.2 Pending 1m
fluentd core/fluentd-addon v0.0.1 Pending 1m
...To delete: kubectl delete -f addon.yaml
The Addon Manager is distributed with the addonctl binary which allows a default Addon CR generation given spec parameters yaml resource files, and python scripts. Pre-alpha currently, this tool can be more useful for initial addon generation, allowing the user to then manually modify it.
To use: addonctl --help
A control plane for managing addons
Usage:
addonctl [command]
Available Commands:
create Create the addon resource with the supplied arguments
help Help about any command
Flags:
-c, --channel string Channel for the addon package
--cluster-name string Name of the cluster context being used
--cluster-region string Cluster region
--deps string Comma seperated dependencies list in the format 'pkgName:pkgVersion'
--desc string Description of the addon
--dryrun Outputs the addon spec but doesn't submit
-h, --help help for addonctl
--install string File or directory of resource yaml to submit as install step
-n, --namespace string Namespace where the addon will be deployed
-p, --params string Params to supply to the resource yaml
--prereqs string File or directory of resource yaml to submit as prereqs step
--secrets string Comma seperated list of secret names which are validated as part ofthe addon-manager-system namespace
--selector string Selector applied to all resources?
-t, --type string Addon package type
-v, --version string Addon package version
Use "addonctl [command] --help" for more information about a command.addonctl create my-addon -n my-addon-ns \
--type composite \
--version v0.2 \
--cluster-name my.cluster.k8s.local \
--cluster-region us-west-2 \
--selector app:myaddon \
--prereqs ./prereq_resources.yaml \
--install ./install_resources.yaml \
--dryrunPlease see CONTRIBUTING.md.
Please see DEVELOPER.md.
Instance Manager - Kube Forensics - Active Monitor - Upgrade Manager - Minion Manager - Governor