Added ArgoCD template for creating seed environment and other helpful components

ArgoCD-managed-seed/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

ArgoCD-managed-seed/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: argocd-apps
+description: A Helm chart for Kubernetes
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.16.0"

ArgoCD-managed-seed/README.md

@@ -0,0 +1,63 @@
+# KKP Components management via GitOps
+
+For KKP to function effectively, we need to install a bunch of components like seed-mla (monitoring logging and alerting stack), minio, nginx-ingress-controller, user-cluster-mla stack, etc.
+
+Using a GitOps tool to manage these seed components can be very useful. This folder offers a slightly opinionated tooling to achive the same. The workflow to get this setup would be like below:
+
+1. Install KKP using kkp-installer. (optionally, skip-charts for dex, nginx and cert-manager. See note below.)
+1. (Optionally) Setup indiviual seed clusters. Seed could be master-seed or standalone seed. 
+1. Install ArgoCD as helm-chart in each seed that you want to managed via GitOps.
+1. Install content of this folder as helm-chart in each seed to deploy various compoentns in each seed. Take a look at [values.yaml](./values.yaml) for various customizations possible for customizing what gets installed in each seed.
+
+> Note: If you are deploying this on master-seed and choosing to deploy Cert-manager, Nginx Ingress controller and Dex to be managed by ArgoCD, then remember to remove installation of them via `kubermatic-installer` via `--skip-charts='cert-manager,nginx-ingress-controller,dex'`
+
+This helm-chart is an opinionated view on how the customization files are stored. If your customization files are stored in different naming convention, please look at the [_helper templates](./templates/_helpers.tpl) where the path are generated and adjust them as necessary.
+
+Currently, this helm chart sets up ArgoCD `Applications` for various KKP components via Git Repo based helm-charts. So we will have 2 sources for helm charts..
+1. The Kubermatic Git repo charts
+1. Your installation specific local repo store `values.yaml` for your installation specific customizations. 
+
+See below for the folder structure of your local repo structure. As mentioned above, if your directory structure is different, you would need to adjust the functions defined in `_helpers.tpl` for generating ArgoCD Applications properly.
+
+You can use `helm template` command to check if the generated ArgoCD Application definitions look fine or not.
+
+## Folder and File structure:
+
+The current helm-chart templates assume below file structure for your customization repository:
+
+```shell
+├── Makefile
+├── <environment> # All files for the given environment e.g. dev
+│   ├── values.yaml # environment level common values
+│   ├── clusterIssuer.yaml
+│   ├── settings
+│   │   ├── 00_kubermaticsettings.yaml
+│   │   └── seed-cluster-<seed1>.yaml
+│   │   └── seed-cluster-<seed2>.yaml
+│   ├── common   # any files common across seeds in the given environment
+│   │   ├── custom-ca-bundle.yaml 
+│   │   └── sc-kubermatic-fast.yaml
+│   ├── <seed1>  # seed specific files
+│   │   ├── argoapps-values.yaml  # This is where we control what ArgoCD apps to get installed in given seed.
+│   │   ├── values-usermla.yaml  # customize user-cluster mla stack in this file.
+│   │   └── values.yaml # customize seed stack's values e.g. minio, seed prometheus, etc.
+│   └── <master-seed>  # master specific files
+│       ├── k8cConfig.yaml # master seed needs kubermatic-configuration yaml as well.
+│       ├── seed-kubeconfig-secret-<seed1>.yaml  # secret to hold kubeconfig for each seed.
+│       ├── seed-kubeconfig-secret-<seed2>.yaml  # secret to hold kubeconfig for each seed.
+│   │   ├── argoapps-values.yaml  # This is where we control what ArgoCD apps to get installed in given seed.
+│   │   ├── values-usermla.yaml  # customize user-cluster mla stack in this file.
+│   │   └── values.yaml # customize seed stack's values e.g. minio, seed 
+├── <environment2>     # Similar directory structure as above but for other environment e.g. PROD
+└── values-argocd.yaml  # argoCD helm values
+```
+
+## Deploy
+Check the [Makefile](./proposed-dir-structure/Makefile) on getting some ideas on how to deploy these.
+
+TODO: Remove
+```shell
+export KKP_VERSION=v2.23.2
+helm template argo-apps --set kkpVersion=$KKP_VERSION -f ./test/values.yaml . | kubectl apply -f - 
+```
+

ArgoCD-managed-seed/proposed-dir-structure/Makefile

@@ -0,0 +1,36 @@
+KKP_VERSION=v2.23.6
+INSTALL_DIR=/opt/kubermatic/releases/${KKP_VERSION}
+
+# Install seed
+install-kkp-dev:
+	${INSTALL_DIR}/kubermatic-installer deploy \
+	  --charts-directory ${INSTALL_DIR}/charts --config ./dev/vj1-master/k8cConfig.yaml --helm-values ./dev/vj1-master/values.yaml --storageclass aws \
+	  --skip-charts='cert-manager,nginx-ingress-controller,dex'
+
+create-long-lived-master-seed-kubeconfig:
+	${INSTALL_DIR}/kubermatic-installer convert-kubeconfig /opt/personal/k8s-adventure/src/kubeone161/k1init/vj1-master-kubeconfig > ./seed-ready-kube-config
+
+# Setup Seed1 - DEV Master
+deploy-argo-dev-master:
+	helm upgrade --install argocd --version 5.36.10 --namespace argocd --create-namespace argo/argo-cd -f values-argocd.yaml --set 'server.ingress.hosts[0]=argocd.vj1.lab.kubermatic.io' --set 'server.ingress.tls[0].hosts[0]=argocd.vj1.lab.kubermatic.io'
+
+deploy-argo-apps-dev-master:
+	helm template argo-apps --set kkpVersion=${KKP_VERSION} -f ./dev/vj1-master/argoapps-values.yaml /opt/kubermatic/community-components/ArgoCD-managed-seed | kubectl apply -f -
+
+# Setup Seed2 - DEV India Seed
+deploy-argo-dev-seed:
+	helm upgrade --install argocd --version 5.36.10 --namespace argocd --create-namespace argo/argo-cd -f values-argocd.yaml --set 'server.ingress.hosts[0]=argocd.india.vj1.lab.kubermatic.io' --set 'server.ingress.tls[0].hosts[0]=argocd.india.vj1.lab.kubermatic.io'
+
+deploy-argo-apps-dev-seed:
+	helm template argo-apps --set kkpVersion=${KKP_VERSION} -f ./dev/india/argoapps-values.yaml /opt/kubermatic/community-components/ArgoCD-managed-seed | kubectl apply -f -
+
+create-long-lived-seed-kubeconfig:
+	${INSTALL_DIR}/kubermatic-installer convert-kubeconfig /opt/personal/k8s-adventure/src/kubeone161/k1init-seed/vj1-seed-kubeconfig > ./seed-ready-kube-config
+
+### Local testing
+create-kind-cluster:
+	kind create cluster --config=./kind-install/cluster-nodeport.yaml --image kindest/node:v1.27.3
+deploy-argo-kind-cluster:
+	helm upgrade --install argocd --version 5.36.10 --namespace argocd --create-namespace argo/argo-cd -f values-argocd.yaml --set 'server.ingress.hosts[0]=argocd.dreamit.local' --set 'server.ingress.tls[0].hosts[0]=argocd.dreamit.local'
+deploy-argo-apps-kind-cluster:
+	helm template argo-apps --set kkpVersion=${KKP_VERSION} -f ./dev/kind/argoapps-values.yaml /opt/kubermatic/community-components/ArgoCD-managed-seed | kubectl apply -f -

ArgoCD-managed-seed/templates/_helpers.tpl

@@ -0,0 +1,34 @@
+{{/* vim: set filetype=mustache: */}}
+
+{{/*
+Create path for given chart in the provided git repository
+Path would be - base path + kkp version + /charts + specific name of the chart 
+*/}}
+{{- define "kkp.chart.pathprefix" -}}
+{{ if .Values.kkpChartsInCustomRepo }}
+{{- printf "%s" .Values.kkpBasePath -}}
+{{ else }}
+{{- printf "." -}}
+{{ end  }}
+{{- end -}}
+
+{{- define "git-tag-version" -}}
+{{ .Values.environment }}-kkp-{{ .Values.kkpVersion }}
+{{- end -}}
+
+{{- define "argo-cd-apps.env-specific-values-file.path" -}}
+{{- printf "%s/%s" .Values.environment .Values.envSpecificValuesFileName -}}
+{{- end -}}
+
+{{- define "argo-cd-apps.seed-override-values-file.path" -}}
+{{- printf "%s/%s/%s" .Values.environment .Values.seed .Values.seedOverrideValuesFileName -}}
+{{- end -}}
+
+{{- define "argo-cd-apps.env-specific-kkp-settings.path" -}}
+{{- printf "%s/%s" .Values.environment .Values.envSpecificSettingFolderName -}}
+{{- end -}}
+
+{{- define "argo-cd-apps.user-mla-values-file.path" -}}
+{{- printf "%s/%s/%s" .Values.environment .Values.seed .Values.userMlaValuesFileName -}}
+{{- end -}}
+

ArgoCD-managed-seed/templates/argocd-apps-backup.yaml

@@ -0,0 +1,37 @@
+{{ if .Values.velero }}
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: velero
+
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: velero
+  namespace: argocd
+spec:
+  project: default
+  sources:
+  - repoURL: '{{ .Values.kkpRepoURL }}'
+    path: {{ template "kkp.chart.pathprefix" . }}/charts/backup/velero
+    targetRevision: {{ .Values.kkpVersion }}
+    helm:
+      valueFiles:
+        - values.yaml
+        - $values/{{ template "argo-cd-apps.env-specific-values-file.path" . }}
+        - $values/{{ template "argo-cd-apps.seed-override-values-file.path" . }}
+  # This is your private repo where you have stored the values.yaml customizations which you used to provide to KKP installer. 
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: {{ .Values.valuesPathPrefix }}
+    ref: values
+  # TODO: In 2.24+, add a installCRD option in values.yaml since current Velero chart version is v1 and it does not install CRDs automatically
+  - repoURL: '{{ .Values.kkpRepoURL }}'
+    path: {{ template "kkp.chart.pathprefix" . }}/charts/backup/velero/crd
+    targetRevision: {{ .Values.kkpVersion }}
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: velero
+{{ end }}

ArgoCD-managed-seed/templates/argocd-apps-kkp-core.yaml

@@ -0,0 +1,160 @@
+{{ if or .Values.seedSettings .Values.seedExtras }}
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kubermatic
+{{ end }}
+
+{{ if or .Values.oauth }}
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: oauth
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oauth
+  namespace: argocd
+spec:
+  project: default
+  sources:
+  - repoURL: '{{ .Values.kkpRepoURL }}'
+    path: {{ template "kkp.chart.pathprefix" . }}/charts/oauth
+    targetRevision: {{ .Values.kkpVersion }}
+    helm:
+      valueFiles:
+        - values.yaml
+        - $values/{{ template "argo-cd-apps.env-specific-values-file.path" . }}
+        - $values/{{ template "argo-cd-apps.seed-override-values-file.path" . }}
+  # This is your private repo where you have stored the values.yaml customizations which you used to provide to KKP installer. 
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: {{ .Values.valuesPathPrefix }}
+    ref: values
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: oauth
+{{ end }}
+
+{{ if .Values.nginx }}
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: nginx-ingress-controller
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: nginx-ingress-controller
+  namespace: argocd
+spec:
+  project: default
+  sources:
+  - repoURL: '{{ .Values.kkpRepoURL }}'
+    path: {{ template "kkp.chart.pathprefix" . }}/charts/nginx-ingress-controller
+    targetRevision: {{ .Values.kkpVersion }}
+    helm:
+      valueFiles:
+        - values.yaml
+        - $values/{{ template "argo-cd-apps.env-specific-values-file.path" . }}
+        - $values/{{ template "argo-cd-apps.seed-override-values-file.path" . }}
+  # This is your private repo where you have stored the values.yaml customizations which you used to provide to KKP installer. 
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: {{ .Values.valuesPathPrefix }}
+    ref: values
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: nginx-ingress-controller
+  # ngnix ingress app to be synced automatically so that we can access ArgoCD via it!
+  syncPolicy:
+    automated: {}
+{{ end }}
+
+
+{{ if .Values.certManager }}
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cert-manager
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cert-manager
+  namespace: argocd
+spec:
+  project: default
+  sources:
+  - repoURL: '{{ .Values.kkpRepoURL }}'
+    path: {{ template "kkp.chart.pathprefix" . }}/charts/cert-manager
+    targetRevision: {{ .Values.kkpVersion }}
+    helm:
+      parameters:
+      - name: "cert-manager.installCRDs"
+        value: "true"
+      valueFiles:
+        - values.yaml
+        - $values/{{ template "argo-cd-apps.env-specific-values-file.path" . }}
+        - $values/{{ template "argo-cd-apps.seed-override-values-file.path" . }}
+  # This is your private repo where you have stored the values.yaml customizations which you used to provide to KKP installer. 
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: {{ .Values.valuesPathPrefix }}
+    ref: values
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: cert-manager
+  # cert-manager app to be synced automatically so that we get certificates issued for KKP dashboard and ArgoCD to start with
+  syncPolicy:
+    automated: {}
+{{ end }}
+
+{{ if and .Values.seedSettings .Values.isMasterSeed}}
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: master-settings-{{ .Values.seed }}
+  namespace: argocd
+spec:
+  # TODO: Create segregated projects - should be an option to use different project.
+  # project: kubermatic-config
+  project: default
+  sources:
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: {{ template "argo-cd-apps.env-specific-kkp-settings.path" . }}
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: kubermatic
+{{ end }}
+
+{{ if .Values.seedExtras }}
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: seed-extras-{{ .Values.seed }}
+  namespace: argocd
+spec:
+  # TODO: Create segregated projects
+  # project: kubermatic-config
+  project: default
+  sources:
+  - repoURL: '{{ .Values.repoURL }}'
+    targetRevision: {{ template "git-tag-version" . }}
+    path: '{{ .Values.environment }}'
+    directory:
+      recurse: true
+      exclude: 'values.yaml'
+      include: '{common/*,{{ .Values.seed }}/*}'
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: kubermatic
+{{ end }}