From e64b250d4235a931829c05e399880c3f32dd9a45 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Tue, 23 Jan 2024 14:58:03 -0500 Subject: [PATCH 1/6] draft adr multitenancy for zarf init --- adr/0023-kubernetes-multitenancy.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 adr/0023-kubernetes-multitenancy.md diff --git a/adr/0023-kubernetes-multitenancy.md b/adr/0023-kubernetes-multitenancy.md new file mode 100644 index 0000000000..adcfda1736 --- /dev/null +++ b/adr/0023-kubernetes-multitenancy.md @@ -0,0 +1,25 @@ +# 22. Kubernetes Multitenancy with `zarf init` + +Date: 2024-01-23 + +## Status + +Pending + +## Context + +Currently today, when allowing `zarf init` to handle cluster creation, Zarf doesn't have ability to automatically or semi-automatically provision itself across multiple nodes. The idea here would be to allow for horizontal scalability across multiple virtual or physical nodes for site reliability and automatic failover. + +References: + +* https://github.com/defenseunicorns/zarf/issues/1041 +* https://github.com/defenseunicorns/zarf/issues/1040 +* https://github.com/defenseunicorns/zarf/blob/cf9acb50e5a2240e6bd2af994e5904cd0f73fd55/src/pkg/cluster/state.go#L29 + +## Decision + +... + +## Consequences + +... From c151be51620899f9d180d23535edc258e40a82d1 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Tue, 23 Jan 2024 14:58:14 -0500 Subject: [PATCH 2/6] fix typo --- adr/0023-kubernetes-multitenancy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/0023-kubernetes-multitenancy.md b/adr/0023-kubernetes-multitenancy.md index adcfda1736..d122e8234b 100644 --- a/adr/0023-kubernetes-multitenancy.md +++ b/adr/0023-kubernetes-multitenancy.md @@ -1,4 +1,4 @@ -# 22. Kubernetes Multitenancy with `zarf init` +# 22. Kubernetes Multi-tenancy with `zarf init` Date: 2024-01-23 From 1d6f39e6b8e4020845952c15886c035a51a4a3c9 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Tue, 23 Jan 2024 16:29:49 -0500 Subject: [PATCH 3/6] fix up adr to better represent what is being accomplished --- ...netes-multitenancy.md => 0023-multi-node-kubernetes.md} | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) rename adr/{0023-kubernetes-multitenancy.md => 0023-multi-node-kubernetes.md} (61%) diff --git a/adr/0023-kubernetes-multitenancy.md b/adr/0023-multi-node-kubernetes.md similarity index 61% rename from adr/0023-kubernetes-multitenancy.md rename to adr/0023-multi-node-kubernetes.md index d122e8234b..f02aa06922 100644 --- a/adr/0023-kubernetes-multitenancy.md +++ b/adr/0023-multi-node-kubernetes.md @@ -1,4 +1,4 @@ -# 22. Kubernetes Multi-tenancy with `zarf init` +# 22. Multi-Node Kubernetes with `zarf init` Date: 2024-01-23 @@ -12,9 +12,8 @@ Currently today, when allowing `zarf init` to handle cluster creation, Zarf does References: -* https://github.com/defenseunicorns/zarf/issues/1041 -* https://github.com/defenseunicorns/zarf/issues/1040 -* https://github.com/defenseunicorns/zarf/blob/cf9acb50e5a2240e6bd2af994e5904cd0f73fd55/src/pkg/cluster/state.go#L29 +* https://github.com/defenseunicorns/zarf-package-bare-metal +* https://github.com/defenseunicorns/zarf/issues/1002 ## Decision From e34eea7a047da7109cdfa49c3d907b6b212cd838 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Wed, 8 May 2024 18:07:10 -0400 Subject: [PATCH 4/6] wip: adr for multi node k3s --- adr/0023-multi-node-kubernetes.md | 47 ++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/adr/0023-multi-node-kubernetes.md b/adr/0023-multi-node-kubernetes.md index f02aa06922..bf07ecb9da 100644 --- a/adr/0023-multi-node-kubernetes.md +++ b/adr/0023-multi-node-kubernetes.md @@ -10,14 +10,59 @@ Pending Currently today, when allowing `zarf init` to handle cluster creation, Zarf doesn't have ability to automatically or semi-automatically provision itself across multiple nodes. The idea here would be to allow for horizontal scalability across multiple virtual or physical nodes for site reliability and automatic failover. +The first pass would consider scaling horizontally with the Embedded DB model. There would be minimal changes on the current k3s.service config. The change required here would be to include a shared token. By default, if K3S doesn't receive a token it will auto generate one. You can reset this on an existing cluster by running `k3s token rotate --new-token=foo`. + +If one wanted to specify a token in advance they could simply modify their existing `zarf deploy/init`, `--set K3S_ARGS` command to include `--token=foo` + +For example: +```shell +zarf init --components=k3s,git-server --confirm --set K3S_ARGS=\"--disable=traefik --disable=metrics-server --disable=servicelb --tls-san=1.2.3.4 --token=foo\"" +``` + +This results in a line as such for example: + +```ini +ExecStart=/usr/sbin/k3s server --write-kubeconfig-mode=700 --write-kubeconfig /root/.kube/config --disable=traefik --disable=metrics-server --disable=servicelb --tls-san=1.2.3.4 --token=foo +``` + +The difference on the agent side requires a few changes. We must specify three pieces of information: + +* That we want to spin up a K3S agent only, not any other Zarf components. +* The IP of the `server`. +* The shared token specified when creating the `server`. + +This would need to be the results k3s.service file. + +```ini +ExecStart=/usr/sbin/k3s agent --server=https://1.2.3.4:6443 --token=foo +``` + +One approach could be to introduce constants into [k3s.service](packages/distros/k3s/common/k3s.service), that would allow us to reuse it. A new component would essentially set some of those variables. + +For example: + +| Variable | Server | Agent | +|---------------------------------|---------------------------------------------------|---------| +| `###ZARF_CONST_K3S_MODE###` | `server` | `agent` | +| `###ZARF_CONST_K3S_INTERNAL###` | ` --write-kubeconfig-mode=700 --write-kubeconfig` | empty | + | `###ZARF_VAR_K3S_ARGS###` | `--token=foo` | `--server https://1.2.3.4:6443 --token=foo` | + +The new k3s.service file would look like: + +```init +ExecStart=/usr/sbin/k3s ###ZARF_CONST_K3S_MODE######ZARF_CONST_K3S_INTERNAL### ###ZARF_VAR_K3S_ARGS### +``` + References: * https://github.com/defenseunicorns/zarf-package-bare-metal * https://github.com/defenseunicorns/zarf/issues/1002 +* https://docs.k3s.io/datastore/ha-embedded +* https://docs.k3s.io/cli/agent ## Decision -... +TBD ## Consequences From 3a06a79cf9f8fe9e1a83c0d28bed522a406cf851 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Wed, 8 May 2024 18:09:25 -0400 Subject: [PATCH 5/6] wip: adding an additional thought about how you'd run it --- adr/0023-multi-node-kubernetes.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/adr/0023-multi-node-kubernetes.md b/adr/0023-multi-node-kubernetes.md index bf07ecb9da..3f5e45f57e 100644 --- a/adr/0023-multi-node-kubernetes.md +++ b/adr/0023-multi-node-kubernetes.md @@ -53,6 +53,12 @@ The new k3s.service file would look like: ExecStart=/usr/sbin/k3s ###ZARF_CONST_K3S_MODE######ZARF_CONST_K3S_INTERNAL### ###ZARF_VAR_K3S_ARGS### ``` +If this were the case then adding a new k3s agent would be run as (assuming that we had an init package that only had k3s with both the `k3s` and `k3s-agent` as optional packages and nothing else required: + +```shell +zarf init --components=k3s-agent --confirm --set K3S_ARGS=\"--server=https://1.2.3.4:6443 --token=foo\"" +``` + References: * https://github.com/defenseunicorns/zarf-package-bare-metal From 0bc7fda4818ef32affcf2fd1aabafbdc0731a043 Mon Sep 17 00:00:00 2001 From: Danny Gershman Date: Wed, 5 Jun 2024 11:45:56 -0400 Subject: [PATCH 6/6] Update adr/0023-multi-node-kubernetes.md Co-authored-by: Lucas Rodriguez --- adr/0023-multi-node-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/0023-multi-node-kubernetes.md b/adr/0023-multi-node-kubernetes.md index 3f5e45f57e..1ec47c9e2b 100644 --- a/adr/0023-multi-node-kubernetes.md +++ b/adr/0023-multi-node-kubernetes.md @@ -10,7 +10,7 @@ Pending Currently today, when allowing `zarf init` to handle cluster creation, Zarf doesn't have ability to automatically or semi-automatically provision itself across multiple nodes. The idea here would be to allow for horizontal scalability across multiple virtual or physical nodes for site reliability and automatic failover. -The first pass would consider scaling horizontally with the Embedded DB model. There would be minimal changes on the current k3s.service config. The change required here would be to include a shared token. By default, if K3S doesn't receive a token it will auto generate one. You can reset this on an existing cluster by running `k3s token rotate --new-token=foo`. +The first pass would consider scaling horizontally with the [High Availability Embedded etcd](https://docs.k3s.io/datastore/ha-embedded) model. There would be minimal changes on the current k3s.service config. The change required here would be to include a shared token. By default, if K3S doesn't receive a token it will auto generate one. You can reset this on an existing cluster by running `k3s token rotate --new-token=foo`. If one wanted to specify a token in advance they could simply modify their existing `zarf deploy/init`, `--set K3S_ARGS` command to include `--token=foo`