From 4be44e799db9dcdee300ee0b8e31aaee6ccb910c Mon Sep 17 00:00:00 2001 From: razzle Date: Tue, 30 Apr 2024 08:46:11 -0500 Subject: [PATCH 01/11] wip Signed-off-by: razzle --- site/src/content/docs/faq.mdx | 28 ---- site/src/content/docs/ref/init-package.mdx | 166 +++++++++++++++++++-- 2 files changed, 154 insertions(+), 40 deletions(-) diff --git a/site/src/content/docs/faq.mdx b/site/src/content/docs/faq.mdx index 1e662a262a..e4bca91bba 100644 --- a/site/src/content/docs/faq.mdx +++ b/site/src/content/docs/faq.mdx @@ -24,34 +24,6 @@ No, the Zarf binary and init package can be downloaded from the [Releases Page]( Zarf is statically compiled and written in [Go](https://golang.org/) and [Rust](https://www.rust-lang.org/), so it has no external dependencies. For Linux, Zarf can bring a Kubernetes cluster using [K3s](https://k3s.io/). For Mac and Windows, Zarf can leverage any available local or remote cluster the user has access to. Currently, the K3s installation Zarf performs does require a [Systemd](https://en.wikipedia.org/wiki/Systemd) based system and `root` (not just `sudo`) access. -## What is the Zarf Agent? - -The Zarf Agent is a [Kubernetes Mutating Webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) that is installed into the cluster during `zarf init`. The Agent is responsible for modifying [Kubernetes PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) objects [Image](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container.Image) fields to point to the Zarf Registry. This allows the cluster to pull images from the Zarf Registry instead of the internet without having to modify the original image references. The Agent also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. - -## Why doesn't the Zarf Agent create secrets it needs in the cluster? - -During early discussions and [subsequent decision](https://github.com/defenseunicorns/zarf/blob/main/adr/0005-mutating-webhook.md) to use a Mutating Webhook, we decided to not have the Agent create any secrets in the cluster. This is to avoid the Agent having to have more privileges than it needs as well as to avoid collisions with Helm. The Agent today simply responds to requests to patch PodSpec and GitRepository objects. - -The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created as [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. - -## How can a Kubernetes resource be excluded from the Zarf Agent? - -Resources can be excluded at the namespace or resources level by adding the `zarf.dev/agent: ignore` label. - -## What happens to resources that exist in the cluster before `zarf init`? - -During the [`zarf init`](/commands/zarf_init) operation, the Zarf Agent will patch any existing namespaces with the `zarf.dev/agent: ignore` label to prevent the Agent from modifying any resources in that namespace. This is done because there is no way to guarantee the images used by pods in existing namespaces are available in the Zarf Registry. - -If you would like to adopt pre-existing resources into a Zarf deployment you can use the `--adopt-existing-resources` flag on [`zarf package deploy`](/commands/zarf_package_deploy/) to adopt those resources into the Helm Releases that Zarf manages (including namespaces). This will add the requisite annotations and labels to those resources and drop the `zarf.dev/agent: ignore` label from any namespaces specified by those resources. - -:::note - -Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces). It is recommended that you do not deploy resources into the `default` or `kube-*` namespaces with Zarf. - -Additionally, when adopting resources, you should ensure that the namespaces you are adopting are dedicated to Zarf, or that you go back and manually add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise you may see [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors. - -::: - ## How can I improve the speed of loading large images from Docker on `zarf package create`? Due to some limitations with how Docker provides access to local image layers, `zarf package create` has to rely on `docker save` under the hood which is [very slow overall](https://github.com/defenseunicorns/zarf/issues/1214) and also takes a long time to report progress. We experimented with many ways to improve this, but for now recommend leveraging a local docker registry to speed up the process. diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index 660e04e290..df245fb6fe 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -6,26 +6,164 @@ sidebar: import Mermaid from "@components/Mermaid.astro"; -The 'init' package is a special Zarf Package (denoted by `kind: ZarfInitConfig` in its `zarf.yaml`) that initializes a cluster with the requisite air gap services when running `zarf init`. This allows future Zarf Packages to store any required resources (i.e. container images and git repositories) so that they can be retrieved later. +In a traditional Kubernetes deployment, clusters pull resources (e.g. cluster images, OCI artifacts, Git repos) from external sources. -The default 'init' package that Zarf ships is defined in the `zarf.yaml` that lives at the [root of the Zarf repository](https://github.com/defenseunicorns/zarf/blob/main/zarf.yaml), and is constructed from composed components that provide a foundation for customization. If you would like to change the behavior of the 'init' package you can do so by modifying this `zarf.yaml` or any of the composed components that it references and running `zarf package create` at the root of the repository. You can learn more about creating a custom init package in the [Creating a Custom 'init' Package Tutorial](/tutorials/8-custom-init-packages). +However, in an air-gapped environment, these external providers are not available, or exist at different locations to their definitions within Kubernetes manifests. -Upon deployment, the init package creates a `zarf` namespace within your K8s cluster and deploys pods, services, and secrets to that namespace based on the components selected for deployment. +Zarf solves this problem with its 'init' package, a special Zarf Package (traditionally deployed first) that provides the necessary mechanisms to enable air-gapped Kubernetes, and deliver DevSecOps across air-gaps. -## Required Component +:::note[tldr;] -Zarf's mutation capabilities require that the [`zarf-agent`](/faq#what-is-the-zarf-agent) component of the init package is deployed and active within the cluster, meaning that it cannot be disabled and is always running. This component intercepts requests to create resources and uses the `zarf-state` secret to mutate them to point to their air gap equivalents. It is automatically deployed whenever a `zarf init` command is executed. +Don't care about the details and just want to get deploying as quickly as possible? Run the following after connecting to a cluster: -| Component | Description | -| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| zarf-agent | A Kubernetes mutating webhook installed during `zarf init` that converts Pod specs and Flux GitRepository objects to match their air gap equivalents. | +```bash +zarf tools download-init +zarf init --confirm +``` + +Want to see a guided `zarf init`? Check out the [Zarf Init tutorial](/tutorials/1-initializing-a-k8s-cluster/). + +::: + +{/* The 'init' package initializes a cluster with the requisite air gap services, allowing future Zarf Packages to store any required resources (i.e. container images, OCI artifacts, and git repositories) so that they can be accessed by the cluster. */} + +## `zarf.yaml` + +The package definition 'init' is similar to writing any other Zarf Package, but with a few key differences: + +Starting with `kind` and `metadata`: + +```yaml {3, 6, 8} +# zarf.yaml +# kind must be ZarfInitConfig +kind: ZarfInitConfig +metadata: + # name *can* be anything, but it is generally recommended to end with 'init' + name: init + # version will be overwritten by the version of the Zarf CLI that creates the package + # version: 0.1.0 +... +``` + +Continuing with `components`: + +In order for Zarf to operate correctly, the following components + +- must be defined and named **exactly** as shown below +- must have the `required` field set to `true` + +```yaml {4, 9, 14, 19} +# zarf.yaml +components: + # This package moves the injector & registries binaries + - name: zarf-injector + required: true + ... + + # Creates the temporary seed-registry + - name: zarf-seed-registry + required: true + ... + + # Creates the permanent registry + - name: zarf-registry + required: true + ... + + # Creates the pod+git mutating webhook + - name: zarf-agent + required: true + ... +``` + +A minimal `zarf.yaml` for the 'init' package would look like this: + +:::note + +In order to reproduce this example, you will need to have the Zarf repository cloned locally. + +```bash +git clone https://github.com/defenseunicorns/zarf.git +cd zarf +mv zarf.yaml zarf.yaml.bak +cat < zarf.yaml +... +EOF +``` + +You can learn more about creating a custom init package in the [Creating a Custom 'init' Package Tutorial](/tutorials/8-custom-init-packages). + +::: + +```yaml +# zarf.yaml +kind: ZarfInitConfig +metadata: + +components: + - name: zarf-injector + required: true + import: + path: packages/zarf-registry + + - name: zarf-seed-registry + required: true + import: + path: packages/zarf-registry + + - name: zarf-registry + required: true + import: + path: packages/zarf-registry + + - name: zarf-agent + required: true + import: + path: packages/zarf-agent +``` + +But _what_ are these components? Let's take a closer look. + +## Components + +### `zarf-agent` + +{/* TODO: document and flesh out how the mutations operate for the agent */} + +The `zarf-agent` is a [Kubernetes Mutating Webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) that intercepts requests to create resources and uses the `zarf-state` secret to mutate them to point to their air-gapped equivalents. + +The `zarf-agent` is responsible for modifying [Kubernetes PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) objects [Image](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container.Image) fields to point to the Zarf Registry. + +This allows the cluster to pull images from the Zarf Registry instead of the internet without having to modify the original image references. + +The `zarf-agent` also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. + +{/* TODO: document argo here as well */} :::note -The `zarf-agent` will mutate any resources it sees that have not already been patched and don't have the `zarf.dev/agent: ignore` label applied. This label is automatically applied to all namespaces that exist prior to `zarf init`, and will prevent the `zarf-agent` from mutating system resources. You can manually apply this label to any additional namespaces or resources after the fact to prevent the `zarf-agent` from acting upon them. See the FAQ entry to learn more about [what happens to resources that were deployed prior to `zarf init`](/faq#what-happens-to-resources-that-exist-in-the-cluster-before-zarf-init). +During the [`zarf init`](/commands/zarf_init) operation, the Zarf Agent will add the `zarf.dev/agent: ignore` label to prevent the Agent from modifying any resources in that namespace. This is done because there is no way to guarantee the images used by pods in existing namespaces are available in the Zarf Registry. + +If you would like to adopt pre-existing resources into a Zarf deployment you can use the `--adopt-existing-resources` flag on [`zarf package deploy`](/commands/zarf_package_deploy/) to adopt those resources into the Helm Releases that Zarf manages (including namespaces). This will add the requisite annotations and labels to those resources and drop the `zarf.dev/agent: ignore` label from any namespaces specified by those resources. + +::: + +:::danger + +Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces). It is recommended that you do not deploy resources into the `default` or `kube-*` namespaces with Zarf. + +Additionally, when adopting resources, you should ensure that the namespaces you are adopting are dedicated to Zarf, or that you go back and manually add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise you may see [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors. ::: +#### Excluding Resources from `zarf-agent` + +Resources can be excluded at the namespace or resources level by adding the `zarf.dev/agent: ignore` label. + +{/* TODO: should prob make sure there is a tech-debt issue to make Flux ns creation better */} + +The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created as [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. + ## Core Components In addition to the required `zarf-agent` component, Zarf also offers components that provide additional functionality and can be enabled as needed based on your desired end-state. @@ -58,7 +196,9 @@ To see a full list of `variables` you can view the [zarf.yaml that defines the r ::: -Beyond the registry, their are also fully-optional components available for the init package. Many of these also have external configurations you can set with `zarf init` (such as `--git-*`), but these components provide an easy way to get started in environments where these core services are needed and may not already exist. +### Optional Components + +The Zarf team also maintains some optional components, included in the default 'init' package. | Components | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -76,7 +216,7 @@ There are two ways to deploy these optional components. First, you can provide a :::tip -The `k3s` component included in Zarf differs from the default `k3s` install in that it disables the installation of `traefik` out of the box. This was done so that people could more intentionally choose if they wanted `traefik` or another ingress provider (or no ingress at all) depending on their needs. If you would like to return `k3s` to its defaults, you can set the `K3S_ARGS` zarf variable to an empty string: +The `k3s` component included differs from the default `k3s` install in that it disables the installation of `traefik` out of the box. This was done so that people could more intentionally choose if they wanted `traefik` or another ingress provider (or no ingress at all) depending on their needs. If you would like to return `k3s` to its defaults, you can set the `K3S_ARGS` Zarf variable to an empty string: ```text root@machine ~ # zarf init --components k3s --set K3S_ARGS="" --confirm @@ -94,7 +234,9 @@ To see a full list of `variables` you can view the [zarf.yaml that defines the g ## What Makes the Init Package Special -Deploying into air gapped environments is a hard problem, particularly when the K8s environment doesn't have a container registry for you to store images in already. This results in a dilemma where the container registry image must be introduced to the cluster, but there is no container registry to push it to as the image is not yet in the cluster - chicken, meet egg. To ensure that our approach is distro-agnostic, we developed a unique solution to seed the container registry into the cluster. +Deploying into air gapped environments is a hard problem, particularly when the K8s environment doesn't have a container registry for you to store images in already. +This results in a dilemma where the container registry image must be introduced to the cluster, but there is no container registry to push it to as the image is not yet in the cluster - chicken, meet egg. +To ensure that our approach is distro-agnostic, we developed a unique solution to seed the container registry into the cluster. This is done with the `zarf-injector` [component](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-injector/zarf.yaml) which injects a single rust binary (statically compiled) and a series of configmap chunks of a `registry:2` image into an ephemeral pod that is based on an existing image in the cluster. This gives us a running registry to bootstrap from and deploy the rest of the 'init' package and any other packages down the line. From a0691df0ab4af9f388b151bf2fae69605367cabf Mon Sep 17 00:00:00 2001 From: razzle Date: Tue, 30 Apr 2024 13:47:04 -0500 Subject: [PATCH 02/11] stash Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index df245fb6fe..ebdc533146 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -176,6 +176,12 @@ In most scenarios, Zarf will also deploy an internal registry using the three co | zarf-seed-registry | Adds a temporary container registry so Zarf can bootstrap itself into the cluster. | | zarf-registry | Adds a long-lived container registry service—[docker registry](https://docs.docker.com/registry/)—into the cluster. | +```md +TODO: + +- +``` + :::note Given the registry is a core part of any Kubernetes deployment you MUST either specify an external registry with the `--registry-*` flags or use the injected registry which is why it cannot be selected with `--components` like the components below. From 8065e0a90ab57d17eb9853a38cb43a83ec02c69b Mon Sep 17 00:00:00 2001 From: razzle Date: Thu, 9 May 2024 03:39:20 -0500 Subject: [PATCH 03/11] update Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 268 +++++++++++---------- 1 file changed, 142 insertions(+), 126 deletions(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index ebdc533146..83eb9a0029 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -5,10 +5,11 @@ sidebar: --- import Mermaid from "@components/Mermaid.astro"; +import Details from "@components/Details.astro"; In a traditional Kubernetes deployment, clusters pull resources (e.g. cluster images, OCI artifacts, Git repos) from external sources. -However, in an air-gapped environment, these external providers are not available, or exist at different locations to their definitions within Kubernetes manifests. +However, in an air-gapped environment, these external providers are not available, or exist at different locations to their references within Kubernetes manifests. Zarf solves this problem with its 'init' package, a special Zarf Package (traditionally deployed first) that provides the necessary mechanisms to enable air-gapped Kubernetes, and deliver DevSecOps across air-gaps. @@ -23,110 +24,60 @@ zarf init --confirm Want to see a guided `zarf init`? Check out the [Zarf Init tutorial](/tutorials/1-initializing-a-k8s-cluster/). +View all init options w/ [`zarf init --help`](/commands/zarf_init/). + ::: {/* The 'init' package initializes a cluster with the requisite air gap services, allowing future Zarf Packages to store any required resources (i.e. container images, OCI artifacts, and git repositories) so that they can be accessed by the cluster. */} -## `zarf.yaml` - -The package definition 'init' is similar to writing any other Zarf Package, but with a few key differences: - -Starting with `kind` and `metadata`: +## Components -```yaml {3, 6, 8} -# zarf.yaml -# kind must be ZarfInitConfig -kind: ZarfInitConfig -metadata: - # name *can* be anything, but it is generally recommended to end with 'init' - name: init - # version will be overwritten by the version of the Zarf CLI that creates the package - # version: 0.1.0 -... -``` +An 'init' package requires a series of specially named, and configured components to ensure the cluster is correctly initialized. These components are: -Continuing with `components`: +- [`zarf-injector`](#zarf-injector) +- [`zarf-seed-registry`](#zarf-seed-registry) +- [`zarf-registry`](#zarf-registry) +- [`zarf-agent`](#zarf-agent) -In order for Zarf to operate correctly, the following components +### `zarf-injector` and `zarf-seed-registry` -- must be defined and named **exactly** as shown below -- must have the `required` field set to `true` +One of the most challenging aspects of deploying into an air-gapped environment is the initial bootstrapping of the cluster. -```yaml {4, 9, 14, 19} -# zarf.yaml -components: - # This package moves the injector & registries binaries - - name: zarf-injector - required: true - ... +A cluster needs a registry to pull images from, but spinning up a registry requires an image to be pulled from a registry - chiken, meet egg. - # Creates the temporary seed-registry - - name: zarf-seed-registry - required: true - ... +To ensure that our approach is distro-agnostic, we developed a unique solution to seed the container registry into the cluster. - # Creates the permanent registry - - name: zarf-registry - required: true - ... +Shoving random data into a cluster is generally a bad idea, and an antipattern overall to containerization. However in the case of Zarf, and air-gapped environments, certain liberties must be taken. - # Creates the pod+git mutating webhook - - name: zarf-agent - required: true - ... -``` +While there is no distro-agnostic to inject images into a cluster, every cluster has support for `configmaps`. However, the size of a `configmap` is limited to 1MB (technically only limited by whatever is configured in `etcd`, the default is 1MB), and the `registry:2` image is around 10MB (as of this writing). So we split the `registry:2` into chunks and inject them into the cluster as `configmaps`. -A minimal `zarf.yaml` for the 'init' package would look like this: +But then we have another problem of how to reassemble the image on the other side, as we don't have any consistent image that exists in the cluster that would have such utilities. This is where the `zarf-injector` Rust binary comes in. -:::note +> For compiling the `zarf-injector` binary, refer to its [README.md](https://github.com/defenseunicorns/zarf/tree/main/src/injector/README.md). -In order to reproduce this example, you will need to have the Zarf repository cloned locally. +The `zarf-injector` binary is statically compiled and injected into the cluster as a `configmap` along with the chunks of the `registry:2` image. During the `zarf-seed-registry`'s deployment, the `zarf-injector` binary is run in a pod that has access to the `configmaps` and reassembles the `registry:2` image. It then hosts a temporary, pull-only Docker registry implemented in Rust so that a real registry can be deployed into the cluster from the hosted `registry:2` image. -```bash -git clone https://github.com/defenseunicorns/zarf.git -cd zarf -mv zarf.yaml zarf.yaml.bak -cat < zarf.yaml -... -EOF -``` +> While the `zarf-injector` component *must* be defined and deployed *before* the `zarf-seed-registry` component, the magic doesn't start until `zarf-seed-registry` is deployed. The `zarf-injector` component for the most part just copies the `zarf-injector` Rust binary (more on that later) to `###ZARF_TEMP###/zarf-injector`. -You can learn more about creating a custom init package in the [Creating a Custom 'init' Package Tutorial](/tutorials/8-custom-init-packages). +When `zarf init` starts deploying the `zarf-seed-registry` component, the following happens: -::: +1. Zarf injects the `zarf-injector` binary and the `registry:2` image chunks into the cluster. +2. Zarf connects to the cluster and grabs a pod that is running an image that is already present in the cluster. +3. Zarf spins up a pod using the existing image, mounts the `configmaps` that contain the `zarf-injector` binary and the `registry:2` image chunks and runs the `zarf-injector` binary. +4. The `zarf-injector` binary reassembles the `registry:2` image and hosts a temporary registry that the cluster can pull from. +{/* TODO: FINISH ME */} -```yaml -# zarf.yaml -kind: ZarfInitConfig -metadata: - -components: - - name: zarf-injector - required: true - import: - path: packages/zarf-registry - - - name: zarf-seed-registry - required: true - import: - path: packages/zarf-registry +:::note - - name: zarf-registry - required: true - import: - path: packages/zarf-registry +The `registry:2` image and the Zarf Agent image can be configured with a custom init package using the `registry_image_*` and `agent_image_*` templates defined in the Zarf repo's [zarf-config.toml](https://github.com/defenseunicorns/zarf/blob/main/zarf-config.toml). This allows you to swap them for enterprise provided / hardened versions if desired such as those provided by [Iron Bank](https://repo1.dso.mil/dsop/opensource/defenseunicorns/zarf/zarf-agent). - - name: zarf-agent - required: true - import: - path: packages/zarf-agent -``` +::: -But _what_ are these components? Let's take a closer look. +> complete me -## Components +### zarf-registry -### `zarf-agent` +### zarf-agent {/* TODO: document and flesh out how the mutations operate for the agent */} @@ -148,39 +99,31 @@ If you would like to adopt pre-existing resources into a Zarf deployment you can ::: -:::danger - -Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces). It is recommended that you do not deploy resources into the `default` or `kube-*` namespaces with Zarf. - -Additionally, when adopting resources, you should ensure that the namespaces you are adopting are dedicated to Zarf, or that you go back and manually add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise you may see [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors. - -::: - #### Excluding Resources from `zarf-agent` Resources can be excluded at the namespace or resources level by adding the `zarf.dev/agent: ignore` label. +Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces) (`default`, `kube-*`, etc...). This is because these namespaces are critical to the operation of the cluster and should not be managed by Zarf. + +Additionally, when adopting resources, ensure that the namespaces specified are dedicated to Zarf, or add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors may occur. + {/* TODO: should prob make sure there is a tech-debt issue to make Flux ns creation better */} -The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created as [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. +The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created in a [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. -## Core Components +#### Using External Registries -In addition to the required `zarf-agent` component, Zarf also offers components that provide additional functionality and can be enabled as needed based on your desired end-state. +{/* TODO: finish this section */} -In most scenarios, Zarf will also deploy an internal registry using the three components described below. However, Zarf can be configured to use an already existing registry with the `--registry-*` flags when running `zarf init` (detailed information on all `zarf init` command flags can be found in the [zarf init CLI](/commands/zarf_init/) section). This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it uploads any images to the externally configured registry. +Zarf can be configured to use an already existing registry with the `--registry-*` flags when running `zarf init` (detailed information on all `zarf init` command flags can be found in the [zarf init CLI](/commands/zarf_init/) section). This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it uploads any images to the externally configured registry. + +## Core Components | Components | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | zarf-injector | Adds a Rust binary to the working directory to be injected into the cluster during registry bootstrapping. | | zarf-seed-registry | Adds a temporary container registry so Zarf can bootstrap itself into the cluster. | -| zarf-registry | Adds a long-lived container registry service—[docker registry](https://docs.docker.com/registry/)—into the cluster. | - -```md -TODO: - -- -``` +| zarf-registry | Adds a long-lived container registry service [docker registry](https://docs.docker.com/registry/) into the cluster. | :::note @@ -198,19 +141,19 @@ The Zarf Registry is initially injected as a series of config maps that bootstra You can further customize how the registry behaves by setting variables such as `REGISTRY_PVC_SIZE` with a [config file](/ref/config-files/) or `--set` on `zarf init`. -To see a full list of `variables` you can view the [zarf.yaml that defines the registry](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-registry/zarf.yaml). +To see a full list of `variables` you can view the [`zarf.yaml` that defines the registry](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-registry/zarf.yaml). ::: ### Optional Components -The Zarf team also maintains some optional components, included in the default 'init' package. +The Zarf team maintains some optional components in the default 'init' package. | Components | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| k3s | REQUIRES ROOT (not sudo). Installs a lightweight Kubernetes Cluster on the local host—[K3s](https://k3s.io/)—and configures it to start up on boot. | -| logging | Adds a log monitoring stack—[promtail/loki/grafana (aka PLG)](https://github.com/grafana/loki)—into the cluster. | -| git-server | Adds a [GitOps](https://about.gitlab.com/topics/gitops/)-compatible source control service—[Gitea](https://gitea.io/en-us/)—into the cluster. | +| k3s | REQUIRES ROOT (not sudo). Installs a lightweight Kubernetes Cluster on the local host [K3s](https://k3s.io/) and configures it to start up on boot. | +| logging | Adds a log monitoring stack [promtail/loki/grafana (aka PLG)](https://github.com/grafana/loki) into the cluster. | +| git-server | Adds a [GitOps](https://about.gitlab.com/topics/gitops/)-compatible source control service [Gitea](https://gitea.io/en-us/) into the cluster. | There are two ways to deploy these optional components. First, you can provide a comma-separated list of components to the `--components` flag, such as `zarf init --components k3s,git-server --confirm`, or, you can choose to exclude the `--components` and `--confirm` flags and respond with a yes (`y`) or no (`n`) for each optional component when interactively prompted. @@ -238,21 +181,7 @@ To see a full list of `variables` you can view the [zarf.yaml that defines the g ::: -## What Makes the Init Package Special - -Deploying into air gapped environments is a hard problem, particularly when the K8s environment doesn't have a container registry for you to store images in already. -This results in a dilemma where the container registry image must be introduced to the cluster, but there is no container registry to push it to as the image is not yet in the cluster - chicken, meet egg. -To ensure that our approach is distro-agnostic, we developed a unique solution to seed the container registry into the cluster. - -This is done with the `zarf-injector` [component](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-injector/zarf.yaml) which injects a single rust binary (statically compiled) and a series of configmap chunks of a `registry:2` image into an ephemeral pod that is based on an existing image in the cluster. This gives us a running registry to bootstrap from and deploy the rest of the 'init' package and any other packages down the line. - -:::note - -The `registry:2` image and the Zarf Agent image can be configured with a custom init package using the `registry_image_*` and `agent_image_*` templates defined in the Zarf repo's [zarf-config.toml](https://github.com/defenseunicorns/zarf/blob/main/zarf-config.toml). This allows you to swap them for enterprise provided / hardened versions if desired such as those provided by [Iron Bank](https://repo1.dso.mil/dsop/opensource/defenseunicorns/zarf/zarf-agent). - -::: - -## Making the Registry Highly-Available +#### Making the Registry Highly-Available By default, the registry included in the init package creates a ReadWriteOnce PVC and is only scheduled to run on one node at a time. This setup is usually enough for smaller and simpler deployments. However, for larger deployments or those where nodes are frequently restarted or updated, you may want to make the registry highly-available. @@ -282,10 +211,102 @@ package: Notably, the `REGISTRY_AFFINITY_CUSTOM` variable overrides the default pod anti-affinity, and `REGISTRY_HPA_AUTO_SIZE` automatically adjusts the minimum and maximum replicas for the registry based on the number of nodes in the cluster. If you prefer to manually set the minimum and maximum replicas, you can use `REGISTRY_HPA_MIN` and `REGISTRY_HPA_MAX` to specify the desired values. +## Putting it All Together + +The package definition 'init' is similar to writing any other Zarf Package, but with a few key differences: + +Starting with `kind` and `metadata`: + +```yaml {3, 6, 8} +# zarf.yaml +# kind must be ZarfInitConfig +kind: ZarfInitConfig +metadata: + # name *can* be anything, but it is generally recommended to end with 'init' + name: init + # version will be overwritten by the version of the Zarf CLI that creates the package + # version: 0.1.0 +... +``` + +In order for Zarf to operate correctly, the following `components`: + +- must be defined, ordered, and named **exactly** as shown below +- must have the `required` field set to `true` + +```yaml {3, 7, 11, 15} +# zarf.yaml +components: + - name: zarf-injector + required: true + ... + + - name: zarf-seed-registry + required: true + ... + + - name: zarf-registry + required: true + ... + + - name: zarf-agent + required: true + ... +``` + +:::note + +In order to reproduce the following example, you will need to have the Zarf repository cloned locally. + +```bash +git clone https://github.com/defenseunicorns/zarf.git +cd zarf +mv zarf.yaml zarf.yaml.bak +cat < zarf.yaml +... +EOF +``` + +You can learn more about creating a custom init package in the [Creating a Custom 'init' Package Tutorial](/tutorials/8-custom-init-packages). + +::: + +A minimal `zarf.yaml` for the 'init' package looks something like: + +```yaml +# zarf.yaml +kind: ZarfInitConfig +metadata: + +components: + - name: zarf-injector + required: true + import: + path: packages/zarf-registry + + - name: zarf-seed-registry + required: true + import: + path: packages/zarf-registry + + - name: zarf-registry + required: true + import: + path: packages/zarf-registry + + - name: zarf-agent + required: true + import: + path: packages/zarf-agent +``` + +{/* technically the most minimal you can go is just `zarf-agent` and using an external registry / git server but idk if it's worth documenting that */} + ## The `zarf init` Lifecycle The `zarf init` lifecycle is _very similar_ to the [`zarf package deploy` lifecycle](/ref/deploy/) except that it sets up resources specific to Zarf such as the `zarf-state` and performs special actions such as the injection procedure. +
B2 @@ -331,9 +352,4 @@ graph TD classDef action fill:#bd93f9,color:#000000 classDef fail fill:#aa0000 `}/> - -:::tip - -Given that these flows are so similar you actually can `zarf package deploy` the init package. This is similar to accepting the defaults for any init-specific command flags and is useful when trying to deploy packages in a more generic way. - -::: +
From 6c8cd8ce5d100f4f0463d5c4c84c24b564e89b35 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 01:18:51 -0500 Subject: [PATCH 04/11] update Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 211 ++++++++------------- 1 file changed, 84 insertions(+), 127 deletions(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index 83eb9a0029..7a4129a078 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -30,7 +30,7 @@ View all init options w/ [`zarf init --help`](/commands/zarf_init/). {/* The 'init' package initializes a cluster with the requisite air gap services, allowing future Zarf Packages to store any required resources (i.e. container images, OCI artifacts, and git repositories) so that they can be accessed by the cluster. */} -## Components +## Core Components An 'init' package requires a series of specially named, and configured components to ensure the cluster is correctly initialized. These components are: @@ -43,9 +43,9 @@ An 'init' package requires a series of specially named, and configured component One of the most challenging aspects of deploying into an air-gapped environment is the initial bootstrapping of the cluster. -A cluster needs a registry to pull images from, but spinning up a registry requires an image to be pulled from a registry - chiken, meet egg. +A cluster needs a registry to pull images from, but spinning up a registry requires an image to be pulled from a registry - chicken, meet egg. -To ensure that our approach is distro-agnostic, we developed a unique solution to seed the container registry into the cluster. +To ensure that our approach is distro-agnostic, the Zarf team developed a unique solution to seed the container registry into the cluster, populate said registry, and redirect cluster resources to use the air-gapped registry. Shoving random data into a cluster is generally a bad idea, and an antipattern overall to containerization. However in the case of Zarf, and air-gapped environments, certain liberties must be taken. @@ -55,17 +55,22 @@ But then we have another problem of how to reassemble the image on the other sid > For compiling the `zarf-injector` binary, refer to its [README.md](https://github.com/defenseunicorns/zarf/tree/main/src/injector/README.md). -The `zarf-injector` binary is statically compiled and injected into the cluster as a `configmap` along with the chunks of the `registry:2` image. During the `zarf-seed-registry`'s deployment, the `zarf-injector` binary is run in a pod that has access to the `configmaps` and reassembles the `registry:2` image. It then hosts a temporary, pull-only Docker registry implemented in Rust so that a real registry can be deployed into the cluster from the hosted `registry:2` image. +The `zarf-injector` binary is statically compiled and injected into the cluster as a `configmap` along with the chunks of the `registry:2` image. During the `zarf-seed-registry`'s deployment, the `zarf-injector` binary is run in a pod that mounts the `configmaps` and reassembles the `registry:2` image. It then hosts a temporary, pull-only Docker registry implemented in Rust so that a real registry can be deployed into the cluster from the hosted `registry:2` image. -> While the `zarf-injector` component *must* be defined and deployed *before* the `zarf-seed-registry` component, the magic doesn't start until `zarf-seed-registry` is deployed. The `zarf-injector` component for the most part just copies the `zarf-injector` Rust binary (more on that later) to `###ZARF_TEMP###/zarf-injector`. +> While the `zarf-injector` component *must* be defined and deployed *before* the `zarf-seed-registry` component, the magic doesn't start until `zarf-seed-registry` is deployed. The `zarf-injector` component for the most part just copies the `zarf-injector` binary to `###ZARF_TEMP###/zarf-injector`. -When `zarf init` starts deploying the `zarf-seed-registry` component, the following happens: +When `zarf init` deploys the `zarf-seed-registry` component, the following happens: 1. Zarf injects the `zarf-injector` binary and the `registry:2` image chunks into the cluster. 2. Zarf connects to the cluster and grabs a pod that is running an image that is already present in the cluster. 3. Zarf spins up a pod using the existing image, mounts the `configmaps` that contain the `zarf-injector` binary and the `registry:2` image chunks and runs the `zarf-injector` binary. + +> Doing this keeps Zarf cluster agnostic however does require that the kubelet be able to reach out to a cluster NodePort service which may require changes to firewall configurations like allowing UDP traffic between nodes if using something like VXLAN tunneling. + 4. The `zarf-injector` binary reassembles the `registry:2` image and hosts a temporary registry that the cluster can pull from. -{/* TODO: FINISH ME */} +5. The `docker-registry` chart in the `zarf-seed-registry` component is then deployed, with its `image.repository` set to the temporary registry that the `zarf-injector` binary is hosting (consumed as the `###ZARF_SEED_REGISTRY###` built-in variable set at runtime). +6. Once the `docker-registry` chart is deployed, the `zarf-seed-registry` component is marked as complete and the `zarf-injector` pod is removed from the cluster. +7. Deployment proceeds to the `zarf-registry` component. :::note @@ -73,79 +78,108 @@ The `registry:2` image and the Zarf Agent image can be configured with a custom ::: -> complete me +### `zarf-registry` -### zarf-registry +The `zarf-registry` component is a long-lived container registry service that is deployed into the cluster. -### zarf-agent +It leverages the same `docker-registry` chart used in `zarf-seed-registry` but with a few key differences: -{/* TODO: document and flesh out how the mutations operate for the agent */} +1. The `image.repository` is set to the value of the built-in variable `###ZARF_REGISTRY###` which is set at runtime to the registry hosted by `zarf-seed-registry`. +{/* you know, why DO we do this? if we kept the repository the same, and kept running the injector, couldnt this handle cluster full-deaths??? (ofc would need to tweak the zarf-agent to not mutate the image.repository for the docker-registry chart under the zarf namespace, but i think it might be doable) */} +2. A `connect` manifest for running [`zarf connect registry`](/commands/zarf_connect_registry/) to tunnel to the Zarf Registry. +3. A configmap to satisfy [KEP-1755](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry) -The `zarf-agent` is a [Kubernetes Mutating Webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) that intercepts requests to create resources and uses the `zarf-state` secret to mutate them to point to their air-gapped equivalents. +:::tip -The `zarf-agent` is responsible for modifying [Kubernetes PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) objects [Image](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container.Image) fields to point to the Zarf Registry. +You can further customize how the registry behaves by setting variables such as `REGISTRY_PVC_SIZE` with a [config file](/ref/config-files/) or `--set` on `zarf init`. -This allows the cluster to pull images from the Zarf Registry instead of the internet without having to modify the original image references. +To see a full list of `variables` you can view the [`zarf.yaml` that defines the registry](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-registry/zarf.yaml). -The `zarf-agent` also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. +::: -{/* TODO: document argo here as well */} +#### Using External Registries -:::note +{/* TODO: finish this section, and provide an example? */} -During the [`zarf init`](/commands/zarf_init) operation, the Zarf Agent will add the `zarf.dev/agent: ignore` label to prevent the Agent from modifying any resources in that namespace. This is done because there is no way to guarantee the images used by pods in existing namespaces are available in the Zarf Registry. +Zarf can be configured to use an already existing registry with the `--registry-*` flags when running [`zarf init`](/commands/zarf_init/). -If you would like to adopt pre-existing resources into a Zarf deployment you can use the `--adopt-existing-resources` flag on [`zarf package deploy`](/commands/zarf_package_deploy/) to adopt those resources into the Helm Releases that Zarf manages (including namespaces). This will add the requisite annotations and labels to those resources and drop the `zarf.dev/agent: ignore` label from any namespaces specified by those resources. +This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it pushes any images contained in the package to the externally configured registry. + +:::note + +Given the registry is a core part of any Kubernetes deployment you MUST either specify an external registry with the `--registry-*` flags or use the injected registry. ::: -#### Excluding Resources from `zarf-agent` +#### Making the Registry Highly-Available -Resources can be excluded at the namespace or resources level by adding the `zarf.dev/agent: ignore` label. +By default, the registry included in the init package creates a `ReadWriteOnce` PVC and is only scheduled to run on one node at a time. -Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces) (`default`, `kube-*`, etc...). This is because these namespaces are critical to the operation of the cluster and should not be managed by Zarf. +This setup is usually enough for smaller and simpler deployments. However, for larger deployments or those where nodes are frequently restarted or updated, you may want to make the registry highly-available. -Additionally, when adopting resources, ensure that the namespaces specified are dedicated to Zarf, or add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors may occur. +This approach requires certain prerequisites, such as a storage class that supports `ReadWriteMany`, or being in an environment that allows you to configure the registry to use an S3-compatible backend. -{/* TODO: should prob make sure there is a tech-debt issue to make Flux ns creation better */} +Additionally, you must provide custom configuration to the registry to ensure it is distributed across all nodes and has the appropriate number of replicas. Below is an example [configuration file](/ref/config-files/) using a ReadWriteMany storage class: -The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created in a [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. +```yaml +# zarf-config.yaml +package: + deploy: + set: + REGISTRY_PVC_ENABLED: "true" + REGISTRY_PVC_ACCESS_MODE: "ReadWriteMany" + REGISTRY_HPA_AUTO_SIZE: "true" + REGISTRY_AFFINITY_CUSTOM: | + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - docker-registry + topologyKey: kubernetes.io/hostname +``` -#### Using External Registries +Notably, the `REGISTRY_AFFINITY_CUSTOM` variable overrides the default pod anti-affinity, and `REGISTRY_HPA_AUTO_SIZE` automatically adjusts the minimum and maximum replicas for the registry based on the number of nodes in the cluster. If you prefer to manually set the minimum and maximum replicas, you can use `REGISTRY_HPA_MIN` and `REGISTRY_HPA_MAX` to specify the desired values. -{/* TODO: finish this section */} +### `zarf-agent` -Zarf can be configured to use an already existing registry with the `--registry-*` flags when running `zarf init` (detailed information on all `zarf init` command flags can be found in the [zarf init CLI](/commands/zarf_init/) section). This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it uploads any images to the externally configured registry. +{/* TODO: document and flesh out how the mutations operate for the agent */} -## Core Components +The `zarf-agent` is a [Kubernetes Mutating Webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) that intercepts requests to create resources and uses the `zarf-state` secret to mutate them to point to their air-gapped equivalents. -| Components | Description | -| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -| zarf-injector | Adds a Rust binary to the working directory to be injected into the cluster during registry bootstrapping. | -| zarf-seed-registry | Adds a temporary container registry so Zarf can bootstrap itself into the cluster. | -| zarf-registry | Adds a long-lived container registry service [docker registry](https://docs.docker.com/registry/) into the cluster. | +The `zarf-agent` is responsible for modifying [Kubernetes PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec) objects [Image](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container.Image) fields to point to the Zarf Registry. -:::note +This allows the cluster to pull images from the Zarf Registry instead of the internet without having to modify the original image references. -Given the registry is a core part of any Kubernetes deployment you MUST either specify an external registry with the `--registry-*` flags or use the injected registry which is why it cannot be selected with `--components` like the components below. +The `zarf-agent` also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. -::: +{/* TODO: document argo here as well */} :::note -The Zarf Registry is initially injected as a series of config maps that bootstraps itself into the cluster and then binds to a NodePort to allow the kubelet to pull images and setup the final registry deployment. Doing this keeps Zarf cluster agnostic however does require that the kubelet be able to reach out to a cluster NodePort service which may require changes to firewall configurations like allowing UDP traffic between nodes if using something like VXLAN tunneling. +During the [`zarf init`](/commands/zarf_init) operation, the Zarf Agent will add the `zarf.dev/agent: ignore` label to prevent the Agent from modifying any resources in that namespace. This is done because there is no way to guarantee the images used by pods in existing namespaces are available in the Zarf Registry. + +If you would like to adopt pre-existing resources into a Zarf deployment you can use the `--adopt-existing-resources` flag on [`zarf package deploy`](/commands/zarf_package_deploy/) to adopt those resources into the Helm Releases that Zarf manages (including namespaces). This will add the requisite annotations and labels to those resources and drop the `zarf.dev/agent: ignore` label from any namespaces specified by those resources. ::: -:::tip +#### Excluding Resources from `zarf-agent` -You can further customize how the registry behaves by setting variables such as `REGISTRY_PVC_SIZE` with a [config file](/ref/config-files/) or `--set` on `zarf init`. +Resources can be excluded at the namespace or resources level by adding the `zarf.dev/agent: ignore` label. -To see a full list of `variables` you can view the [`zarf.yaml` that defines the registry](https://github.com/defenseunicorns/zarf/blob/main/packages/zarf-registry/zarf.yaml). +Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces) (`default`, `kube-*`, etc...). This is because these namespaces are critical to the operation of the cluster and should not be managed by Zarf. -::: +Additionally, when adopting resources, ensure that the namespaces specified are dedicated to Zarf, or add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors may occur. -### Optional Components +{/* TODO: should prob make sure there is a tech-debt issue to make Flux ns creation better */} + +The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created in a [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. + +## Optional Components The Zarf team maintains some optional components in the default 'init' package. @@ -181,50 +215,21 @@ To see a full list of `variables` you can view the [zarf.yaml that defines the g ::: -#### Making the Registry Highly-Available - -By default, the registry included in the init package creates a ReadWriteOnce PVC and is only scheduled to run on one node at a time. This setup is usually enough for smaller and simpler deployments. However, for larger deployments or those where nodes are frequently restarted or updated, you may want to make the registry highly-available. - -This approach requires certain prerequisites, such as a storage class that supports ReadWriteMany, or being in an environment that allows you to configure the registry to use an S3-compatible backend. Additionally, you must provide custom configuration to the registry to ensure it is distributed across all nodes and has the appropriate number of replicas. Below is an example [configuration file](/ref/config-files/) using a ReadWriteMany storage class: - -```yaml -# zarf-config.yaml -package: - deploy: - set: - REGISTRY_PVC_ENABLED: "true" - REGISTRY_PVC_ACCESS_MODE: "ReadWriteMany" - REGISTRY_HPA_AUTO_SIZE: "true" - REGISTRY_AFFINITY_CUSTOM: | - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - docker-registry - topologyKey: kubernetes.io/hostname -``` - -Notably, the `REGISTRY_AFFINITY_CUSTOM` variable overrides the default pod anti-affinity, and `REGISTRY_HPA_AUTO_SIZE` automatically adjusts the minimum and maximum replicas for the registry based on the number of nodes in the cluster. If you prefer to manually set the minimum and maximum replicas, you can use `REGISTRY_HPA_MIN` and `REGISTRY_HPA_MAX` to specify the desired values. - ## Putting it All Together The package definition 'init' is similar to writing any other Zarf Package, but with a few key differences: Starting with `kind` and `metadata`: -```yaml {3, 6, 8} +```yaml {3, 6, 9} # zarf.yaml # kind must be ZarfInitConfig kind: ZarfInitConfig metadata: # name *can* be anything, but it is generally recommended to end with 'init' name: init - # version will be overwritten by the version of the Zarf CLI that creates the package + # version should be empty as it will be set by the Zarf CLI + # (this is ONLY for the 'init' package) # version: 0.1.0 ... ``` @@ -237,6 +242,8 @@ In order for Zarf to operate correctly, the following `components`: ```yaml {3, 7, 11, 15} # zarf.yaml components: + # components (like k3s) that spin up a cluster... + - name: zarf-injector required: true ... @@ -252,6 +259,8 @@ components: - name: zarf-agent required: true ... + + # optional components that need a cluster ... ``` :::note @@ -301,55 +310,3 @@ components: ``` {/* technically the most minimal you can go is just `zarf-agent` and using an external registry / git server but idk if it's worth documenting that */} - -## The `zarf init` Lifecycle - -The `zarf init` lifecycle is _very similar_ to the [`zarf package deploy` lifecycle](/ref/deploy/) except that it sets up resources specific to Zarf such as the `zarf-state` and performs special actions such as the injection procedure. - -
-B2 - B2(handle multipart package)-->B3 - B3(extract archive to temp dir)-->B4 - B4(validate package checksums and signature)-->B5 - B5(filter components by architecture & OS)-->B6 - B6(save SBOM files to current dir)-->B7 - B7(handle deprecations and breaking changes)-->B9 - B9(confirm package deploy):::prompt-->B10 - B10(process deploy-time variables)-->B11 - B11(prompt for missing variables)-->B12 - B12(prompt to confirm components)-->B13 - B13(prompt to choose components in '.group')-->B14 - - subgraph "" - B52 --> |Yes|B14(deploy each component)-->B14 - B14 --> B15{Component is \n zarf-seed-registry} - B15 --> |Yes|B51(initialize zarf-state secret):::action - B51 --> B52{External \n registry configured} - B52 --> |No|B53(run injection process):::action-->B16 - B15 --> |No|B16(run each '.actions.onDeploy.before'):::action-->B16 - B16 --> B17(copy '.files')-->B18 - B18(load Zarf State)-->B19 - B19(push '.images')-->B20 - B20(push '.repos')-->B21 - B21(process '.dataInjections')-->B22 - B22(install '.charts')-->B23 - B23(apply '.manifests')-->B24 - B24(run each '.actions.onDeploy.after'):::action-->B24 - B24-->B25{Success?} - B25-->|Yes|B26(run each\n'.actions.onDeploy.success'):::action-->B26 - B25-->|No|B27(run each\n'.actions.onDeploy.failure'):::action-->B27-->B999 - - B999[Abort]:::fail - end - - B26-->B28(print Zarf connect table) - B28-->B29(save package data to cluster) - - - classDef prompt fill:#4adede,color:#000000 - classDef action fill:#bd93f9,color:#000000 - classDef fail fill:#aa0000 -`}/> -
From f643c090c6e617a6d208f29fae6cb3a49a4f8588 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 01:37:33 -0500 Subject: [PATCH 05/11] update docs for building injector Signed-off-by: razzle --- Makefile | 2 +- src/injector/Makefile | 8 ++++++-- src/injector/README.md | 36 ++++++++++++++++++++---------------- 3 files changed, 27 insertions(+), 19 deletions(-) diff --git a/Makefile b/Makefile index 6adc76e641..db74c92971 100644 --- a/Makefile +++ b/Makefile @@ -171,7 +171,7 @@ build-examples: ## Build all of the example packages @test -s ./build/zarf-package-component-webhooks-$(ARCH)-0.0.1.tar.zst || $(ZARF_BIN) package create examples/component-webhooks -o build -a $(ARCH) --confirm build-injector-linux: ## Build the Zarf injector for AMD64 and ARM64 - docker run --rm --user "$(id -u)":"$(id -g)" -v $$PWD/src/injector:/usr/src/zarf-injector -w /usr/src/zarf-injector rust:1.71.0-bookworm make build-injector-linux + docker run --rm --user "$(id -u)":"$(id -g)" -v $$PWD/src/injector:/usr/src/zarf-injector -w /usr/src/zarf-injector rust:1.71.0-bookworm make build-injector-linux list-sizes ## NOTE: Requires an existing cluster or the env var APPLIANCE_MODE=true .PHONY: test-e2e diff --git a/src/injector/Makefile b/src/injector/Makefile index 8f40fbadbf..a0ba3f467e 100644 --- a/src/injector/Makefile +++ b/src/injector/Makefile @@ -24,8 +24,6 @@ build-injector-linux-amd: ## Build the Zarf injector for AMD64 cargo build --target x86_64-unknown-linux-musl --release; \ fi - du --si target/x86_64-unknown-linux-musl/release/zarf-injector - build-injector-linux-arm: ## Build the Zarf injector for ARM64 rustup target add aarch64-unknown-linux-musl @@ -38,4 +36,10 @@ build-injector-linux-arm: ## Build the Zarf injector for ARM64 cargo build --target aarch64-unknown-linux-musl --release; \ fi +list-sizes: ## List the sizes of the Zarf injector binaries + @echo '\n\033[0;36mSize of Zarf injector binaries:\033[0m\n'; \ + du --si target/x86_64-unknown-linux-musl/release/zarf-injector; \ du --si target/aarch64-unknown-linux-musl/release/zarf-injector + +build-with-docker: ## Build the Zarf injector using Docker + docker run --rm --user "$(id -u)":"$(id -g)" -v $$PWD:/usr/src/zarf-injector -w /usr/src/zarf-injector rust:1.71.0-bookworm make build-injector-linux diff --git a/src/injector/README.md b/src/injector/README.md index 699be652ed..79505519a2 100644 --- a/src/injector/README.md +++ b/src/injector/README.md @@ -1,20 +1,21 @@ # zarf-injector -A tiny (<1MiB) binary statically-linked with musl in order to fit as a configmap +A tiny (<1MiB) binary statically-linked with musl in order to fit as a configmap. -## Building on Ubuntu +See how it gets used during the [`zarf-init`](https://docs.zarf.dev/commands/zarf_init/) process in the ['init' package reference documentation](https://docs.zarf.dev/ref/init-package/). + +## Building in Docker (recommended) ```bash -# install rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path -source $HOME/.cargo/env +make build-with-docker +``` + +## Building on Debian-based Systems -# install build-essential -sudo apt install build-essential -y +Install [Rust](https://rustup.rs/) and `build-essential`. -# build w/ musl -rustup target add x86_64-unknown-linux-musl -cargo build --target x86_64-unknown-linux-musl --release +```bash +make build-injector-linux list-sizes ``` ## Checking Binary Size @@ -22,12 +23,15 @@ cargo build --target x86_64-unknown-linux-musl --release Due to the ConfigMap size limit (1MiB for binary data), we need to make sure the binary is small enough to fit. ```bash -cargo build --target x86_64-unknown-linux-musl --release +make list-sizes +``` + +```sh +$ make build-with-docker +... -cargo build --target aarch64-unknown-linux-musl --release +Size of Zarf injector binaries: -size_linux=$(du --si target/x86_64-unknown-linux-musl/release/zarf-injector | cut -f1) -echo "Linux binary size: $size_linux" -size_aarch64=$(du --si target/aarch64-unknown-linux-musl/release/zarf-injector | cut -f1) -echo "aarch64 binary size: $size_aarch64" +840k target/x86_64-unknown-linux-musl/release/zarf-injector +713k target/aarch64-unknown-linux-musl/release/zarf-injector ``` From 545592250f526d9e271365ed8df674456b385e11 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 01:48:18 -0500 Subject: [PATCH 06/11] update docs Signed-off-by: razzle --- src/injector/README.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/src/injector/README.md b/src/injector/README.md index 79505519a2..0310ef0467 100644 --- a/src/injector/README.md +++ b/src/injector/README.md @@ -1,9 +1,28 @@ # zarf-injector +> If using VSCode w/ the official Rust extension, make sure to open a new window in the `src/injector` directory to make `rust-analyzer` happy. +> +> ```bash +> code src/injector +> ``` + A tiny (<1MiB) binary statically-linked with musl in order to fit as a configmap. See how it gets used during the [`zarf-init`](https://docs.zarf.dev/commands/zarf_init/) process in the ['init' package reference documentation](https://docs.zarf.dev/ref/init-package/). +## What does it do? + +```sh +zarf-injector +``` + +The `zarf-injector` binary serves 2 purposes during 'init'. + +1. It re-assembles a multi-part tarball that was split into multiple ConfigMap entries (located at `./zarf-payload-*`) back into `payload.tar.gz`, then extracts it to the `/zarf-seed` directory. It also checks that the SHA256 hash of the re-assembled tarball matches the first (and only) argument provided to the binary. +2. It runs a pull-only, insecure, HTTP OCI compliant registry server on port 5000 that serves the contents of the `/zarf-seed` directory (which is of the OCI layout format). + +This enables a distro-agnostic way to inject real `registry:2` image into a running cluster, thereby enabling air-gapped deployments. + ## Building in Docker (recommended) ```bash From 5ac0e6634f392010fe76dae7fc87a53cbe485a16 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 02:18:48 -0500 Subject: [PATCH 07/11] cleanup todos Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index 7a4129a078..a81b5c6374 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -28,8 +28,6 @@ View all init options w/ [`zarf init --help`](/commands/zarf_init/). ::: -{/* The 'init' package initializes a cluster with the requisite air gap services, allowing future Zarf Packages to store any required resources (i.e. container images, OCI artifacts, and git repositories) so that they can be accessed by the cluster. */} - ## Core Components An 'init' package requires a series of specially named, and configured components to ensure the cluster is correctly initialized. These components are: @@ -99,8 +97,6 @@ To see a full list of `variables` you can view the [`zarf.yaml` that defines the #### Using External Registries -{/* TODO: finish this section, and provide an example? */} - Zarf can be configured to use an already existing registry with the `--registry-*` flags when running [`zarf init`](/commands/zarf_init/). This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it pushes any images contained in the package to the externally configured registry. @@ -175,8 +171,6 @@ Zarf will refuse to adopt the Kubernetes [initial namespaces](https://kubernetes Additionally, when adopting resources, ensure that the namespaces specified are dedicated to Zarf, or add the `zarf.dev/agent: ignore` label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise [ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) errors may occur. -{/* TODO: should prob make sure there is a tech-debt issue to make Flux ns creation better */} - The Agent does not need to create any secrets in the cluster. Instead, during `zarf init` and `zarf package deploy`, secrets are automatically created in a [Helm Postrender Hook](https://helm.sh/docs/topics/advanced/#post-rendering) for any namespaces Zarf sees. If you have resources managed by [Flux](https://fluxcd.io/) that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you. ## Optional Components From 545200cf0cc601d57e7173e7de50668da53ce3f6 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 12:59:51 -0500 Subject: [PATCH 08/11] Update site/src/content/docs/ref/init-package.mdx Co-authored-by: Lucas Rodriguez --- site/src/content/docs/ref/init-package.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index a81b5c6374..90dbe54f8f 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -63,7 +63,11 @@ When `zarf init` deploys the `zarf-seed-registry` component, the following happe 2. Zarf connects to the cluster and grabs a pod that is running an image that is already present in the cluster. 3. Zarf spins up a pod using the existing image, mounts the `configmaps` that contain the `zarf-injector` binary and the `registry:2` image chunks and runs the `zarf-injector` binary. -> Doing this keeps Zarf cluster agnostic however does require that the kubelet be able to reach out to a cluster NodePort service which may require changes to firewall configurations like allowing UDP traffic between nodes if using something like VXLAN tunneling. +:::note + +Doing this keeps Zarf cluster agnostic, however does require that the kubelet be able to reach out to a cluster NodePort service, which may require changes to firewall configurations like allowing UDP traffic between nodes if using something like VXLAN tunneling. + +::: 4. The `zarf-injector` binary reassembles the `registry:2` image and hosts a temporary registry that the cluster can pull from. 5. The `docker-registry` chart in the `zarf-seed-registry` component is then deployed, with its `image.repository` set to the temporary registry that the `zarf-injector` binary is hosting (consumed as the `###ZARF_SEED_REGISTRY###` built-in variable set at runtime). From 7208a4bbef7114cb3ff7646c3bf3ef208bb91e68 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 13:00:05 -0500 Subject: [PATCH 09/11] Update site/src/content/docs/ref/init-package.mdx Co-authored-by: Lucas Rodriguez --- site/src/content/docs/ref/init-package.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index 90dbe54f8f..66323c0b8c 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -47,7 +47,7 @@ To ensure that our approach is distro-agnostic, the Zarf team developed a unique Shoving random data into a cluster is generally a bad idea, and an antipattern overall to containerization. However in the case of Zarf, and air-gapped environments, certain liberties must be taken. -While there is no distro-agnostic to inject images into a cluster, every cluster has support for `configmaps`. However, the size of a `configmap` is limited to 1MB (technically only limited by whatever is configured in `etcd`, the default is 1MB), and the `registry:2` image is around 10MB (as of this writing). So we split the `registry:2` into chunks and inject them into the cluster as `configmaps`. +While there is no distro-agnostic to inject images into a cluster, every cluster has support for `configmaps`. However, the size of a `configmap` is limited to 1MB (technically only limited by whatever is configured in `etcd`, the default is 1MB), and the `registry:2` image is around 10MB (as of this writing). So we split the `registry:2` image into chunks and inject them into the cluster as `configmaps`. But then we have another problem of how to reassemble the image on the other side, as we don't have any consistent image that exists in the cluster that would have such utilities. This is where the `zarf-injector` Rust binary comes in. From 73c52278139c81f6866f7e6a6a182afbfb5c7994 Mon Sep 17 00:00:00 2001 From: razzle Date: Fri, 10 May 2024 13:04:45 -0500 Subject: [PATCH 10/11] argo Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index 66323c0b8c..ebe12bc5c0 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -47,7 +47,7 @@ To ensure that our approach is distro-agnostic, the Zarf team developed a unique Shoving random data into a cluster is generally a bad idea, and an antipattern overall to containerization. However in the case of Zarf, and air-gapped environments, certain liberties must be taken. -While there is no distro-agnostic to inject images into a cluster, every cluster has support for `configmaps`. However, the size of a `configmap` is limited to 1MB (technically only limited by whatever is configured in `etcd`, the default is 1MB), and the `registry:2` image is around 10MB (as of this writing). So we split the `registry:2` image into chunks and inject them into the cluster as `configmaps`. +While there is no distro-agnostic method to inject images into a cluster, every cluster has support for `configmaps`. However, the size of a `configmap` is limited to 1MB (technically only limited by whatever is configured in `etcd`, the default is 1MB), and the `registry:2` image is around 10MB (as of this writing). So we split the `registry:2` image into chunks and inject them into the cluster as `configmaps`. But then we have another problem of how to reassemble the image on the other side, as we don't have any consistent image that exists in the cluster that would have such utilities. This is where the `zarf-injector` Rust binary comes in. @@ -157,7 +157,7 @@ This allows the cluster to pull images from the Zarf Registry instead of the int The `zarf-agent` also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. -{/* TODO: document argo here as well */} +> Support for mutating `Application` and `Repository` objects in ArgoCD is in [`beta`](/roadmap#beta) and can be tested on non-production clusters. :::note From 89e306f8d4d64c275606648ad08df2b45d142652 Mon Sep 17 00:00:00 2001 From: razzle Date: Mon, 13 May 2024 11:05:52 -0500 Subject: [PATCH 11/11] feedback from review Signed-off-by: razzle --- site/src/content/docs/ref/init-package.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/site/src/content/docs/ref/init-package.mdx b/site/src/content/docs/ref/init-package.mdx index ebe12bc5c0..c2f98e1e6e 100644 --- a/site/src/content/docs/ref/init-package.mdx +++ b/site/src/content/docs/ref/init-package.mdx @@ -157,7 +157,7 @@ This allows the cluster to pull images from the Zarf Registry instead of the int The `zarf-agent` also modifies [Flux GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) objects to point to the local Git Server. -> Support for mutating `Application` and `Repository` objects in ArgoCD is in [`beta`](/roadmap#beta) and can be tested on non-production clusters. +> Support for mutating `Application` and `Repository` objects in ArgoCD is in [`beta`](/roadmap#beta) and should tested on non-production clusters before being deployed to production clusters. :::note @@ -263,15 +263,12 @@ components: :::note -In order to reproduce the following example, you will need to have the Zarf repository cloned locally. +In order to reproduce / build the following example, you will need to have the Zarf repository cloned locally. ```bash git clone https://github.com/defenseunicorns/zarf.git cd zarf mv zarf.yaml zarf.yaml.bak -cat < zarf.yaml -... -EOF ``` You can learn more about creating a custom init package in the [Creating a Custom 'init' Package Tutorial](/tutorials/8-custom-init-packages). @@ -284,6 +281,7 @@ A minimal `zarf.yaml` for the 'init' package looks something like: # zarf.yaml kind: ZarfInitConfig metadata: + name: init components: - name: zarf-injector