Proposal for a Helm Deployment goal implementation (pantsbuild#15882)

Implementation for the proposal of a Helm Deployment goal. Design document can be found in:

https://docs.google.com/document/d/17wZRjiIxM5918ybC_0EnKY42qwd47-1ZporBdua8vRQ/edit?usp=sharing

docs/markdown/Helm/helm-deployments.md

@@ -0,0 +1,208 @@
+---
+title: "Deployments"
+slug: "helm-deployments"
+hidden: false
+createdAt: "2022-07-19T13:16:00.000Z"
+updatedAt: "2022-07-19T13:16:00.000Z"
+---
+> 🚧 Helm deployment support is in alpha stage
+> 
+> Pants has experimental support for managing deployments via the `experimental-deploy` goal. Helm deployments provides with a basic implementation of this goal.
+> 
+> Please share feedback for what you need to use Pants with your Helm deployments by either [opening a GitHub issue](https://github.com/pantsbuild/pants/issues/new/choose) or [joining our Slack](doc:getting-help)!
+
+Motivation
+----------
+
+Helm's ultimate purpose is to simplify the deployment of Kubernetes resources and help in making these reproducible. However it is quite common to deploy the same software application into different kind of environments using slightly different configuration overrides.
+
+This hinders reproducibility since operators end up having a set of configuration files and additional shell scripts that ensure that the Helm command line usued to deploy a piece of software into a given environment is always the same.
+
+Pants solves this problem by providing with the ability to manage the configuration files and the different parameters of a deployment as single unit such that a simple command line as `./pants experimental-deploy ::` will always have the same effect on each of the deployments previously defined.
+
+Defining Helm deployments
+-------------------------
+
+Helm deployments are defined using the `helm_deployment` target which has a series of fields that can be used to guarantee the reproducibility of the given deployment. `helm_deployment` targets need to be added by hand as there is no deterministic way of instrospecting your repository to find sources that are specific to Helm:
+
+```text src/chart/BUILD
+helm_chart()
+```
+```yaml src/chart/Chart.yaml
+apiVersion: v2
+description: Example Helm chart
+name: example
+version: 0.1.0
+```
+```text src/deployment/BUILD
+helm_deployment(name="dev", sources=["common-values.yaml", "dev-override.yaml"], dependencies=["//src/chart"])
+
+helm_deployment(name="stage", sources=["common-values.yaml", "stage-override.yaml"], dependencies=["//src/chart"])
+
+helm_deployment(name="prod", sources=["common-values.yaml", "prod-override.yaml"], dependencies=["//src/chart"])
+```
+```yaml src/deployment/common-values.yaml
+# Default values common to all deployments
+env:
+  SERVICE_NAME: my-service
+```
+```yaml src/deployment/dev-override.yaml
+# Specific values to the DEV environment
+env:
+  ENV_ID: dev
+```
+```yaml src/deployment/stage-override.yaml
+# Specific values to the STAGE environment
+env:
+  ENV_ID: stage
+```
+```yaml src/deployment/prod-override.yaml
+# Specific values to the PRODUCTION environment
+env:
+  ENV_ID: prod
+```
+
+There are quite a few things to notice in the previous example:
+
+* The `helm_deployment` target requires you to explicitly define as a dependency which chart to use.
+* We have three different deployments that using configuration files with the specified chart.
+* One of those value files (`common-values.yaml`) provides with default values that are common to all deployments.
+* Each deployment uses an additional `xxx-override.yaml` file with values that are specific to the given deployment.
+
+> 📘 Source roots
+> 
+> Don't forget to configure your source roots such that each of the shown files in the previous example sit at their respective source root level.
+
+The `helm_deployment` target has many additional fields including the target kubernetes namespace, adding inline override values (similar to using helm's `--set` arg) and many others. Please run `./pants help helm_deployment` to see all the posibilities.
+
+Dependencies with `docker_image` targets
+----------------------------------------
+
+A Helm deployment will in most cases deploy one or more Docker images into Kubernetes. Furthermore, it's quite likely there is going to be at least a few first party Docker images among those. Pants is capable of analysing the Helm chart being used in a deployment to detect those required first-party Docker images using Pants' target addresses to those Docker images.
+
+To illustrate this, let's imagine the following scenario: Let's say we have a first-party Docker image that we want to deploy into Kubernetes as a `Pod` resource kind. For achieving this we define the following workspace:
+
+```text src/docker/BUILD
+docker_image()
+```
+```text src/docker/Dockerfile
+FROM busybox:1.28
+```
+```text src/chart/BUILD
+helm_chart()
+```
+```yaml src/chart/Chart.yaml
+apiVersion: v2
+description: Example Helm chart
+name: example
+version: 0.1.0
+```
+```yaml src/chart/values.yaml
+# Default image in case this chart is used by other tools after being published
+image: example.com/registry/my-app:latest
+```
+```yaml src/chart/templates/pod.yaml
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: my_pod
+  labels:
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
+spec:
+  containers:
+    - name: my_app
+      # Uses the `image` value entry from the deployment inputs
+      image: {{ .Values.image }}
+```
+```text src/deployment/BUILD
+# Overrides the `image` value for the chart using the target address for the first-party docker image.
+helm_deployment(dependencies=["src/chart"], values={"image": "src/docker"})
+```
+
+> 📘 Docker image references VS Pants' target addresses
+> 
+> You should use typical Docker registry addresses in your Helm charts. Because Helm charts are distributable artifacts and may be used with tools other than Pants, you should create your charts such that when that chart is being used, all Docker image addresses are valid references to images in accessible Docker registries. As shown in the example, we recommend that you make the image address value configurable, especially for charts that deploy first-party Docker images.
+> Your chart resources can still use off-the-shelf images published with other means, and in those cases you will also be referencing the Docker image address. Usage of Pants' target addresses is intended for your own first-party images because the image reference of those is not known at the time we create the sources (they are computed later).
+
+With this setup we should be able to run `./pants dependencies src/deployment` and Pants should give the following output:
+
+```text
+src/chart
+src/docker
+```
+
+This should work with any kind of Kubernetes resource that leads to Docker image being deployed into Kubernetes, such as `Deployment`, `StatefulSet`, `ReplicaSet`, `CronJob`, etc. Please get in touch with us in case you find Pants was not capable to infer dependencies in any of your `helm_deployment` targets by either [opening a GitHub issue](https://github.com/pantsbuild/pants/issues/new/choose) or [joining our Slack](doc:getting-help).
+
+> 📘 How the Docker image reference is calculated during deployment?
+> 
+> Pants' will rely on the behaviour of the `docker_image` target when it comes down to generate the final image reference. Since a given image may have more than one valid image reference, **Pants will try to use the first one that is not tagged as `latest`**, falling back to `latest` if none could be found.
+> It's good practice to publish your Docker images using tags other than `latest` and Pants preferred behaviour is to choose those as this guarantees that the _version_ of the Docker image being deployed is the expected one.
+
+Value files
+-----------
+
+It's very common that Helm deployments use a series of files providing with values that customise the given chart. When using deployments that may have more than one YAML file as the source of configuration values, the Helm backend needs to sort the file names in a way that is consistent across different machines, as the order in which those files are passed to the Helm command is relevant. The final order depends on the same order in which those files are specified in the `sources` field of the `helm_deployment` target. For example, given the following `BUILD` file:
+
+```text src/deployment/BUILD
+helm_deployment(name="dev", dependencies=["//src/chart"], sources=["first.yaml", "second.yaml", "last.yaml"])
+```
+
+This will result in the Helm command receiving the value files as in that exact order.
+
+If using any glob pattern in the `sources` field, the plugin will first group the files according to the order in which those glob patterns are listed. In this grouping, files that are resolved by more than one pattern will be part of the most specific group. Then we use alphanumeric ordering for the files that correspond to each of the previous groups. To illustrate this scenario, consider the following list of files:
+
+```
+src/deployment/002-config_maps.yaml
+src/deployment/001-services.yaml
+src/deployment/first.yaml
+src/deployment/dev/daemon_sets.yaml
+src/deployment/dev/services-override.yaml
+src/deployment/last.yaml
+```
+
+And also the following `helm_deployment` target definition:
+
+```text src/deployment/BUILD
+helm_deployment(name="dev", dependencies=["//src/chart"], sources=["first.yaml", "*.yaml", "dev/*-override.yaml", "dev/*.yaml", "last.yaml"])
+```
+
+In this case, the final ordering of the files would be as follows:
+
+```
+src/deployment/first.yaml
+src/deployment/001-services.yaml
+src/deployment/002-config_maps.yaml
+src/deployment/dev/services-override.yaml
+src/deployment/dev/daemon_sets.yaml
+src/deployment/last.yaml
+```
+
+We believe that this approach gives a very consistent and predictable ordering while at the same time total flexibility to the end user to organise their files as they best fit each particular case of a deployment.
+
+In addition to value files, you can also use inline values in your `helm_deployment` targets by means of the `values` field. All inlines values that are set this way will override any entry that may come from value files.
+
+Deploying
+---------
+
+Continuing with the example in the previous section, we can deploy it into Kubernetes using the command `./pants experimental-deploy src/deployment`. This will trigger the following steps:
+
+1. Analyse the dependencies of the given deployment.
+2. Build and publish any first-party Docker image and Helm charts that are part of those dependencies.
+3. Post-process the Kubernetes manifests generated by Helm by replacing all references to first-party Docker images by their real final registry destination.
+4. Initiate the deployment of the final Kubernetes resources resulting from the post-processing.
+
+The `experimental-deploy` goal also supports default Helm pass-through arguments that allow to change the deployment behaviour to be either atomic or a dry-run or even what is the Kubernetes config file (the `kubeconfig` file) and target context to be used in the deployment.
+
+Please note that the list of valid pass-through arguments has been limited to those that do not alter the reproducibility of the deployment (i.e. `--create-namespace` is not a valid pass-through argument). Those arguments will have equivalent fields in the `helm_deployment` target.
+
+For example, to make an atomic deployment into a non-default Kubernetes context you can use a command like the following one:
+
+```
+./pants experimental-deploy src/deployments:prod -- --kube-context my-custom-kube-context --atomic
+```
+
+> 📘 How does Pants authenticate with the Kubernetes cluster?
+>
+> Short answer is: it doesn't. 
+> Pants will invoke Helm under the hood with the appropriate arguments to only perform the deployment. Any authentication steps that may be needed to perform the given deployment have to be done before invoking the `experimental-deploy` goal. If you are planning to run the deployment procedure from your CI/CD pipelines, ensure that all necessary preliminary steps (including authentication with the cluster) are done before the one that triggers the deployment.

docs/markdown/Helm/helm-overview.md

@@ -53,7 +53,7 @@ Adding `helm_chart` targets
 
 Helm charts are identified by the presence of a `Chart.yaml` or `Chart.yml` file, which contains relevant metadata about the chart like its name, version, dependencies, etc. To get started quickly you can create a simple `Chart.yaml` file in your sources folder:
 
-```text Chart.yaml
+```yaml Chart.yaml
 apiVersion: v2
 description: Example Helm chart
 name: example

pants.toml

@@ -119,9 +119,13 @@ venv_use_symlinks = true
 # + src/python/pants/testutil:testutil_wheel
 interpreter_constraints = [">=3.7,<3.10"]
 macos_big_sur_compatibility = true
-resolves = { python-default = "3rdparty/python/user_reqs.lock" }
 enable_resolves = true
 
+[python.resolves]
+python-default = "3rdparty/python/user_reqs.lock"
+helm-post-renderer = "src/python/pants/backend/helm/subsystems/post_renderer.lock"
+helm-k8s-parser = "src/python/pants/backend/helm/subsystems/k8s_parser.lock"
+
 [python-infer]
 assets = true
 unowned_dependency_behavior = "error"

src/python/pants/backend/docker/register.py

@@ -5,13 +5,15 @@
 from pants.backend.docker.goals.tailor import rules as tailor_rules
 from pants.backend.docker.rules import rules as docker_rules
 from pants.backend.docker.target_types import DockerImageTarget
+from pants.backend.docker.target_types import rules as target_types_rules
 
 
 def rules():
     return (
         *docker_rules(),
         *export_codegen_goal.rules(),
         *tailor_rules(),
+        *target_types_rules(),
     )
 
 

src/python/pants/backend/docker/target_types.py

@@ -15,8 +15,10 @@
 from pants.core.goals.run import RestartableField
 from pants.engine.addresses import Address
 from pants.engine.fs import GlobMatchErrorBehavior
+from pants.engine.rules import collect_rules, rule
 from pants.engine.target import (
     COMMON_TARGET_FIELDS,
+    AllTargets,
     AsyncFieldMixin,
     BoolField,
     Dependencies,
@@ -26,6 +28,7 @@
     StringField,
     StringSequenceField,
     Target,
+    Targets,
 )
 from pants.util.docutil import bin_name, doc_url
 from pants.util.strutil import softwrap
@@ -369,3 +372,18 @@ class DockerImageTarget(Target):
 
         """
     )
+
+
+class AllDockerImageTargets(Targets):
+    pass
+
+
+@rule
+def all_docker_targets(all_targets: AllTargets) -> AllDockerImageTargets:
+    return AllDockerImageTargets(
+        [tgt for tgt in all_targets if tgt.has_field(DockerImageSourceField)]
+    )
+
+
+def rules():
+    return collect_rules()

src/python/pants/backend/experimental/helm/register.py

@@ -1,36 +1,37 @@
 # Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
-from pants.backend.helm.goals import lint, package, publish, tailor
+from pants.backend.helm.dependency_inference import deployment
+from pants.backend.helm.goals import deploy, lint, package, publish, tailor
 from pants.backend.helm.target_types import (
     HelmArtifactTarget,
     HelmChartTarget,
+    HelmDeploymentTarget,
     HelmUnitTestTestsGeneratorTarget,
     HelmUnitTestTestTarget,
 )
 from pants.backend.helm.target_types import rules as target_types_rules
-from pants.backend.helm.test.unittest import rules as test_rules
-from pants.backend.helm.util_rules import chart, sources, tool
+from pants.backend.helm.test.unittest import rules as unittest_rules
 
 
 def target_types():
     return [
         HelmArtifactTarget,
         HelmChartTarget,
+        HelmDeploymentTarget,
         HelmUnitTestTestTarget,
         HelmUnitTestTestsGeneratorTarget,
     ]
 
 
 def rules():
     return [
-        *chart.rules(),
         *lint.rules(),
+        *deploy.rules(),
+        *deployment.rules(),
         *package.rules(),
         *publish.rules(),
         *tailor.rules(),
-        *test_rules(),
-        *sources.rules(),
-        *tool.rules(),
+        *unittest_rules(),
         *target_types_rules(),
     ]

src/python/pants/backend/helm/dependency_inference/chart.py

@@ -16,6 +16,7 @@
     HelmChartMetaSourceField,
     HelmChartTarget,
 )
+from pants.backend.helm.target_types import rules as helm_target_types_rules
 from pants.backend.helm.util_rules.chart_metadata import HelmChartDependency, HelmChartMetadata
 from pants.engine.addresses import Address
 from pants.engine.internals.selectors import Get, MultiGet
@@ -160,5 +161,6 @@ def rules():
     return [
         *collect_rules(),
         *artifacts.rules(),
+        *helm_target_types_rules(),
         UnionRule(InferDependenciesRequest, InferHelmChartDependenciesRequest),
     ]