Merge pull request #3924 from monopole/architecture

Add ARCHITECTURE.md

ARCHITECTURE.md

@@ -0,0 +1,299 @@
+# Architecture
+
+* _Updated: December 2021_
+
+This document describes the repository organization and the kustomize
+build process. It's meant to lower the barrier to learning and
+contributing to the code base.
+
+If not kept up to date, it will just be a historical snapshot.
+
+
+## Repository layout
+
+[human-edited docs]: https://github.com/kubernetes-sigs/cli-experimental/tree/master/site
+[generated docs]: https://github.com/kubernetes-sigs/cli-experimental/tree/master/docs
+[rendered docs]: https://kubectl.docs.kubernetes.io
+[openapi]: https://kubernetes.io/blog/2016/12/kubernetes-supports-openapi
+
+[`api` module]: https://github.com/kubernetes-sigs/kustomize/blob/master/api/go.mod
+[`api`]: #the-api-module
+[`cmd/config` module]: https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/go.mod
+[`cmd/config`]: #the-cmdconfig-module
+[`kustomize` module]: https://github.com/kubernetes-sigs/kustomize/blob/master/kustomize/go.mod
+[`kustomize`]: #the-kustomize-module
+[`kyaml` module]: https://github.com/kubernetes-sigs/kustomize/blob/master/kyaml/go.mod
+[`kyaml`]: #the-kyaml-module
+[`kyaml/kio.Filter`]: https://github.com/Kubernetes-sigs/kustomize/blob/master/kyaml/kio/kio.go
+[`go-yaml`]: https://github.com/go-yaml/yaml/tree/v3
+
+
+[3922]: https://github.com/kubernetes-sigs/kustomize/issues/3922
+
+
+
+ |  directory  | purpose      |
+ | ---------:  | :---------- |
+ |       `api` | The [`api`] module, holding high level kustomize code, suitable for import by other programs.  |
+ |       `cmd` | Various Go programs aiding repo management. See also `hack`. As an outlier, includes the special [`cmd/config`] module. |
+ |      `docs` | Old home of documentation; contains pointers to new homes: [human-edited docs], [generated docs] and [rendered docs]. |
+ |  `examples` | Full kustomization examples that run as pre-merge tests.  |
+ | `functions` | Examples of plugins in KRM function form. TODO([3922]): Move under `plugin`. |
+ |      `hack` | Various shell scripts to help with code management. |
+ | `kustomize` | The [`kustomize`] module holds the `main.go` for kustomize. |
+ |     `kyaml` | The [`kyaml`] module, holding Kubernetes-specific YAML editing packages used by the [`api`] module. Wraps [`go-yaml`] v3.|
+ |    `plugin` | Examples of Kustomize plugins. |
+ | `releasing` | Instructions for releasing the various modules. |
+ |      `site` | Old generated documentation, kept to provide redirection links to the new docs. |
+
+
+## Modules
+
+[semantically versioned]: https://semver.org
+[Go modules]: https://github.com/golang/go/wiki/Modules
+
+The [Go modules] in the kustomize repository are [semantically versioned].
+
+
+### `kustomize`
+
+> _Depends on [`api`], [`cmd/config`], [`kyaml`]_
+
+The [`kustomize` module] contains the `main.go` for `kustomize`, buildable with
+
+```
+(cd kustomize; go install .)
+```
+
+[appears in kubectl]: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/kustomize/kustomize.go
+
+Below this are packages containing
+[cobra](http://github.com/spf13/cobra) commands implementing `build`,
+`edit`, `fix`, etc., packages linked together by `main.go`.
+
+These command packages are intentionally public, semantically
+versioned, and can be used in other programs.  Specifically, the
+`kustomize build` command [appears in kubectl] as `kubectl kustomize`.
+
+The code in the `build` package is dominated by flag validation,
+with minimal business logic.  The critical lines are something
+like
+
+```
+# Make a kustomizer.
+k := krusty.MakeKustomizer(
+  HonorKustomizeFlags(krusty.MakeDefaultOptions()),
+)
+
+# Run the kustomizer, sending location of kustomization.yaml
+m := k.Run(fSys, "/path/to/dir")
+
+# Write the result as YAML.
+writer.Write(m.AsYaml())
+```
+
+The `krusty` package is in the [`api`] module.
+
+### `api`
+
+> _Depends on [`kyaml`] and code generated from builtin plugin modules_
+
+The [`api` module] is used by CLI programs like `kustomize` and `kubectl`
+to read and honor `kustomization.yaml` files and all that implies.
+
+The main public packages in the [`api` module] are
+
+ | package   |             |
+ | --------: | :---------- |
+ | `filters` | Implementations of [`kyaml/kio.Filter`] used by kustomize to transform Kubernetes objects. |
+ |  `konfig` | Configuration methods and constants in the kustomize API. |
+ |  `krusty` | Primary API entry point.  Holds the kustomizer and hundreds of tests for it. |
+ |  `loader` | Loads kustomization files and the files they refer to, enforcing security rules. |
+ |  `resmap` | The primary internal data structure over which the kustomizer and filters work. |
+ |   `types` | The `Kustomization` object and ancillary structs. |
+
+### `cmd/config`
+
+> _Depends on [`kyaml`]_
+
+This module contains cobra commands and kyaml-based functionality to
+provide unix-like file manipulation commands to kustomize like `grep`
+and `tree`.  These commands may be included in any program that
+manipulates k8s YAML (e.g. kustomize).
+
+### `kyaml`
+
+> _Has no in-repo dependence_
+
+The [`kyaml` module] is a kubernetes-focussed enhancement of [go-yaml].
+
+The YAML manipulation performed by a kustomize is based on these libraries.
+
+These libraries evolve independently of kustomize, and other programs depend on them.
+
+The key public packages in the [`kyaml` module] include
+
+ | package   |             |
+ | --------: | :---------- |
+ | `errors` | Wrapper for the go-errors/errors lib |
+ | `filesys` | A kustomize-specific file system abstraction, to ease writing tests |
+ | `fn/framework` | An SDK for writing KRM Functions in Go |
+ | `fn/runtime` | Implements the runtime for KRM Function extensions |
+ | `kio` | Libraries for reading and writing collections of Kubernetes resources as RNodes |
+ | `openapi` | Loads and accesses openapi schemas for schema-aware resource manipultaion |
+ | `resid` | Representations to aid in unique identification of Kubernetes resources |
+ | `yaml` | A Kubernetes-focused wrapper of [go-yaml], notably including the RNode object |
+
+
+-------
+
+## How _kustomize build_ works
+
+The command `kustomize build` accepts a single string argument,
+which must resolve to a directory, possibly in a git repository,
+called the _kustomization root_.
+
+This directory must contain a file called `kustomization.yaml`, with
+YAML that marshals into a single instance of a `Kustomization` object.
+
+For the remainder of this document, the word _kustomization_ refers to
+either of these things.
+
+This kustomization is the access point to a directed, acyclic graph of
+Kubernetes objects, including other kustomizations, to include in a
+build.
+
+Execution of `build` starts and ends in the [`api`] module,
+frequently dipping into the [`kyaml`] module for lower level
+YAML manipulation.
+
+### The `build` flow
+
+  - Validate command lines arguments and flags.
+
+  - Make a `Kustomizer` as a function of those arguments.
+
+  - Call `Run` on the kustomizer, passing it the path to the
+    kustomization.
+
+    `Run` returns an instance of `ResMap`, the `api` package's
+    representation of a set of kubernetes `Resource` objects.
+
+    This structure offers resource lookup methods (map behavior),
+    but also retains the resources in the order they were
+    specified in kustomization files (list behavior).
+
+    Post-run, the objects are fully hydrated, per the
+    instructions in the kustomization.
+
+  - Marshal the objects as YAML to a file or `stdout`.
+
+
+### The `Run` function
+
+  - Create various objects
+
+    - A `ResMap` factory.
+
+      Makes  `ResMaps` from byte streams, other `ResMaps`, etc.
+
+    - A file `loader.Loader`.
+
+      It's fed an appropriate set of restrictions, and the path to the kustomization.
+
+    - A plugin loader.
+
+      It finds plugins (transformers, generators or validators)
+      and prepares them for running.
+
+    - A `KustTarget` encapsulating all of the above.
+
+      A KustTarget contains one `Kustomization` and represents
+	  everything that kustomization can reach.  This will include
+	  other `KustTarget` instances, each having a smaller purview than
+	  the one referencing it.
+
+  - Call `KustTarget.Load` to load its kustomization.
+
+    This step deals with deprecations and field changes.
+
+  - Load [openapi] data specified by the kustomization.
+
+    This is needed to recognize k8s kinds and their special
+    properties, e.g. which kinds are cluster-scoped, which kinds
+    refer to others, etc.
+
+  - Call `KustTarget.makeCustomizedResmap` to create the `ResMap` result.
+
+    This visits everything referenced by the kustomization,
+    performing all generation, transformation and validation.
+
+  - Finish the `Run` with
+
+    - Optional reordering of objects in `ResMap`, overriding the
+      FIFO rule.
+
+    - Optional addition of _kustomize build annotations_ to the
+      resources.  E.g. from which repo and file the resource was
+      read, the fact that kustomize touched the resource, etc.
+      These kustomize-specific annotations are intended for
+      server-side data analytics, file structure traceability and
+      reconstruction, etc.
+
+### The `makeCustomizedResmap` function
+
+  This function starts the process of object transformation,
+  as well as accumulation of recursively referenced data.
+
+  - Call `ra := KustTarget.AccumulateTarget`.
+
+    The result, `ra`, is a resource accumulator that contains
+    everything referred to by the current kustomization, now fully
+    hydrated.
+
+  - Uniquify names of generated objects by appending content hashes.
+
+    This cannot be done until the objects are complete.
+
+  - Fix all name references (given that names may have changed).
+
+    E.g. if a ConfigMaps was given a generated name, all objects that
+    refer to that ConfigMap must be given its name.
+
+  - Resolve vars, replacing them with whatever they refer to (a legacy feature).
+
+### The `AccumulateTarget` function
+
+  - Call `AccumulateResources` over the `resources` field (this can recurse).
+  - Call `AccumulateComponents` over the `components` field (this can recurse),
+  - Load legacy (pre-plugin) global kustomize configuration,
+  - Load legacy (pre-openapi) _Custom Resource Definition_ data.
+  - In the context of the data loaded above, run the kustomization's
+    - generators,
+    - transformers,
+    - and validators.
+  - Accumulate `vars` (make note of them for later replacement).
+
+### `AccumulateResources` and component accumulation
+
+  - If the path is a file:
+    - Accumulate the objects in the file (treating them
+      as opaque kubernetes objects).
+
+  - If the path is a directory:
+    - Create a new `KustTarget` referring to that directory's kustomization.
+	- Call `subRa := KustTarget.AccumulateTarget`.
+    - Call `ra.MergeAccumulator(subRa)`
+	This completes a recursion.
+
+  - If the path is a git URL:
+    - Clone the repository to a temporary directory.
+    - Process the path optionally specified in the URL
+      as a path in the clone.
+	- If no path specified, work from the repository root.
+
+
+That's as deep as this discussion will go.
+
+The deeper this document goes into the details, the faster
+it will get out of date.