A given cluster/environment should have a directory in git where all the kubernetes resources reside in a format as close to their natural Kubernetes YAML format. We default to the directory config-root but any folder will do.
We follow the naming conventions defined by Anthos Config Management such that within the cluster/environments directory in git:
cluster/contains all the sources which are cluster wide (like Custom Resource Definitions or these kubernetes resources:Namespace,ClusterRole,ClusterRoleBinding)namespaces/contains all the namespaced resources. We also use the first level directory to denote the directory name. So for resources in namespacefoomay be innamespaces/foo/myapp/deployment.yamlfor example.
Also note that any resource in a specific namespace will need the metadata.namespace property set to the namespace. Otherwise resources default to using the default namespace.
You can use the jx-gitops namespace command to set the namespaces on your kubernetes resources. Or to ensure namespaces are set based on the child directory name within namespaces/ use jx-gitops namespace --dir-mode --dir config-root/namespaces
Many tools can be used to fetch the YAML files from repositories and modify them such as any permutation of:
However we see those as tools that should be used at build time to generate the YAML and then check it into Git. The most important thing is to standise on what gets checked into git + that gets released: namely the standard kubernetes resources and custom resources.
We obviously don't want to commit raw kubernetes Secret YAML into git!
You can use sealed secrets.
We highly recommend using Kubernetes External Secrets which means you can check in the ExternalSecret resources into git which are a reference to the actual secret values from some provider:
- Alibaba Cloud KMS Secret Manager
- Amazon Secret Manager
- Amazon Parameter Store
- Azure Key Vault
- Google Secret Manager
- HashiCorp Vault
If you use jx-gitops extsecret jx-gitops helm template commands all of your kubernetes Secretresources will get automatically converted to ExternalSecret resources so you can safely check them into your git repository. You may also find the document on Secret Mapping useful
We should strive to support as many deployment tools as possible. The above layouts should work fine with tools such as:
kubectl apply- Tekton Pipelines using
kubectl apply - Flux
- Anthos Config Management
Proposing pull requests on your git repositories are an excellent way to get reviews from your team and to run any automated tooling to verify the resources.
We recommend your Pull Request pipeline should re-run the generation scripts and check in the output to your Pull Requests to ensure that the Pull Request is complete and folks can review it to see exactly what is going to be changed.
e.g. by running:
make build commitIf a Pull Request upgrades a version of a helm chart or kpt package with a simple one liner, seeing a second commit on the Pull Request with the actual changes to the kubernetes Deployment resource in terms of changes to images, volumes, environment variables and so forth is extremely useful.
This is not a requirement but using a Makefile to trigger the various tools like helm, kpt or kustomize can help provide the same CLI UX and use the same pipeline irrespective of which permutation of tools are used.
e.g. to fetch/build/enrich the YAML:
make all To apply into the current kubernetes cluster:
make apply We recommend you lint your YAML files to ensure a consistent layout. If using tools like helm we recommend splitting YAML files into a file per resource to simplify understanding and to make things easier to process with tools.
e.g. use jx-gitops split in your Makefile.