Add OCI Support

[aknysh]

what

• Update to Go 1.21
• Update/improve Atmos help for atmos-specific commands
• Improve atmos vendor pull command error handling
• Add Atmos custom commands to atmos.yaml and docs
• Add examples of Rego policies for Atmos components and stacks validation. Update docs. See https://pr-437.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/components/validation/
• Add Sprig functions support to Go templates in Atmos vendoring
• Improve context functionality in imports with Go templates. Update docs. See https://pr-437.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/stacks/imports/
• Add OCI support to atmos vendor command

why

• Update to Go 1.21.
  1.21 is the latest Go version with new features and improvements

• Update/improve Atmos help for atmos-specific commands.
  Atmos supports Terraform commands that are not Terraform native. Update the commands help to better describe the commands and don't call the native Terraform help on them (since it fails). For example:

atmos terraform clean --help

'atmos terraform clean' command deletes the following folders and files from the component's directory:

 - '.terraform' folder
 - folder that the 'TF_DATA_DIR' ENV var points to
 - '.terraform.lock.hcl' file
 - generated varfile for the component in the stack
 - generated planfile for the component in the stack
 - generated 'backend.tf.json' file

Usage: atmos terraform clean <component> -s <stack> <flags>

Use '--skip-lock-file' flag to skip deleting the lock file.

For more details refer to https://atmos.tools/cli/commands/terraform/clean

atmos terraform shell --help

'atmos terraform shell' command starts a new SHELL configured with the environment for an Atmos component in a Stack to allow executing all native terraform commands
inside the shell without using the atmos-specific arguments and flags.

Usage: atmos terraform shell <component> -s <stack>

The command does the following:

 - Processes the stack config files, generates the required variables for the Atmos component in the stack, and writes them to a file in the component's folder
 - Generates a backend config file for the Atmos component in the stack and writes it to a file in the component's folder (or as specified by the Atmos configuration setting)
 - Creates a Terraform workspace for the component in the stack
 - Drops the user into a separate shell (process) with all the required paths and ENV vars set
 - Inside the shell, the user can execute all Terraform commands using the native syntax

For more details refer to https://atmos.tools/cli/commands/terraform/shell

• Improve atmos vendor pull command error handling.
  If a user specifies the flags incorrectly, the command now shows a more descriptive error message. For example, executing atmos vendor pull infra/vpc2 will generate the following error message:

• Add Atmos custom commands to atmos.yaml and docs
  The new custom commands extend Atmos functionality. Especially useful is the atmos describe eks upgrade <eks_component> -s <stack> command that shows all the steps that needs to be performed to upgrade an EKS cluster to the next Kubernetes version, including checking for failing Pods and containers, EKS add-ons versions, and for the deleted k8s API versions (which need to be fixed by updating the Helm Charts before upgrading to the next Kubernetes version)

For example, to describe the steps to upgrade the eks Atmos component in the stack tenant1-ue1-dev, execute the following command:

atmos describe eks upgrade eks -s tenant1-ue1-dev

• Add examples of Rego policies for Atmos components and stacks validation. Update docs.
  Some useful examples of OPA Rego policies to validate Atmos components and stacks added to the examples and described in the docs

• Add Sprig functions support to Go templates in Atmos vendoring.
  The Sprig library provides over 70 template functions for Go’s template language.
  Atmos supported the Sprig functions in imports. This update adds it to Atmos vendoring and the functions can be used in the attributes in the component.yaml vendoring config file

• Improve context functionality in imports with Go templates.
  Before this update, imports with Go templates could use only one schema at the time in one file, either a list of strings (paths to the imported stack manifests), or a list of objects with context. Now, both schemas can be mixed in one import. For example:

import:
  - path: mixins/region/region_tmpl
    context:
      region: "{{ .region }}"
      environment: "{{ .environment }}"

  - path: "orgs/cp/{{ .tenant }}/{{ .stage }}/_defaults"
  - "catalog/eks/cluster/defaults"
  - "catalog/eks/karpenter/defaults"

Also, imports with Go templates now support two new attributes:

• skip_templates_processing - (boolean) Skip template processing for the imported file. Can be used if the imported file uses Go templates to configure external systems, e.g. Datadog

• ignore_missing_template_values - (boolean) Ignore the missing template values in the imported file. Can be used if the imported file uses Go templates to configure external systems, e.g. Datadog. In this case, Atmos will process all template values that are provided in the context, and will skip the missing values in the templates for the external systems without throwing an error. The ignore_missing_template_values setting is different from skip_templates_processing in that skip_templates_processing skips the template processing completely in the imported file, while ignore_missing_template_values processes the templates using the values provided in the context and skips all the missing values

For example:

import:
    - path: "<path_to_imported_file>"
      context: {}
      skip_templates_processing: true
      ignore_missing_template_values: false
    - path: "<path_to_imported_file>"
      context: {}
      skip_templates_processing: false
      ignore_missing_template_values: true

• Add OCI support to atmos vendor command

Atmos now supports vendoring components from OCI registries.

To specify a repository in an OCI registry, use the oci://<registry>/<repository>:tag scheme in the sources and mixins.

Components from OCI repositories are downloaded as Docker image tarballs, then all the layers are processed, un-tarred and un-compressed, and the component's source files are written into the component's directory.

For example, to vendor the vpc component from the public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc
AWS public ECR registry, use the following uri:

uri: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:latest"

The schema of a component.yaml file is as follows:

# This is an example of how to download a Terraform component from an OCI registry (https://opencontainers.org), e.g. AWS Public ECR

# 'component.yaml' in the component folder is processed by the 'atmos' commands:
# 'atmos vendor pull -c infra/vpc2' or 'atmos vendor pull --component infra/vpc2'

apiVersion: atmos/v1
kind: ComponentVendorConfig
metadata:
  name: stable/aws/vpc
  description: Config for vendoring of the 'stable/aws/vpc' component
spec:
  source:
    # Source 'uri' supports the following protocols: OCI (https://opencontainers.org), Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP,
    # and all URL and archive formats as described in https://github.com/hashicorp/go-getter
    # In 'uri', Golang templates are supported  https://pkg.go.dev/text/template
    # If 'version' is provided, '{{.Version}}' will be replaced with the 'version' value before pulling the files from 'uri'
    # Download the component from the AWS public ECR registry (https://docs.aws.amazon.com/AmazonECR/latest/public/public-registries.html)
    uri: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:{{.Version}}"
    version: "latest"
    # Only include the files that match the 'included_paths' patterns
    # If 'included_paths' is not specified, all files will be matched except those that match the patterns from 'excluded_paths'
    # 'included_paths' support POSIX-style Globs for file names/paths (double-star `**` is supported)
    # https://en.wikipedia.org/wiki/Glob_(programming)
    # https://github.com/bmatcuk/doublestar#patterns
    included_paths:
      - "**/*.*"

When the following command is executed

atmos vendor pull -c infra/vpc2

Atmos downloads the Docker image tarball from the AWS public ECR repository public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:latest, then all the layers are processed, un-tarred and un-compressed, and the component's source files are written into the component's directory.

Pulling sources for the component 'infra/vpc2' 
from 'public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:latest' 
into 'examples/complete/components/terraform/infra/vpc2'

[osterman]

Why are we changing from Trace to Info?

[aknysh]

b/c the command does not show anything at all, which is confusing and not clear if it did anything or not - you have to go to the file system and check all the files manually

[aknysh]

atmos terraform clean is for people, not for processes, so it should show meaningful outputs