Entigo Infralib Agent

Provision and continuously update a complete Kubernetes platform on AWS or Google Cloud from a single YAML file.

Documentation · Website · Modules · Command reference

In production since 2023. Tested nightly against live AWS and Google Cloud accounts. Used by Estonia's Information System Authority (RIA), the Health and Welfare Information Systems Centre (TEHIK).

What it does

You describe the platform you want in one YAML file — Kubernetes, ArgoCD, observability, DNS and TLS, SSO. This tool creates the cloud resources it needs, resolves module versions, and runs the pipelines that apply them. On every subsequent run it updates everything to the latest tested release and applies your config changes.

It is a Go CLI, not an LLM agent.

Install

go install github.com/entigolabs/entigo-infralib-agent@latest

docker pull entigolabs/entigo-infralib-agent            # Docker Hub
docker pull public.ecr.aws/entigolabs/entigo-infralib-agent   # ECR Public

Installing with go install places the binary in $(go env GOPATH)/bin as entigo-infralib-agent. Examples below use ei-agent, the name used when building from source.

Documentation

Full configuration reference, command flags, and operational guides: docs.entigo.com. The modules this tool applies live in entigo-infralib.

License

Entigo Infralib Agent is licensed under the GNU Affero General Public License v3.0. The full text is in LICENSE. The Infralib modules and the module releases are under the same license.

What this means in practice:

Running the agent to manage your own infrastructure carries no source obligation. The AGPL covers this program, not the infrastructure it provisions or the applications you deploy on the resulting platform. Your config files, Terraform inputs and workloads are yours.
If you modify the agent and let others interact with it over a network, the AGPL requires you to offer them the modified source. This is the clause that distinguishes the AGPL from the GPL, and it is the one to read carefully if you plan to build a hosted service around a fork.
Redistribution of modified versions — including modified images — requires that recipients get the source under the same license.

This summary is for orientation only; the LICENSE file governs, and if the network clause is material to your plans, take your own legal advice.

Infralib is developed by Entigo and is currently maintained by the Entigo team rather than an external contributor community. Entigo Platform is a separate commercial product that builds on Infralib; Infralib itself does not depend on it and continues to work without it.

Reference

Requirements
Compiling Source
Installation
Docker
Commands
- Bootstrap
- Run
- Update
- Destroy
- Delete
- Service Account
- Pull
- Custom Parameters
Config
Migration Helper

Requirements

AWS Service Account with administrator access, credentials provided by AWS or environment variables.

or

Google Cloud Service Account with owner access, credentials provided by GCP or gcloud cli tool.

Compiling Source

go build -o bin/ei-agent main.go

Installation

go install github.com/entigolabs/entigo-infralib-agent@latest

This will build and install the binary to $(go env GOPATH)/bin directory. Make sure that the directory is in your PATH. When using this method, replace ei-agent with entigo-infralib-agent in the example commands.

Docker

Prebuilt Docker image is available from

Docker Hub entigolabs/entigo-infralib-agent

or

Amazon ECR Gallery public.ecr.aws/entigolabs/entigo-infralib-agent

Building a local Docker image

docker build -t entigolabs/entigo-infralib-agent .

Running the Docker image

By default, the docker image executes the Run command. Config.yaml needs to be mounted into the container. This is required only for the first run or when overriding an existing config.

docker run --pull always -it --rm -v "$(pwd)/config.yaml":"/etc/ei-agent/config.yaml" -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_REGION=$AWS_REGION -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN -e CONFIG=/etc/ei-agent/config.yaml entigolabs/entigo-infralib-agent

To execute the bootstrap, override the default command.

docker run --pull always -it --rm -v "$(pwd)/config.yaml":"/etc/ei-agent/config.yaml" -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_REGION=$AWS_REGION -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN -e CONFIG=/etc/ei-agent/config.yaml entigolabs/entigo-infralib-agent ei-agent bootstrap

Using with AWS Profile

Example shell script that checks if the AWS SSO login is valid and runs the agent with the specified profile. Make sure to replace the value of AWS_PROFILE with your chosen AWS profile name. Initialize the AWS profile with aws configure sso --profile profile-name if it doesn't exist. Alternatively to SSO, you can manually edit the ~/.aws/credentials file to add the profile with access key and secret key with token if using temporary credentials. In that case, you can remove the SSO login check from the script.

#!/bin/bash

AWS_REGION="eu-north-1"
AWS_PROFILE="profile-name"

echo "Checking SSO login..."
aws sts get-caller-identity --profile "${AWS_PROFILE}" || {
    echo "SSO login required. Running: aws sso login --profile ${AWS_PROFILE}"
    aws sso login --profile "${AWS_PROFILE}"
}

docker run --pull always -it --rm \
    -v "$(pwd)/config":"/etc/ei-agent/config" \
    -v "$(pwd)/config.yaml":"/etc/ei-agent/config.yaml" \
    -v "$HOME/.aws:/root/.aws:ro" \
    -e AWS_REGION="${AWS_REGION}" \
    -e AWS_PROFILE="${AWS_PROFILE}" \
    -e CONFIG=/etc/ei-agent/config.yaml \
    entigolabs/entigo-infralib-agent ei-agent run

Commands

For bootstrap, run and update commands you must either provide a config file or a prefix value. This is required for creating and finding AWS resources. Bootstrap adds that value as an environment variable for the agent pipeline.

bootstrap

Creates the required cloud resources and pipelines for executing the agent run and update commands. If the pipeline already exists, the agent image version will be updated if needed and a new execution of the run command will be started. For AWS, CodePipeline is used, for GCloud, Cloud Run Jobs are used.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
config - config file path and name, only needed for first run or when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources (default: config prefix) [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
start - start pipeline execution after creating (default: true) [$START]

Example

bin/ei-agent bootstrap --config=config.yaml --prefix=infralib

run

Processes config steps, creates and executes CodePipelines which apply Entigo Infralib terraform modules. Run command only executes a single cycle of the pipeline. Can be used to apply config changes.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
skip-bucket-creation-delay - skip bucket creation delay (default: false) [$SKIP_BUCKET_CREATION_DELAY]
config - config file path and name, only needed for first run or when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources (default: config prefix) [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - optional role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
steps - optional comma separated list of steps to run [$STEPS]
allow-parallel - allow running steps in parallel on first execution cycle (default: true) [$ALLOW_PARALLEL]
pipeline-type - pipeline execution type (local | cloud), local is meant to be run inside the infralib image (default: cloud) [$PIPELINE_TYPE]
print-logs - print terraform/helm logs to stdout when using local execution (default: true) [$PRINT_LOGS]
logs-path - optional path for storing terraform/helm logs when running local pipelines [$LOGS_PATH]
terraform-cache - use terraform caching (default: true, when using pipeline-type local, default is false) [$TERRAFORM_CACHE]

Example

bin/ei-agent run --config=config.yaml --prefix=infralib

update

Processes config steps, creates and executes CodePipelines which apply Entigo Infralib terraform modules. Update command updates all modules to the latest or specified versions. Returns if there are no updates available.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
skip-bucket-creation-delay - skip bucket creation delay (default: false) [$SKIP_BUCKET_CREATION_DELAY]
config - config file path and name, only needed for first run or when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources (default: config prefix) [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - optional role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
steps - optional comma separated list of steps to run [$STEPS]
pipeline-type - pipeline execution type (local | cloud), local is meant to be run inside the infralib image (default: cloud) [$PIPELINE_TYPE]
print-logs - print terraform/helm logs to stdout when using local execution (default: true) [$PRINT_LOGS]
logs-path - optional path for storing terraform/helm logs when running local pipelines [$LOGS_PATH]
terraform-cache - use terraform caching (default: true, when using pipeline-type local, default is false) [$TERRAFORM_CACHE]

Example

bin/ei-agent update --config=config.yaml --prefix=infralib

destroy

Executes the destroy pipelines in reverse config order. Warning! This will remove the resources provisioned by the step pipelines. Agent automatically approves the changes.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
config - config file path and name, only needed when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources (default: config prefix) [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
yes - skip confirmation prompt (default: false) [$YES]
steps - optional comma separated list of steps to destroy [$STEPS]
pipeline-type - pipeline execution type (local | cloud), local is meant to be run inside the infralib image (default: cloud) [$PIPELINE_TYPE]
print-logs - print terraform/helm logs to stdout when using local execution (default: true) [$PRINT_LOGS]
logs-path - optional path for storing terraform/helm logs when running local pipelines [$LOGS_PATH]

Example

bin/ei-agent destroy --config=config.yaml --prefix=infralib

delete

Processes config steps, removes resources used by the agent, including buckets, pipelines, and roles/service accounts. Warning! Execute destroy pipelines in reverse config order before running this command. This command will remove all pipelines and resources created by terraform will otherwise remain.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
config - config file path and name, only needed when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources (default: config prefix) [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
yes - skip confirmation prompt (default: false) [$YES]
delete-bucket - delete the bucket used by terraform state (default: false) [$DELETE_BUCKET]
delete-service-account - delete the service account created by service-account command (default: false) [$DELETE_SERVICE_ACCOUNT]

Example

bin/ei-agent delete --config=config.yaml --prefix=infralib

service-account

Creates a service account and a key for the account. Key will be outputted to the stdout. This account can be used for running the agent in a CI/CD pipeline.

Optionally, prefix will be used to pull the config and config is used to check for an encryption module. If present, the created key will be encrypted with customer encryption. More info in Encryption.

OPTIONS:

config - config file path and name, only needed when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
role-arn - role arn for assume role, used when creating aws resources in external account [$ROLE_ARN]
rotate-credentials - optional, generate new credentials for an existing service account, default false. Warning! This will delete any previous keys. [$ROTATE_CREDENTIALS]
trust-role - optional, instead of generating keys adds a trust relationship in AWS role or allows impersonation of the service account in GCloud. Value needs to be arn for AWS and full principal for GCloud, e.g. serviceAccount:email or user:email. [$TRUST_ROLE]
remove-user - optional, used with trust-role, removes an existing service account user in AWS or credentials in GCloud, default false. [$REMOVE_USER]

Example

bin/ei-agent service-account --prefix=infralib

pull

Pulls agent config yaml and the config folders from the S3/GCloud bucket. Use the force flag to overwrite existing local files.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
config - config file path and name, only needed when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
force - overwrite existing local files, default false. Warning! Force deletes the /config subfolder before writing. [$FORCE]

Example

bin/ei-agent pull --prefix=infralib

provision

Used by Infralib wrapper layer. provision is executed by a step pipeline. It wraps the Infralib output and, when a wrapper block is configured in the agent config, forwards raw stdout log lines and a compact plan summary to the backend over gRPC. Without a wrapper config the invocation is fully transparent. Infralib output goes only to the pipeline's normal stdout. All the OPTIONS are optional and any missing values fallback to running transparently.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
wrapper-config - optional wrapper api config yaml (resolved from secret manager by the pipeline) [$WRAPPER_CONFIG]
step - optional step name for the current pipeline execution [$INFRALIB_STEP]
prefix-step - optional step name with cloud prefix [$TF_VAR_prefix]
command - optional infralib command to execute (plan | apply | plan-destroy | apply-destroy | argocd-plan | argocd-apply | argocd-plan-destroy | argocd-apply-destroy) [$COMMAND]
entrypoint - optional path to the infralib-tool entrypoint script (default: entrypoint-core.sh) [$INFRALIB_ENTRYPOINT]
campaign-id - optional agent-run identifier forwarded to the backend handshake; empty runs the wrapper transparently [$CAMPAIGN_ID]
pipeline-index - optional release iteration index forwarded to the backend handshake [$PIPELINE_INDEX]
insecure - optional allow insecure gRPC connection (default: false) [$INSECURE]

Custom Parameters

Agent has helpful commands for managing custom parameters that can be used in the config file with the {{ .output-custom.key }} replacement tag. These commands are:

add-custom - for adding or updating a custom parameter
delete-custom - for deleting a custom parameter
get-custom - for getting a custom parameter value
list-custom - for listing all custom parameters created by the agent

Optionally, prefix will be used to pull the config and config is used to check for an encryption module. If present, the created key will be encrypted with customer encryption. More info in Encryption.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
config - config file path and name, only needed when overriding an existing config [$CONFIG]
prefix - prefix used when creating cloud resources [$PREFIX]
project-id - project id used when creating gcloud resources [$PROJECT_ID]
location - location used when creating gcloud resources [$LOCATION]
zone - zone used in gcloud run jobs [$ZONE]
google-application-credentials-json - optional, gcloud service account credentials JSON string [$GOOGLE_APPLICATION_CREDENTIALS_JSON]
key - key for the custom parameter [$KEY]
value - value for the custom parameter [$VALUE]
overwrite - overwrite existing custom parameter value, default false [$OVERWRITE]

Example

bin/ei-agent add-custom --key=custom-key --value=custom-value

Version

Shows the agent version, build date and commit hash.

Config

Config is provided with a yaml file:

prefix: string
sources:
  - url: https://github.com/entigolabs/entigo-infralib-release | path
    version: stable | semver | branch
    include: []string
    exclude: []string
    force_version: bool
    username: string
    password: string
    repo_path: string
    ca_file: string
    use_oci_digests: bool
    verify_signature: bool
destinations:
  - name: string
    git:
      url: string
      key: string
      key_password: string
      username: string
      password: string
      author_name: string
      author_email: string
      insecure: bool
      ca_file: string
notifications:
  - name: string
    context: string
    message_types: []string
    slack:
      token: string
      channel_id: string
    teams:
      webhook_url: string
    api:
      url: string
      wrapper_url: string
      headers: map[string]string
      oauth:
        client_id: string
        client_secret: string
        token_url: string
        scopes: []string
schedule:
  update_cron: string
agent_version: latest | semver
base_image_source: string
base_image_version: stable | semver
enable_opentofu: bool
use_oci_proxy: bool
provider:
  inputs: map[string]string
  aws:
    default_tags:
      tags: map[string]string
    endpoints: map[string]string
    ignore_tags:
      key_prefixes: []string
      keys: []string
  kubernetes:
    ignore_annotations: []string
    ignore_labels: []string
steps:
  - name: string
    type: terraform | argocd-apps
    approve: minor | major | never | always | force | reject
    manual_approve_run: always | changes | removes | never | reject
    manual_approve_update: always | changes | removes | never | reject
    base_image_source: string
    base_image_version: stable | semver
    vpc:
      attach: bool
      id: string
      subnet_ids: multiline string
      security_group_ids: multiline string
    kubernetes_cluster_name: string
    argocd_namespace: string
    modules:
      - name: string
        source: string
        version: stable | semver
        http_username: string
        http_password: string
        default_module: bool
        inputs: map[string]interface{}
    provider:
      inputs: map[string]string
      aws:
        default_tags:
          tags: map[string]string
        endpoints: map[string]string
        ignore_tags:
          key_prefixes: []string
          keys: []string
      kubernetes:
        ignore_annotations: []string
        ignore_labels: []string

Complex values need to be as multiline strings with | symbol.

Source version is overwritten by module version. Default version is stable which means latest release of the source repository.

prefix - prefix used for AWS/GCloud resources, bucket folders/files and terraform resources, limit 10 characters, overwritten by the prefix flag/env var
sources - list of source repositories for Entigo Infralib modules
- url - url of the source repository or path to the local directory. Path must start with ./ or ../ Path will set force_version to true and use local as the version. Path only works with the local pipeline execution type.
- version - highest version of Entigo Infralib modules to use
- include - list of module sources to exclusively include from the source repository
- exclude - list of module sources to exclude from the source repository
- force_version - sets the specified version to all modules that use this source, useful for specifying a branch or tag instead of semver or digest for OCI, default false. Modules with forced version always allow running in parallel during executions. Warning! Before changing from true to false, force a version that follows semver.
- username - username for git authentication
- password - password for git authentication, it's recommended to use custom replacement tags, e.g. "{{ .output-custom.git-password}}"
- repo_path - path to the git repository root directory, default uses Go's TempDir to create a directory named after the repository url. Use debug logging to see the path. Warning! Agent prunes the repo to match the remote.
- ca_file - name of the CA certificate file in the ./ca-certificates folder to use for git authentication
- use_oci_digests - use OCI digests for module sources instead of version tags when using OCI source with version tag, default false
- verify_signature - verify the OCI index signature when using OCI entigolabs source, true value forces use_oci_digests to true as well, default false
destinations - list of destinations where the agent will push the generated step files, in addition to the default bucket
- name - name of the destination
- git - git repository must be accessible by the agent. For authentication, use either key or username/password. For the key and password, it's recommended to use custom replacement tags, e.g. "{{ .output-custom.git-key }}"
  - url - url of the git repository
  - key - PEM encoded private key for authentication
  - key_password - optional, password for the private key
  - insecure_host_key - accept any host key when using private key, default false
  - username - username for authentication
  - password - password for authentication
  - author_name - author name for commits, default Entigo Infralib Agent
  - author_email - author email for commits, default no-reply@localhost
  - insecure - allow insecure connection, default false
  - ca_file - name of the CA certificate file in the ./ca-certificates folder to use for git authentication
notifications - send notifications with selected types, each notifier can only use one subtype
- name - name of the notifier
- context - optional, extra context added to the notification
- message_types - list of types of messages to send, possible values started | approvals | sources | progress | schedule | success | failure, default [approvals, failure]. More info in Message types
- api - send notifications to a custom API
  - url - url for the api
  - wrapper_url - optional, enables gRPC connection while provisioning for sending logs and plan summaries. Full URL of the backend endpoint (https://host[:port][/path]). The path segment is preserved and prepended to gRPC method names. When omitted, provision runs the entrypoint transparently. When set, the config is stored in Secret Manager and injected into each pipeline execution as the WRAPPER_CONFIG env var.
  - headers - key-value pair of headers to add to the request
  - oauth - optional oauth2 configuration for the api
- slack - send notifications to slack
  - token - slack access token, it's recommended to use custom replacement tags, e.g. "{{ .output-custom.slack-token }}"
  - channel_id - slack channel id
- teams - send notifications to teams
  - webhook_url - webhook url for the teams channel, possible options include Teams Workflow or Power Automate, more info in go-teams-notify GitHub
schedule - allows scheduling CodePipeline/Cloud Run Job executions. More info in Scheduling
- update_cron - cron expression in UTC for scheduling agent update executions.
agent_version - image version of Entigo Infralib Agent to use
base_image_source - source of Entigo Infralib Base Image to use
base_image_version - image version of Entigo Infralib Base Image to use, default uses the version from step
enable_opentofu - make Infralib use OpenTofu instead of Terraform, default true.
use_oci_proxy - replace OCI source url host with proxy if registry proxy module has been applied, excluding modules that are part of the same step as the proxy module, default false
provider - provider values to add for all terraform steps
- inputs - variables for provider tf file
- aws - aws provider default, ignore tags and endpoints to add
- kubernetes - kubernetes provider ignore annotations and labels to add
steps - list of steps to execute
- name - name of the step
- type - type of the step
- approve - deprecated, approval type for the step, possible values minor | major | never | always | force | reject, default always. More info in Auto approval logic
- manual_approve_update - approval type for the step when using the update command, possible values always | changes | removes | never | reject, default removes. More info in Auto approval logic
- manual_approve_run - approval type for the step when using the run command, possible values always | changes | removes | never | reject, default changes. More info in Auto approval logic
- base_image_source - source of Entigo Infralib Base Image to use
- base_image_version - image version of Entigo Infralib Base Image to use, default uses the newest module version
- vpc - vpc values to add
  - attach - attach vpc to code build/cloud run job, if other fields are empty then uses default vpc based on typed output of a vpc module, default nil. When nil, the value will be set based on the step type, for argocd-apps steps the value will be set to true
  - id - vpc id for code build/cloud run job, gcloud default {{ .toutput.vpc.vpc_name }}, aws default {{ .toutput.vpc.vpc_id }}
  - subnet_ids - vpc subnet ids for code build/cloud run job, gcloud default [{{ .toutput.vpc.private_subnets[0] }}], aws default [{{ .toptout.vpc.control_subnets | .toutput.vpc.private_subnets }}]
  - security_group_ids - vpc security group ids for code build/cloud run job, gcloud no default, aws default [{{ .toutput.vpc.pipeline_security_group }}]
- kubernetes_cluster_name - kubernetes cluster name for argocd-apps steps, gcloud default {{ .toutput.gke.cluster_name }}, aws default {{ .toutput.eks.cluster_name }}
- argocd_namespace - kubernetes namespace for argocd-apps steps, default argocd
- modules - list of modules to apply
  - name - name of the module
  - source - source of the terraform module, can be an external git repository beginning with git:: or git@
  - version - highest version of the module to use
  - http_username - username for external repository authentication
  - http_password - password for external repository authentication
  - default_module - when using tmodule replacement, default module will be used if multiple modules of the same type exist, default false
  - inputs - optional, map of inputs for the module, string values need to be quoted. If missing, inputs are optionally read from a yaml file that must be located in the ./config/<stepName> directory with a name <moduleName>.yaml
- provider - provider values to add
  - inputs - variables for provider tf file
  - aws - aws provider default, ignore tags and endpoints to add
  - kubernetes - kubernetes provider ignore annotations and labels to add

Including and excluding modules in sources

Agent associates modules with sources for requesting module files. Module association is done by checking if a source includes a module. This is done by checking the source list from top to bottom. Source includes a module if:

source config include property contains the module source
source config exclude property does not contain the module source
module source path is a subdirectory in source modules folder

For modules in steps with type argocd-apps, agent currently prefixes the k8s part of the path in the module source automatically.

Example

sources:
  - url: https://github.com/entigolabs/entigo-infralib-release
    exclude: ["argocd"]
    include:
      - aws/eks
  - url: https://github.com/entigolabs/entigo-infralib

Auto approval logic

Each step can be configured how it automatically approves infrastructure changes for the agent's run and update commands. To decide when to auto approve changes. If the planning stage of a step finds no changes, then the pipeline apply stage will be skipped. If only one of the manual_approve_* properties is set for a step, then the other property uses the default value. Possible values for the manual_approve_run and manual_approve_update are:

always - will ask for user manual approval when resources are added, changed or removed.
changes - will ask for user manual approval when resources are changed or removed. (default for manual_approve_run).
removes - will ask for user manual approval when resources are removed (default for manual_approve_update).
never - will never ask for manual approval. (DANGEROUS!)
reject - stop the pipeline instead of approving, marks the step as failed. This can be used to generate plan files without applying them.

Step property approve has been deprecated and replaced by manual_approve_run and manual_approve_update. If none of those fields is set then approve will be used with default always value for backwards compatibility.

When using the approve property, auto approve type is only considered when resources will be changed. Adding resources doesn't require manual approval. Destroying resources always requires manual approval, except when using type force. Approve always means that manual approval is required, never means that agent approves automatically. Types major and minor require manual approval only when any of the step modules has a major or minor semver version change. Modules with external source require manual approval.

Overriding config values

Step, module and input field values can be overwritten by using replacement tags {{ .type.key }}. Possible replacement tags are:

Type	Key / Format	Example	Description
`agent`	version.stepName.moduleName	`.agent.version.infra.eks`	Configured version of the specified module.
	accountId	`.agent.accountId`	Configured AWS account ID.
	region	`.agent.region`	Configured cloud provider region.
`config`	fieldName	`.config.prefix`	Value from the provided config field. Config replacement does not support indexed paths.
`module`	name	`.module.name`	Name of the module itself (for module inputs and input files only).
	source	`.module.source`	Source of the module itself (for module inputs and input files only).
`optout`	stepName.moduleName.key	`.optout.infra.eks.cluster_arn`	Optional value from Terraform output from specific step/module. Defaults to empty string.
`output`	stepName.moduleName.key	`.output.infra.eks.cluster_arn`	Value from Terraform output from specific step/module.
`output-custom`	key	`.output-custom.param-key`	Value from AWS SSM parameter or GCloud SM.
`step`	name	`.step.name`	Name of the step containing the module.
`tinput`	type.Key	`.tinput.argocd.argocd.global.domain`	Value from a module inputs in the current step. Falls back to values files for argocd-apps steps
`tmodule`	type	`.tmodule.eks`	Name of the module with a specified type.
`toptmodule`	type	`.toptmodule.eks`	Optional name of the module with a specified type.
`toptout`	type.key	`.toptout.eks.cluster_arn`	Optional value from Terraform output based on module type. Defaults to empty string.
`toutput`	type.key	`.toutput.eks.cluster_arn`	Value from Terraform output based on module type.
`tsmodule`	type	`.tsmodule.eks`	Name of the typed module in the current step.

For output types, if the value is not found from terraform output, then the value is requested from AWS SSM Parameter Store or Google Cloud Secret Manager.

For example, {{ .output.stepName.moduleName.key-1 }} will be overwritten with the value from terraform output moduleName__key-1. As a fallback, uses SSM Parameter Store parameter /entigo-infralib/config.prefix-stepName-moduleName-parentStep/key-1.

It's possible to build a custom array by using yaml multiline string, even mixing replaced values with inputted values. For example creating a list of strings for terraform:

inputs:
  key-1: |
    ["{{ .output.stepName.moduleName.key-1 }}", "value-1", "value-2"]

List indexes

If the parameter type is StringList then it's possible to use an index to get a specific value, e.g. {{ .output.stepName.moduleName.key-1[0] }} or a slice by using a range, e.g. [0-1]. In case of terraform output, it's also possible to use a map key as the index, e.g. {{ .output.stepName.moduleName.key-1[key-2] }}.

Escaping replacement tags

Replacement tags support escaping with inner {{`{{ }}`}} tags. For example, {{`{{ .dbupdate }}`}} will be replaced with {{ .dbupdate }}. This can be used to pass helm template values through the agent.

Optional replacement tags

If the output value is optional then use optout or toptout, it will replace the value with an empty string if the module or output is not found. Optional tag can be combined with the | operation to add (multiple) fallback values. Quotation marks can be used to provide a default value. For example {{ .optout.stepName.ModuleName.key-1 | "default" }}.

Including files in steps

It's possible to include files in steps by adding the files into a ./config/<stepName>/include subdirectory. File names can't include main.tf, provider.tf or backend.conf as they are reserved for the agent. For ArgoCD, reserved name is argocd.yaml and named files for every module module-name.yaml. Files will be copied into the step directory which is used by terraform and ArgoCD as step context.

Including CA certificates

It's possible to include CA certificates by adding the files into a ./ca-certificates subdirectory. Files will be copied into the bucket root and each step directory for Infralib.

Notifications

The agent emits lifecycle events at three nested levels of granularity so external systems can observe progress and outcomes:

Agent execution — the entire agent invocation. Emits a start event and a terminal success or failure event.
Pipeline — each release iteration applied by the agent. A run command applies a single release (one pipeline). An update command can apply multiple releases sequentially, each emitting its own start, success, or failure events.
Step — each configuration step within a pipeline. Emits start, success, failure, or skipped events.

Failure cascade. The levels nest: a step failure surfaces as a pipeline failure, which surfaces as an agent failure.

Subscriptions. Lifecycle events are grouped into message types. Each notifier configured under notifications subscribes to specific message types via message_types, so the same event stream can be routed to different channels at different levels of detail (e.g. failure to a low-noise human channel, progress to an API integration). When message_types is omitted, the default is [approvals, failure].

Delivery. Notifier delivery is asynchronous and best-effort within the agent. A notifier returning an error is logged but does not abort the agent or block other notifiers.

Pre-initialization failures. Failures that occur before the notification manager is constructed (cloud provider client setup, reading the root config from the bucket, parsing the notifications block itself) cannot be delivered through this system. Those are the responsibility of whatever runs the agent (Kubernetes Job, AWS CodeBuild, Cloud Run Job, etc.) via process exit code and container logs.

Message types

started — the agent execution has started. Fires once at the beginning of the execution, after the notification manager is constructed. Includes the command and the agent's cloud identifier.
success — the agent execution finished successfully. Fires once at the end of a successful execution.
failure — an agent-level failure. Fires once on any error reachable after the notification manager has been constructed (resource setup, encryption setup, updater construction, or a propagated pipeline failure). Includes the error message.
progress — pipeline and step lifecycle. Carries:
- Pipeline starting / success / failure for each release iteration, with the source versions being applied.
- Step starting / success / failure / skipped for each configuration step. A step is skipped when no changed modules are found.
approvals — the pipeline is waiting for manual approval. Includes the planned changes and a link to the pipeline. Also fires when an approval is granted.
modules — list of modules that will be applied across all steps. Fires once near the start of the agent execution.
sources — list of sources and their resolved releases. Fires once at the start of the release loop.
schedule — emitted when the agent's update schedule is added, modified, or removed during bootstrap.

API

When configuring API notifications, the agent will send requests to the specified URL. The OpenAPI specification for the endpoints is in the openapi/notification-api.yaml.

Encryption

Agent uses default cloud provider encryption settings if no encryption module is present in config.

Currently, infralib only supports customer provided encryption in AWS with KMS. When KMS module is present in the config file, agent will use the KMS arn from the module terraform output to configure the S3 bucket and CloudWatch log groups to use KMS by default. Agent will also use the KMS when creating Parameter Store parameters and Secret Manager secrets. Agent applies those changes only when a previous execution has successfully applied the KMS module. Meaning, only objects that have been put in S3 after the KMS module was applied will be encrypted with it.

Scheduling

Agent supports scheduling agent CodePipeline/Cloud Run Job executions by configuring the schedule section in the config file. bootstrap command must be executed initially to create the required agent jobs.

Cron expressions must be in UTC timezone and valid according to the selected cloud provider. Removing the expression will also remove the scheduled execution.

AWS

Uses EventBridge Scheduler. Cron expression format is Minutes Hours Day-of-month Month Day-of-week Year.

Initial creation of schedule requires admin privileges to create the required role that can start the CodePipeline execution. Subsequent updates can be also be done with a generated service account.

GCloud

Uses Google Cloud Scheduler. Uses unix-cron format Minutes Hours Day-of-month Month Day-of-week.

Scheduler doesn't support all Cloud locations. Agent will try to use the configured location. If not supported, agent will use a fallback location based on the region:

Region Prefix	Fallback Location
africa	europe-west1
asia	asia-east1
australia	australia-southeast1
europe	europe-west1
me	me-central2
northamerica	northamerica-northeast1
southamerica	southamerica-east1
us	us-central1

Supported locations can change in the future. If configured location starts supporting scheduler then agent will create a new schedule in that location. Older schedule must be manually removed, otherwise both schedules will start Job executions.

Migration Helper

Agent includes 3 commands to help migrate from existing terraform state to Entigo Infralib modules: migrate-config, migrate-plan and migrate-validate.

Config command requires a terraform v4 state file. Plan and validate commands require the state file and a terraform plan file. Infralib state and plan files can be obtained from the bucket used by agent. It's possible to combine the approval type reject with run command argument steps to generate plan files without applying them for the chosen steps. Plan files need to be manually converted into json format by using terraform.

First generate an import configuration with migrate-config. Modify the generated lines to remove any resources that don't need migrating. If needed, split any config items into source and destination blocks. That's only necessary if source and destination don't have any matching fields from name, module and indexes.

Using the import config, generate terraform import commands with the migrate-plan command. After executing the commands, run the pipelines with the approval type reject to generate a new plan. Optionally, use the migrate-validate command to validate the new plan along with the new state.

Migration Commands

migrate-config

Outputs a list of resources with instance indexes based on the state file that are not matched by the provided import file. More info in Migration Helper.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
state-file - path to the previous terraform state file [$STATE_FILE]
import-file - optional, path to the import file [$IMPORT_FILE]

Example

bin/ei-agent migrate-config --state-file=state-file.json

migrate-plan

Generates import and rm terraform commands based on the input files. Warning! Always check the import and rm commands before executing them. More info in Migration Helper.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
state-file - path to the previous terraform state file [$STATE_FILE]
plan-file - path to the terraform plan file [$PLAN_FILE]
import-file - path to the import file [$IMPORT_FILE]
types-file - optional, path for type identifications file [$TYPES_FILE]

Example

bin/ei-agent migrate-plan --state-file=state-file.json --plan-file=plan.json --import-file=import-file.yaml

migrate-validate

Validate a terraform plan file based on the import config and infralib terraform state. Outputs 3 types of warnings:

If plan wants to create or remove a resource that should be in the infralib state file.
If plan wants to create or remove a resource that has the same type as the type in the import file.
If plan wants to change a value of a resource.

More info in Migration Helper.

OPTIONS:

logging - logging level (debug | info | warn | error) (default: info) [$LOGGING]
state-file - path to the new terraform state file [$STATE_FILE]
plan-file - path to the terraform plan file [$PLAN_FILE]
import-file - path to the import file [$IMPORT_FILE]

Example

bin/ei-agent migrate-plan --state-file=state-file.json --import-file=import-file.yaml

Import File

Import configuration file is used to tell the migration commands which type of resources should be migrated.

import:
  - type: string
    name: string
    module: string
    index_keys: []int | []string
    source:
      module: string
      name: string
      index_key: int | string
      index_keys: []int | []string
    destination:
      module: string
      name: string
      index_key: int | string
      index_keys: []int | []string

type - type of the resource to import
name - optional, overwrites source and destination names
module - optional, overwrites source and destination modules
index_keys - optional, overwrites source and destination index_keys
source - source is a resource from the previous state file
destination - destination is a resource from the plan file
- module - optional, module name of the resource
- name - optional, name of the resource
- index_key - optional, index key of the resource instance
- index_keys - optional, index keys of the resource instance

If module and name are empty then agent will try to find the resource from the state/plan file based on the type. If there are more than 1 resource with that type then a warning is printed. Agent will generate import commands for all resources of that type.

Index key and index keys are mutually exclusive. Index keys can be used to map instances in different order to the new state. If index hasn't been set, then agent will use keys from state resource instances.

Plan command generates import commands for the destination resources and rm commands for the source resources.

Type Identifications File

Type identifications file is used to map a resource type to an identification used by terraform for the import command. Agent will replace any placeholder {} values with values from the resource instance attributes. Default file is located in migrate/types.yaml.

typeIdentifications:
  - identification: string
    listSeparator: string
    types: []string

identification - identification used by terraform for the import command, e.g "{id}"
listSeparator - optional, separator for identification values that are lists, default /
types - list of resource types that use the identification

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
argocd		argocd
aws		aws
cli		cli
commands		commands
common		common
examples		examples
gcloud		gcloud
gen/wrapper/v1alpha1		gen/wrapper/v1alpha1
git		git
migrate		migrate
model		model
notify		notify
oci		oci
openapi		openapi
proto		proto
service		service
terraform		terraform
test		test
util		util
wrapper		wrapper
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Entigo Infralib Agent

What it does

Install

Documentation

License

Reference

Requirements

Compiling Source

Installation

Docker

Building a local Docker image

Running the Docker image

Using with AWS Profile

Commands

bootstrap

run

update

destroy

delete

service-account

pull

provision

Custom Parameters

Version

Config

Including and excluding modules in sources

Auto approval logic

Overriding config values

List indexes

Escaping replacement tags

Optional replacement tags

Including files in steps

Including CA certificates

Notifications

Message types

API

Encryption

Scheduling

AWS

GCloud

Migration Helper

Migration Commands

migrate-config

migrate-plan

migrate-validate

Import File

Type Identifications File

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages