gitlab-ci.yml

GitLab CI templates and snippets

terraform

The CI templates terraform/Terraform.gitlab-ci.yml is based on the GitLab terraform CI template. Following these templates the terraform lock file .terraform-lock.hcl is not part of the cache and should be part of the repository.

The template modifies the GitLab terraform CI template so that the GitLab flow can be used (s. example project GitLab Terraform GitLab Flow).

The template is based on terraform/Terraform.base.gitlab-ci.yml and includes the header yaml and a yaml for each job:

The destroy job depends only on validate to facilitate executing destroy manually. If you want destroy to depend on deploy use:

include:
  - project: 'packages-and-registries/devops-ci-artifacts'
    file:    '/terraform/Terraform.gitlab-ci.yml'

image:
  name: lwith/gitlab-devops:latest

destroy:
  extends:
    - .deploy
  needs:
    - !reference [.deploy, needs ]
    - deploy
  dependencies:
    - !reference [.deploy, dependencies]
    - build

The simplest usage is

include:
  - project: 'packages-and-registries/devops-ci-artifacts'
    file:    '/terraform/Terraform.gitlab-ci.yml'

image:
  name: lwith/gitlab-devops:latest

A CI with extra before scripts in the deploy job is the following

include:
  - project: 'packages-and-registries/devops-ci-artifacts'
    file:    '/terraform/Terraform.gitlab-ci.yml'

image:
  name: lwith/gitlab-devops:latest

deploy:
  extends: 
    - .deploy
  before_script:
    - !reference [.deploy, before_script]
    - !reference [.before_script_ssh_prepare_id, before_script]
    - !reference [.before_script_ssh_agent_add_id, before_script] 
    - !reference [.before_script_ansible_requirements, before_script]

There are a couple of before_scripts that can be used by reference-tags in '/terraform/Terraform.base.gitlab-ci.before-scripts.yml'.

The most often important ones are described here.

before_script_secrets_vars

fetches the secrets in $TF_ROOT/secrets from vault:
- default.yml if the file it exists
- ${ENVIRONMENT}.yml if the file it exists
fetches the environment variable values from .env-files in `$CI_PROJECT_DIR/environment``
- default.env if the file it exists
- ${ENVIRONMENT}.env if the file exists

All files are passed through envsubst before they are processed. If the differences between the environments can be expressed through environment variables, it is possible to use only the default.*-files.

before_script_secrets_env_files

downloads the secure files
fetches the environment variable values from .env-files in .secret_files

before_script_vault_token

fetches a vault token for the role ${VAULT_TOKEN_ROLE} with vault_token.sh

before_script_ansible_requirements

installs the ansible roles defined in $ANSIBLE_PATH/ansible-requirements.yml into $ANSIBLE_ROLES_PATH
installs the ansible collections defined in $ANSIBLE_PATH/ansible-requirements.yml into $ANSIBLE_COLLECTIONS_PATH with timeout $ANSIBLE_GALAXY_TIMEOUT

trivy

The CI snippet trivy/Trivy.gitlab-ci.yml is an adapted Trivy.gitlab-ci.yml from section "GitLab CI using Trivy container" on GitLab CI.

The HTML and the JSON templates are copied from aquasecurity/trivy.

The reports are placed in $CI_PROJECT_DIR/.trivy-reports.

The full image name has to be placed in $FULL_IMAGE_NAME.

The jobs are assigned to stage scan.

vault

The CI snippet vault/Vault.tools.gitlab-ci.yml combines all vault CI snippets:

vault secrets
vault token
vault kv puts
vault kv2 patches

and puts all shell scripts into artifacts. Some scripts use jq and jc and outputs commands using vault.

The job is assigned to stage vault_tools_sh.

If in a job other artifacts are defined, use

  dependencies:
    - vault_tools_sh

vault secrets (v2)

The shell script vault_secrets_v2.sh uses VAULT_ID_TOKEN to authenticate. This token must be configured in a CI job e.g. by:

  id_tokens:
    ID_TOKEN:
      aud: $CI_SERVER_URL

vault secrets

The CI snippet vault/Vault.secrets.gitlab-ci.yml (vault/Vault.gitlab-ci.yml is deprecated) puts the shell scripts vault_secrets.sh into artifacts. The script uses jq and jc and outputs commands using vault.

The purpose is to simplify fetching secrets from vault.

The job is assigned to stage vault_secrets_sh.

If in a job other artifacts are defined, use

  dependencies:
    - vault_secrets_sh

to ensure fetching vault_secrets.sh from artifacts.

The shell script vault_secrets.sh interpretes a yaml file describing vault secrets, for instance:

secrets:
  - VAULT_AUTH_ROLE: terraform
    SSH_PASSPHRASE:
      path:  gitlab/with_de
      field: ssh_passphrase
    SSH_PRIVATE_KEY:
      path:  gitlab/with_de
      field: ssh_private_key
    TF_VAR_hcloud_token:
      path:  gitlab/with_de
      field: hcloud_token
  - VAULT_AUTH_ROLE: mailcow
    MAILCOW_ADMIN_USER:
      mount: gitlab
      path:  mailcow
      field: mailcow_admin_user
    MAILCOW_ADMIN_PASSWORD:
      mount: gitlab
      path:  mailcow
      field: mailcow_admin_password
    MAILCOW_MAILBOX_PASSWORDS:
      mount:  gitlab
      path:   mailcow/mailbox_passwords
      format: json

path is mandatory, mount, field and format are optional.

The syntax is closely related to use the vault kv get command and related to Use Vault secrets in a CI job in GitLab Premium.

The usage is

./vault_secrets.sh <secrets> [option]

The script by default outputs the commands to fetch the secrets described in the yaml file secrets from vault.

option can be

--debug / -d
output the commands to fetch the secrets from vault, do not use sub shell for vault and thus propagate errors
--test / -t
output the commands to fetch the secrets from vault, only try fetching secrets from vault, do not export the secrets
--markdown / -m
output a markdown table documenting the secrets

There are two usages:

./vault_secrets.sh secrets.yml

shows the commands produced by vault_secrets.sh.

./vault_secrets.sh secrets.yml >.secrets && . .secrets && rm .secrets

executes the commands produced by vault_secrets.sh in the execution context.

You have to set VAULT_ADDR and possibly VAULT_CACERT for using vault_secrets.sh.

It is a good pratice to use [!reference tags]https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags) by defining in the CI definition

.before-script-vault:
  before_script:
    - ./vault_secrets.sh secrets.yml >.secrets && . .secrets && rm .secrets

Test

You can test if all secrets are accessable by

The CI snippet also puts the shell script vault_secrets_test.sh into artifacts. This script tests if all secrets are accessable.

./vault_secrets.sh secrets.yml --test >.secrets && . .secrets && rm .secrets

Markdown for vault_secrets

You can output a Markdown table documenting the secrets by

./vault_secrets.sh secrets.yml --markdown

For the secrets yaml example above the result is

variable	role	path	field
SSH_PRIVATE_KEY	terraform	gitlab/ssh	ssh_private_key
TF_VAR_ssh_key_name	terraform	gitlab/ssh	ssh_key_name
DNS_API_TOKEN	terraform	gitlab/dns	dns_api_token
APPLICATION_REGISTRY_AUTH	application	gitlab/applications	registry_auth
TEST_LONG_PATH	application	gitlab/applications/subfolder	test_long_path

This output can pasted into the README.md of the project for documentation purpose.

vault token

The CI snippet vault/Vault.token.gitlab-ci.yml puts the shell scripts vault_token.sh into artifacts. The script use vault to fetch a vault token and exports the token.

The purpose is to simplify fetching a vault to token.

The job is assigned to stage vault_token_sh.

If in a job other artifacts are defined, use

  dependencies:
    - vault_token_sh

to ensure fetching vault_token.sh from artifacts.

The usage is

./vault_token.sh <vault-auth-role> [option]

The shell script vault_token.sh fetches a vault token for the role and exports the token in the environment variable VAULT_TOKEN.

It is a good pratice to use YAML anchors for scripts by defining in the CI definition

.before-script-vault: &before-script-vault
  - ./vault_token.sh terraform

vault kv puts

The CI snippet vault/Vault.kv_puts.gitlab-ci.yml puts the shell scripts vault_kv_puts.sh into artifacts. The script uses jq and jc and outputs commands using vault.

The purpose is to simplify putting key value pairs into vault.

The job is assigned to stage vault_kv_puts_sh.

If in a job other artifacts are defined, use

  dependencies:
    - vault_kv_puts_sh

to ensure fetching vault_kv_puts.sh from artifacts.

The shell script vault_kv_puts.sh interpretes a yaml file describing vault key value pairs, for instance:

kv_puts:
  - VAULT_AUTH_ROLE: mailcow
    gitlab/mailcow/kv:
      TLSA_dns_record_value: TLSA_DNS_RECORD_VALUE
      DKIM_dns_record_value: DKIM_DNS_RECORD_VALUE
    gitlab/mailcow/kv2:
      TLSA_dns_record_value2: TLSA_DNS_RECORD_VALUE2
      DKIM_dns_record_value2: DKIM_DNS_RECORD_VALUE2
  - VAULT_AUTH_ROLE: terraform
    gitlab/terraform/kv:
      TOKEN: token
      KEY:   key
    gitlab/terraform/kv2:
      TOKEN2: token2
      KEY2:   key2

The usage is

./vault_kv_puts.sh <vault-auth-role>

The script by default outputs the commands to put the key value pairs described in the yaml file kv_puts into vault.

option can be

--markdown / -m
output a markdown table documenting the key value pairs

There are two usages:

./vault_kv_puts.sh kv_puts.yml

shows the commands produced by vault_kv_puts.sh.

./vault_kv_puts.sh kv_puts.yml >.kv_puts && . .kv_puts && rm .kv_puts

executes the commands produced by vault_kv_puts.sh in the execution context.

You have to set VAULT_ADDR and possibly VAULT_CACERT for using vault_kv_puts.sh.

It is a good pratice to use YAML anchors for scripts by defining in the CI definition

.after-script-vault: &after-script-vault
  - ./vault_kv_puts.sh kv_puts.yml >.kv_puts && . .kv_puts && rm .kv_puts

Markdown for vault_kv_puts

You can output a Markdown table documenting the secrets by

./vault_kv_puts.sh kv_puts.yml --markdown

For the kv puts yaml example above the result is

role	path	key	value
mailcow	gitlab/mailcow/kv	TLSA_dns_record_value	TLSA_DNS_RECORD_VALUE
mailcow	gitlab/mailcow/kv	DKIM_dns_record_value	DKIM_DNS_RECORD_VALUE
mailcow	gitlab/mailcow/kv2	TLSA_dns_record_value2	TLSA_DNS_RECORD_VALUE2
mailcow	gitlab/mailcow/kv2	DKIM_dns_record_value2	DKIM_DNS_RECORD_VALUE2
terraform	gitlab/mailcow/kv	TLSA_dns_record_value	TLSA_DNS_RECORD_VALUE
terraform	gitlab/mailcow/kv	DKIM_dns_record_value	DKIM_DNS_RECORD_VALUE
terraform	gitlab/mailcow/kv2	TLSA_dns_record_value2	TLSA_DNS_RECORD_VALUE2
terraform	gitlab/mailcow/kv2	DKIM_dns_record_value2	DKIM_DNS_RECORD_VALUE2

This output can pasted into the README.md of the project for documentation purpose.

vault kv2 patches

The CI snippet vault/Vault.kv2_patches.gitlab-ci.yml puts the shell scripts vault_kv2_patches.sh into artifacts. The script uses jq and jc and outputs commands using vault.

The purpose is to simplify putting key value pairs into vault.

The job is assigned to stage vault_kv2_patches_sh.

If in a job other artifacts are defined, use

  dependencies:
    - vault_kv2_patches_sh

to ensure fetching vault_kv2_patches.sh from artifacts.

The shell script vault_kv2_patches.sh interpretes a yaml file describing vault key value pairs, for instance:

kv2_patches:
  - VAULT_AUTH_ROLE: mailcow
    TLSA_DNS_RECORD_VALUE:
      mount: gitlab
      path:  mailcow/kv
      field: TLSA_dns_record_value
    DKIM_DNS_RECORD_VALUE: 
      mount: gitlab
      path:  mailcow/kv
      field: DKIM_dns_record_value
  - VAULT_AUTH_ROLE: terraform
    TOKEN: 
      path:  gitlab/terraform/kv
      field: token
    KEY:
      path:  gitlab/terraform/kv
      field: key

path and field are mandatory, mount is optional.

The usage is

./vault_kv2_patches.sh <vault-auth-role>

The script by default outputs the commands to put the key value pairs described in the yaml file kv2_patches into vault.

option can be

--markdown / -m
output a markdown table documenting the key value pairs

There are two usages:

./vault_kv2_patches.sh kv2_patches.yml

shows the commands produced by vault_kv2_patches.sh.

./vault_kv2_patches.sh kv2_patches.yml >.kv2_patches && . .kv2_patches && rm .kv2_patches

executes the commands produced by vault_kv2_patches.sh in the execution context.

You have to set VAULT_ADDR and possibly VAULT_CACERT for using vault_kv2_patches.sh.

It is a good pratice to use YAML anchors for scripts by defining in the CI definition

.after-script-vault: &after-script-vault
  - ./vault_kv2_patches.sh kv2_patches.yml >.kv2_patches && . .kv2_patches && rm .kv2_patches

markdown

You can output a Markdown table documenting the secrets by

./vault_kv2_patches.sh kv2_patches.yml --markdown

For the kv2 patches yaml example above the result is

role	path	key	value
mailcow	gitlab/mailcow/kv	TLSA_dns_record_value	TLSA_DNS_RECORD_VALUE
mailcow	gitlab/mailcow/kv	DKIM_dns_record_value	DKIM_DNS_RECORD_VALUE
mailcow	gitlab/mailcow/kv2	TLSA_dns_record_value2	TLSA_DNS_RECORD_VALUE2
mailcow	gitlab/mailcow/kv2	DKIM_dns_record_value2	DKIM_DNS_RECORD_VALUE2
terraform	gitlab/mailcow/kv	TLSA_dns_record_value	TLSA_DNS_RECORD_VALUE
terraform	gitlab/mailcow/kv	DKIM_dns_record_value	DKIM_DNS_RECORD_VALUE
terraform	gitlab/mailcow/kv2	TLSA_dns_record_value2	TLSA_DNS_RECORD_VALUE2
terraform	gitlab/mailcow/kv2	DKIM_dns_record_value2	DKIM_DNS_RECORD_VALUE2

This output can pasted into the README.md of the project for documentation purpose.

Name		Name	Last commit message	Last commit date
Latest commit History 343 Commits
terraform		terraform
trivy		trivy
vault		vault
.ansible-lint		.ansible-lint
.gitignore		.gitignore
.talismanrc		.talismanrc
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

gitlab-ci.yml

terraform

before_script_secrets_vars

before_script_secrets_env_files

before_script_vault_token

before_script_ansible_requirements

trivy

vault

vault secrets (v2)

vault secrets

Test

Markdown for vault_secrets

vault token

vault kv puts

Markdown for vault_kv_puts

vault kv2 patches

markdown

About

Releases

Packages

Contributors 3

Languages

l-with/DevOps-GitLab-CI-Artifacts

Folders and files

Latest commit

History

Repository files navigation

gitlab-ci.yml

terraform

before_script_secrets_vars

before_script_secrets_env_files

before_script_vault_token

before_script_ansible_requirements

trivy

vault

vault secrets (v2)

vault secrets

Test

Markdown for vault_secrets

vault token

vault kv puts

Markdown for vault_kv_puts

vault kv2 patches

markdown

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 3

Languages

Packages