0.7.0

Calculate component dependencies from stack imports. Add `deps` output for each component @aknysh (#32)

what

• Calculate component dependencies from stack imports
• Add deps output for each component

why

• Reduce unnecessary stack runs in CI/CD systems like Spacelift

• The provider already calculates these parameters:

  • imports - a list of all imports (all levels) for the stack
  • stacks - a list of all stacks in the infrastructure where the component is defined

• Both imports and stacks are not 100% suitable to correctly determine the YAML config files that a component depends on

  • imports is too broad. The provider returns all direct and indirect imports for the stack, even those that don't define any variables for the component. This will trigger the component's stack in Spacelift even if the unrelated imports are modified resulting unrelated stack runs in Spacelift
  • stacks is too broad and too narrow at the same time. On the one hand, it detects all stacks in the infrastructure where the component is defined, but we don't need to trigger a particular stack in Spacelift if some other top-level YAML stack configs are modified. On the other hand, it misses the cases where a YAML stack config file specifies global variables, which are applied to the component as well

For example, in examples/data-sources/utils_stack_config_yaml/stacks/catalog/eks-defaults.yaml we define eks-iam terraform component. The provider returns the following:

stacks:
  - catalog/eks-defaults
  - globals
  - uw2-dev
  - uw2-globals
  - uw2-prod
  - uw2-staging
  - uw2-uat

imports:
  - catalog/eks-defaults
  - catalog/rds-defaults
  - catalog/s3-defaults
  - globals
  - uw2-globals

imports returns the unrelated catalog/s3-defaults and catalog/rds-defaults which will unnecessary trigger this component stack in Spacelift.

stacks returns all top-level YAML stack configs in the infrastructure all of which will unnecessary trigger this component stack in Spacelift.

We need to return only those YAML config files if any of the following conditions is true:

1. The imported config file has the global vars section and it's not empty
2. The imported config file has the component type section, which has a vars section which is not empty
3. The imported config file has the "components" section, which has the component type section, which has the component section
4. The imported config file has the "components" section, which has the component type section, which has the base component section

For example:

vars:
  stage: dev
terraform:
  vars: {}
components:
  terraform:
    aurora-postgres:
      vars:
        instance_type: db.r4.large
        cluster_size: 1
    aurora-postgres-2:
      component: aurora-postgres
      vars:
        instance_type: db.r4.xlarge
  helmfile: {}

vars and terraform.vars are global vars and component-type vars sections
components is the components section
components.terraform is the component type section (as is components.helmfile) - those are component types and get processed similarly
components.terraform.aurora-postgres is the base component section
components.terraform.aurora-postgres-2 is the component section

The changes in this PR return the correct YAML config dependencies for the components, e.g. for eks-iam:

eks-iam:
  deps:
    - catalog/eks-defaults
    - globals
    - uw2-globals

because globals and uw2-globals have global variables for the component, and catalog/eks-defaults defines the component itself.

This will trigger the Spacelift component stack only if these files are modified (in addition to the component project files and the top-level stack, e.g. uw2-prod, themselves).

0.6.0

Change go version in release action @nitrocode (#39)

what

• Updates github release action to use 1.16

Do the go.sum and go.mod files also need to be updated ?

why

• The go releaser worked for me locally because I'm using go 1.16 but did not work in github action as go 1.15 still does not allow darwin arm releases.

references

• Previous PR #38
• Closes #33

0.5.0

Use go 1.6.3 @nitrocode (#38)

what

• Update to go 1.6.3

why

• Support darwin arm

See https://github.com/cloudposse/terraform-provider-utils/runs/2536496044?check_suite_focus=true

      • building binaries
         • DEPRECATED: skipped darwin/arm64 build on Go < 1.16 for compatibility, check https://goreleaser.com/deprecations/#builds-for-darwinarm64 for more info.

references

• Closes #33

0.4.4

🚀 Enhancements

Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.5.0 to 2.6.1 @dependabot (#31)

Bumps github.com/hashicorp/terraform-plugin-sdk/v2 from 2.5.0 to 2.6.1.

Changelog

Sourced from github.com/hashicorp/terraform-plugin-sdk/v2's changelog.

2.6.1 (April 23, 2021)

BUG FIXES:

• Updated the GPG key used to verify Terraform installs in response to the Terraform GPG key rotation. (#750)

2.6.0 (April 21, 2021)

ENHANCEMENTS:

• Made TF_ACC_TERRAFORM_VERSION more permissive, accepting values in either vX.Y.Z or X.Y.Z formats. (#731)
• Upgraded to use terraform-plugin-go v0.3.0 (#739)

Commits

• 5747fc3 v2.6.1 [skip ci]
• 89b02dd Update CHANGELOG.md
• 0e71dce Update CHANGELOG.md
• 6df3604 bump terraform-exec to 0.13.3
• db1d38f remove internal/vault/helper
• aeeb59e Link to v1-maint support policy.
• 112e216 v2.6.0 [skip ci]
• 861cea2 Update CHANGELOG.md
• 06bde0d Update to tagged v0.3.0 release.
• e42067a Upgrade terraform-plugin-go to 0.3.0 (dev)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

0.4.3

Fix target branch of workflow @Nuru (#28)

what

• Set auto-release to run on push to main branch

why

• Was set to run on push to master branch, but this repo does not have a master branch, so it never ran

🐛 Bug Fixes

Use thread safe function @Nuru (#29)

what

• Use thread safe function to generate object IDs

why

• Terraform performs actions in parallel, so using a global variable is not safe

panic: d.nx != 0
plugin.terraform-provider-utils_v0.4.2: 
plugin.terraform-provider-utils_v0.4.2: goroutine 61 [running]:
plugin.terraform-provider-utils_v0.4.2: crypto/sha1.(*digest).checkSum(0xc000497778, 0x0, 0x0, 0xec943cb500000000)

0.4.2

what

• [stack_processor]: Add process_stack_deps input var to the stack_config data source

why

• Make the settings configurable (false by default)
• Not all provider invocations need to process all stack dependencies for the components (e.g. Spacelift module needs it, remote-state does not)
• Makes invocations without processing stack dependencies 2-3 times faster

0.4.1

what

• [stack_processor]: Read and parse only YAML files

why

• Non-YAML files in the stacks folder cause the YAML parser to panic

0.4.0 Add imports to stack output. Add `stacks` to each component output

what

• Add imports to stack output
• Add stacks to each component output

why

• imports attribute shows all imported stacks for the current stack - can be used in CI/CD pipelines to determine stack dependencies
• stacks attribute shows all the stacks the component (and its base component, if present) is declared in - can be used in CI/CD pipelines (e.g. Spacelift) to determine all stacks that the component depends on,
  and to provision triggers for all the dependencies (once any of the stack config files changes, the component's job will be triggered)

test

imports:
  - eks/eks-defaults
  - globals
  - uw2-globals

terraform:
  aurora-postgres:
    backend:
      acl: bucket-owner-full-control
      bucket: eg-uw2-root-tfstate
      dynamodb_table: eg-uw2-root-tfstate-lock
      encrypt: true
      key: terraform.tfstate
      region: us-west-2
      role_arn: arn:aws:iam::XXXXXXXXXXXX:role/eg-gbl-root-terraform
      workspace_key_prefix: aurora-postgres
    backend_type: s3
    stacks:
    - globals
    - uw2-dev
    - uw2-globals
    - uw2-prod
    - uw2-staging
    - uw2-uat
    env:
      ENV_TEST_1: test1
      ENV_TEST_2: test2
      ENV_TEST_3: test3
      ENV_TEST_4: test4
      ENV_TEST_5: test5
      ENV_TEST_6: test6
      ENV_TEST_7: test7
    settings:
      spacelift:
        autodeploy: false
        workspace_enabled: false
      version: 0
    vars:
      cluster_size: 1
      environment: uw2
      instance_type: db.r4.large
      namespace: eg
      region: us-west-2
      stage: dev
  aurora-postgres-2:
    backend:
      acl: bucket-owner-full-control
      bucket: eg-uw2-root-tfstate
      dynamodb_table: eg-uw2-root-tfstate-lock
      encrypt: true
      key: terraform.tfstate
      region: us-west-2
      role_arn: arn:aws:iam::XXXXXXXXXXXX:role/eg-gbl-root-terraform
      workspace_key_prefix: aurora-postgres
    backend_type: s3
    component: aurora-postgres
    stacks:
    - globals
    - uw2-dev
    - uw2-globals
    - uw2-prod
    - uw2-staging
    - uw2-uat
    env:
      ENV_TEST_1: test1_override2
      ENV_TEST_2: test2_override2
      ENV_TEST_3: test3
      ENV_TEST_4: test4
      ENV_TEST_5: test5
      ENV_TEST_6: test6
      ENV_TEST_7: test7
      ENV_TEST_8: test8
    settings:
      spacelift:
        autodeploy: true
        branch: dev
        triggers: []
        workspace_enabled: true
      version: 0
    vars:
      cluster_size: 1
      environment: uw2
      instance_type: db.r4.xlarge
      namespace: eg
      region: us-west-2
      stage: dev

helmfile:
  alb-controller:
    stacks:
    - eks/eks-defaults
    - uw2-dev
    - uw2-prod
    - uw2-staging
    - uw2-uat
    env: {}
    settings: {}
    vars:
      account_number: "1234567890"
      chart_values:
        enableCertManager: true
      environment: uw2
      installed: true
      namespace: eg
      region: us-west-2
      ssm_region: us-west-2
      stage: dev

0.3.1

Ensure results have unique IDs (#20)

0.3.0

what

• Add settings sections to data_source_stack_config_yaml data source to provide settings for Terraform and helmfile components

• Add env sections to data_source_stack_config_yaml data source to provide ENV vars for Terraform and helmfile components

• Workaround for a deep-merge bug in mergo.Merge()

• Fix failing tests

• Add stack_processor_test.go

why

• settings sections are deep-merged and used for consumption by external services (e.g. for Spacelift and Terraform Cloud Terraform modules)
• env sections are deep-merged and used to specify ENV vars for consumption by external services (e.g. for Spacelift and Terraform Cloud)
• Workaround for a deep-merge bug in mergo.Merge(). When deep-merging slice of maps in a for loop,
  mergo modifies the source of the previous loop iteration if it's a complex map and mergo gets a pointer to it,
  not only the destination of the current loop iteration.
• stack_processor_test.go to test the provider outputs using a YAML stack config

test

Click to show

Given this config:

terraform:
  vars: {}
  settings:
    spacelift:
      workspace_enabled: false
      autodeploy: false
  env:
    ENV_TEST_1: test1
    ENV_TEST_2: test2
    ENV_TEST_3: test3

    aurora-postgres:
      vars:
        instance_type: db.r4.large
        cluster_size: 1
      env:
        ENV_TEST_4: test4
        ENV_TEST_5: test5
        ENV_TEST_6: test6
        ENV_TEST_7: test7

    aurora-postgres-2:
      component: aurora-postgres
      vars:
        instance_type: db.r4.xlarge
      settings:
        spacelift:
          workspace_enabled: true
          autodeploy: true
          branch: "dev"
          triggers: []
      env:
        ENV_TEST_1: test1_override2
        ENV_TEST_2: test2_override2
        ENV_TEST_8: test8

    eks:
      vars:
        spotinst_instance_profile: eg-gbl-dev-spotinst-worker
        spotinst_oceans:
          main:
            desired_group_size: 1
            max_group_size: 3
            min_group_size: 1
            kubernetes_version: null
            ami_release_version: null
            attributes: null
            disk_size: 100
            instance_types: null
            ami_type: "AL2_x86_64"
            tags: null
      settings:
        spacelift:
          workspace_enabled: true
          autodeploy: true
          branch: "test"
          triggers: []
      env:
        ENV_TEST_1: test1_override
        ENV_TEST_2: test2_override
        ENV_TEST_4: test4

it produces the following outputs:

uw2_uat_aurora_postgres_2_settings = {
  "spacelift" = {
    "autodeploy" = true
    "branch" = "dev"
     "triggers" =  []
    "workspace_enabled" = true
  }
}

uw2_dev_aurora_postgres_2_env = {
  "ENV_TEST_1" = "test1_override2"
  "ENV_TEST_2" = "test2_override2"
  "ENV_TEST_3" = "test3"
  "ENV_TEST_4" = "test4"
  "ENV_TEST_5" = "test5"
  "ENV_TEST_6" = "test6"
  "ENV_TEST_7" = "test7"
  "ENV_TEST_8" = "test8"
}

uw2_uat_aurora_postgres_settings = {
  "spacelift" = {
    "autodeploy" = false
    "workspace_enabled" = false
  }
}

uw2_dev_aurora_postgres_env = {
  "ENV_TEST_1" = "test1"
  "ENV_TEST_2" = "test2"
  "ENV_TEST_3" = "test3"
  "ENV_TEST_4" = "test4"
  "ENV_TEST_5" = "test5"
  "ENV_TEST_6" = "test6"
  "ENV_TEST_7" = "test7"
}

uw2_uat_eks_settings = {
  "spacelift" = {
    "autodeploy" = false
    "branch" = "test"
    "triggers" = []
    "workspace_enabled" = true
  }
}

uw2_dev_eks_env = {
  "ENV_TEST_1" = "test1_override"
  "ENV_TEST_2" = "test2_override"
  "ENV_TEST_3" = "test3"
  "ENV_TEST_4" = "test4"
}