Add a --compare-to flag to enable comparison between different runs

[hugorut]

Problem

Currently, Infracost enables users to see cost diffs between changes and the planned Terraform state. However, many Infracost users want to run cost diffs between branches or historical snapshots. See #1192 as an example of this.

Furthermore, Infracost parse HCL method currently is not supported by the diff command because parsing HCL does not use Terraform state. To support native HCL diffs we must allow users to provide a project state external to Terraform.

Proposed solution

To solve these problems the Infracost team proposes to add a --compare-to flag to the breakdown, diff and output
commands. Using this flag users can pass an Infracost JSON file (--format json --out-file infracost-run.json) to compare against. Below are some common scenarios using this interface.

Note for community
Please 👍 if you'd like to upvote or receive updates. Add a comment if you have feedback or your use-case can't be handled with this proposal.

1. Compare cost estimates across branches for a single project

# Checkout master to generate an Infracost output JSON that represents a cost snapshot of the project.
git checkout master
infracost breakdown --path . --terraform-parse-hcl  --format json --out-file infracost-master.json

# Checkout the feature branch that holds your infrastructure changes.
git checkout mybranch

# Compare the cost estimates across branches.
infracost diff --path . --terraform-parse-hcl --compare-to infracost-master.json

2. Compare cost estimates across branches for a multi-project setup

Multi-project setups include a config-file or Terragrunt being used.

# infracost-config.yaml
version: 0.1
projects:
  - path: infra/dev
    usage_file: infra/dev/infracost-usage.yml
  - path: infra/prod
    usage_file: infra/prod/infracost-usage.yml

# Checkout master to generate an Infracost output JSON that represents a cost snapshot of the project.
git checkout master
infracost breakdown --config-file infracost-config.yaml --terraform-parse-hcl --format json --out-file infracost-master.json

# Checkout the feature branch that holds your infrastructure changes.
git checkout mybranch

# Compare the cost estimates across branches.
infracost diff --config-file infracost-config.yaml --terraform-parse-hcl --compare-to infracost-master.json

# Generate an Infracost output JSON snapshot (that can also be stored in CI artifacts).
infracost breakdown --config-file infracost-config.yaml \
                    --terraform-parse-hcl \
                    --compare-to infracost-master.json \
                    --format json \
                    --out-file 2022-04-14-infracost-run.json

# Comment on a GitHub pull request with the combined project output.
infracost comment github --path 2022-04-14-infracost-run.json \
                         --repo $GITHUB_REPOSITORY \
                         --github-token ${{github.token}} \
                         --pull-request ${{github.event.pull_request.number}} \
                         --behavior update

3. Compare cost estimates across branches for a multi-project setup using a build matrix

Sometimes a build matrix is already being used, thus config-files are not an option (unlike case number 2).

name: Multi-project matrix
on: [ pull_request ]

jobs:
  multi-project-matrix:
    name: Multi-project matrix
    runs-on: ubuntu-latest

    strategy:
      matrix:

    steps:
      - uses: actions/checkout@v2

      - name: Setup Infracost
        uses: infracost/actions/setup@v1
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Run Infracost
        run: |
          # Checkout master to generate an Infracost output JSON that represents a cost snapshot of the project.
          git checkout master
          infracost breakdown --path ${{ matrix.dir }} \
                              --terraform-parse-hcl \
                              --format json \
                              --out-file /tmp/infracost_${{ matrix.dir }}_master.json

          # Checkout the feature branch that holds your infrastructure changes.
          git checkout mybranch

          # Generate an Infracost output JSON using the master snapshot to compare against.
          infracost breakdown --path  ${{ matrix.dir }} \
                              --terraform-parse-hcl \
                              --compare-to /tmp/infracost_${{ matrix.dir }}_master.json \
                              --format json \
                              --out-file /tmp/infracost_${{ matrix.dir }}.json

      - name: Upload Infracost breakdown
        uses: actions/upload-artifact@v2
        with:
          name: infracost_jsons
          path: /tmp/infracost_${{ matrix.dir }}.json

  multi-project-matrix-merge:
    name: Multi-project matrix merge
    runs-on: ubuntu-latest
    needs: [ multi-project-matrix ]

    steps:
      - uses: actions/checkout@v2

      - name: Download all Infracost JSON files
        uses: actions/download-artifact@v2
        with:
          path: /tmp

      - name: Setup Infracost
        uses: infracost/actions/setup@v1
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Combined Infracost projects into 1 run JSON file
        run: |
          # Combine all the project outputs into a single file with all the project costs (e.g. if you want to store it as a CI artifact).
          infracost output --path /tmp/infracost_jsons/*.json \
                           --format json \
                           --out-file 2022-04-14-infracost-run.json 

      - name: Post Infracost comment
        run: |
          # Post a comment to a GitHub PR with a summary table containing all the project costs.
          infracost comment github --path "/tmp/infracost_jsons/*.json" \
                                   --repo $GITHUB_REPOSITORY \
                                   --github-token ${{github.token}} \
                                   --pull-request ${{github.event.pull_request.number}} \
                                   --behavior update

4. Compare historical cost estimate snapshots

Assuming you have infracost-last-month.json and infracost-last-week.json files:

# Compare the current project against a snapshot.
infracost diff --path . --terraform-parse-hcl --compare-to infracost-last-month.json

# Compare two historical snapshots.
infracost output --path infracost-last-week.json --compare-to infracost-last-month.json --format diff

Out of scope

We've decided that it's best to add the --compare-to flag to breakdown, diff and output commands, but not to comment. This has been done to keep the comment interface simpler to understand. Additionally, we believe there are very few use cases where users would want to comment on PRs comparing two historical snapshots. However, this is still possible by using a combination of output and comment:

infracost output --path last-week.json --compare-to last-month.json --format json --out-file infracost.json

infracost comment github --path infracost.json \
                         --repo $GITHUB_REPOSITORY \
                         --github-token ${{github.token}} \
                         --pull-request ${{github.event.pull_request.number}} \
                         --behavior update

[brettcurtis]

I'm looking specifically at 3. Compare cost estimates across branches for a multi-project setup using a build matrix when thinking about this. This makes sense for our first workflow in a root module as well as child module development.

In root modules our use case is slightly different in that we have each environment setup as its own workflow and the only workflow that triggers on PR open, or sync is a sandbox environment. If all tests pass and code review is approved, we merge to main (we also have an option to destroy the sandbox on PR close if we want). The next workflow (let's say a acceptance environment) is triggered off that merge to main and the rest of the workflows (exploratory, production etc..) are triggered off the successful run of the previous workflow. If I understand correctly the functionality is already there for comparing plan diffs for my latter workflows so the combination of the two should get me where I need to go! This is an example of the workflows I describe a multi-region setup for google. It represents something similar to what I'm trying to get infracosts for. (This is just my open/playground area so I'm not building stuff that costs money :D )

For child modules we use kitchen-terraform and destroy after PR close so this action should work for us there too and give developers a clear understanding of how their change may impact costs. This is an example of what we'd use that action on. Not too different than what Google is doing with their public modules.

Can't wait to mess around with it all!

[hugorut]

@brettcurtis, @MNThomson, @lemeurherve the --compare-to functionality is live with the v0.9.23 release. Thanks for all your wonderful feedback and help making this a reality. You can check out our docs example here