feat: stream output for custom workflows

[ascandella]

Fixes #2054

I noticed that in #1937 which originally added this feature (e.g. https://github.com/runatlantis/atlantis/pull/1937/files#diff-edf527ba8643ff7bfca5f560491ea7055af472f5d6f3bbda127f1776b63d4b06L179) that the documentation around setting up terragrunt for custom workflows removed the -no-color option.

I'm not sure if this was by mistake, but to allow colorization in the in-browser terminal I've added ansi.Strip() for parsing all command output from the new runner abstraction. If this is out of scope I can back those changes out (but we should probably update the docs to re-add the -no-color flags to terragrunt workflows, otherwise the output is mangled in e.g. GitHub comments).

Tested by updating my atlantis instance to a build of this branch and removing the -no-color flags from my workflow.

I'm using the following server configuration:

repos:
  - id: "/.*/"
    workflow: terragrunt
    pre_workflow_hooks:
      - run: >
          terragrunt-atlantis-config generate
          --output atlantis.yaml
          --autoplan --automerge
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -out=$PLANFILE
      - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply $PLANFILE

Output as usual in GitHub comments (I have --enable-diff-markdown-format on):

[jamengual]

you could check : #1751 and #1325 or ask the contributors.

[ascandella]

I don't think this is a particularly good place for this, but noticed it was where the other exec functionality was put. Not totally sure on the naming convention of this project, but it couldn't be in runtime because of import cycles with the terraform package.

[ascandella]

This code was extracted as-is from terraform_client.go

[ascandella]

you could check : #1751 and #1325 or ask the contributors.

@jamengual can you clarify on the "waiting-for-response" label? Is that with regards to the ANSI stripping, or something else?

[jamengual]

you could check : #1751 and #1325 or ask the contributors.

@jamengual can you clarify on the "waiting-for-response" label? Is that with regards to the ANSI stripping, or something else?

usually PRs do not get updated same day and linger for quite a while, it is a way to track when we are waiting for something from the contributor, nothing special

[ascandella]

@ascandella need any help with this? Very interested in this feature!

Nope, this is ready to go on my end, just waiting for review.

[mustafa89]

Thanks, waiting eagerly for this. Any chances for a review @jamengual ?

[adutchak]

We're looking forward for having this feature too :)

[SudoSpartanDan]

@ascandella I think they fixed some build bugs in the upstream, can you fetch from upstream so we can get green checkmarks on this PR?

[ascandella]

@SudoSpartanDan thanks for the heads up. I've merged the latest master into this branch

[frank-bee]

I tried to work with the docker image produced by this PR plus terragrunt. Somehow I always went into lock problems but not really reproducible.
Maybe an upstream Problem?anyone else faced similar issues?

[jamengual]

I'm interesting in merging this to the prerelease but @frank-bee comment makes me wonder @ascandella is there anything you could assist here? @frank-bee can you describe your atlantis config and atlantis.yaml?

[ascandella]

I've been running this branch on my install for about a month now without issues. I haven't rebuilt/redeployed since I merged master in last week though. I'll try that now.

@frank-bee very curious what you mean by "lock problems". Could you provide any more detail/logs/observed behavior?

[frank-bee]

Hi @jamengual and @ascandella ,

here some more info...

"lock problems"
error message: The default workspace at path teams/aibse is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.

atlantis repo config

version: 3
automerge: true
delete_source_branch_on_merge: true
projects:
  - name: aibse
    dir: teams/aibse
    workflow: terragrunt
    terraform_version: v1.2.2
    autoplan:
      when_modified: ["*.tf", "*.hcl"]
      enabled: true
  - name: platform
    dir: teams/platform
    workflow: terragrunt
    terraform_version: v1.2.2
    autoplan:
      when_modified: ["*.tf", "*.hcl"]
      enabled: true
workflows:
  terragrunt:
    plan:
      steps:
        - env:
            name: TERRAGRUNT_TFPATH
            command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - run: terragrunt plan -out=$PLANFILE
        - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
        - env:
            name: TERRAGRUNT_TFPATH
            command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - run: terragrunt apply $PLANFILE

atlantis server config

# repos lists the config for specific repos.
repos:
  # id can either be an exact repo ID or a regex.
  # If using a regex, it must start and end with a slash.
  # Repo ID's are of the form {VCS hostname}/{org}/{repo name}, ex.
  # github.com/runatlantis/atlantis.
  - id: /.*/
    # branch is an regex matching pull requests by base branch
    # (the branch the pull request is getting merged into).
    # By default, all branches are matched
    branch: main
    # apply_requirements sets the Apply Requirements for all repos that match.
    apply_requirements: [mergeable, undiverged]
    # workflow sets the workflow for all repos that match.
    # This workflow must be defined in the workflows section.
    workflow: default
    # allowed_overrides specifies which keys can be overridden by this repo in
    # its atlantis.yaml file.
    allowed_overrides: [apply_requirements, workflow, delete_source_branch_on_merge]
    # allowed_workflows specifies which workflows the repos that match
    # are allowed to select.
    allowed_workflows: [default]
    # allow_custom_workflows defines whether this repo can define its own
    # workflows. If false (default), the repo can only use server-side defined
    # workflows.
    allow_custom_workflows: true
    # delete_source_branch_on_merge defines whether the source branch would be deleted on merge
    # If false (default), the source branch won't be deleted on merge
    delete_source_branch_on_merge: false
    # pre_workflow_hooks defines arbitrary list of scripts to execute before workflow execution.
    #pre_workflow_hooks:
    #  - run: my-pre-workflow-hook-command arg1
    # post_workflow_hooks defines arbitrary list of scripts to execute after workflow execution.
    #post_workflow_hooks:
    #  - run: my-post-workflow-hook-command arg1
# workflows lists server-side custom workflows
workflows:
  default:
    plan:
      steps:
        - init
        - plan
    apply:
      steps: [apply]

[endriu0]

From my own experience with Atlantis / Terragrunt I run into a lot of issues with autoinit.

Due to this I went with below setup. Granted I also had to account for workspaces being used so that could be part of the reason.

                steps:
                  - env:
                      name: TERRAGRUNT_TFPATH
                      command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
                  - env:
                      name: TERRAGRUNT_AUTO_INIT
                      value: false
                  - run: terragrunt init -input=false -no-color
                  - run: if [[ $WORKSPACE != "default" ]]; then terragrunt workspace select -no-color $WORKSPACE; fi
                  - run: terragrunt validate -no-color
                  - run: terragrunt plan -no-color -out=$PLANFILE

[ascandella]

@frank-bee thanks for the info. Out of curiosity, are you saying that these problems happen only when running Atlantis from this branch?

Just trying to rule things out, since it's not really clear to me how the changes here could affect the locking behavior.

[mustafa89]

@frank-bee if there is something kindly update here. This PR missed the pre release and we would very much like to have it.

[frank-bee]

@ascandella and @mustafa89 I have no updates here. Also cannot reproduce it at the moment because I switched to the released atlantis and do not work with terragrunt anymore.
If nobody else faced thesee kind of problems, please merge and close this PR!

[SudoSpartanDan]

@jamengual can we get this into the next release? Our team would love to start using this feature

[jamengual]

@jamengual can we get this into the next release? Our team would love to start using this feature

I'm not the only one that needs to review the code, so we need to wait for someone else to review it.

[SudoSpartanDan]

@jamengual who else can review? This PR's been sitting for a bit and I know folks have been waiting for this feature.

[mustafa89]

I was testing this in the pre-release and it seems the same problem @frank-bee has raised happened to us. I saw multiple MRs where one of the workspaces started showing the same

The default workspace at path ... is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.

to the extent I had to kill the processes in the atlantis pod.

I've verified it and it's definitely a bug. This happens in specific scenarios only when Terraform has to wait for an input as far as I've seen.

To reproduce:

Declare a variable without a value, so Terraform waits for the input. On 0.19.4 terraform simply bypasses this prompt and exits and releases the command lock.

variable "test_variable" {
  description = "Testing"
  type        = string
}

With this change you can see the plan commands stuck when you do a ps aux.

[ascandella]

@mustafa89 thanks for the details. Looking at the built-in terraform client, it seems to append -input=false to plan and apply commands.

Does it work for you if you add the same flags to your workflow? For example:

workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -input=false -no-color -out=$PLANFILE
      - run: terragrunt show -no-color -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply -input=false $PLANFILE -no-color

If so, then this is probably more of a documentation fix than something that can be done in code, since we're running arbitrary shell commands (with the run: config) and probably shouldn't be mangling those without the user's knowledge.

[mustafa89]

Good point, I guess -input=false should be added regardless when working with Atlantis. I just added it and as expected, all good now. Thanks for the help 👍

[ascandella]

Cool, glad that worked! Opened #2350 to update the terragrunt workflow docs