aws_codepipeline with Github OAuth causing persistent changes

[cornfeedhobo]

Terraform Version

Terraform v0.11.1
Terraform AWS Provider v1.6.0

Affected Resource(s)

Please list the resources as a list, for example:

• aws_codepipeline

Terraform Configuration Files

resource "aws_codepipeline" "build" {
  name     = "pipeline-test"
  role_arn = "pipeline-test"

  artifact_store {
    type     = "S3"
    location = "pipeline-test-bucket"
  }

  stage {
    name = "Source"

    action {
      name     = "Source"
      category = "Source"
      owner    = "ThirdParty"
      provider = "GitHub"
      version  = "1"

      output_artifacts = ["code"]

      configuration {
        OAuthToken           = "${var.github_token}"
        Owner                = "${var.github_owner}"
        Repo                 = "${var.github_repo}"
        Branch               = "${var.github_branch}"
        PollForSourceChanges = "true"
      }
    }
  }

  stage {
    name = "Build"

    action {
      name     = "Build"
      category = "Build"
      owner    = "AWS"
      provider = "CodeBuild"
      version  = "1"

      input_artifacts  = ["code"]
      output_artifacts = ["package"]

      configuration {
        ProjectName = "${var.project_name}"
      }
    }
  }
}

Expected Behavior

Subsequent executions of terraform apply should not result in updates to the source attributes.

Actual Behavior

Running terraform plan/terraform apply always results in a change:

  ~ aws_codepipeline.build
      stage.0.action.0.configuration.%:          "4" => "5"
      stage.0.action.0.configuration.OAuthToken: "" => "REDACTED"

And AWS is incapable of accessing Github, even though the token is valid, tested, and with the correct scopes.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan
2. terraform apply
3. terraform apply

[cornfeedhobo]

The solution is to use the environment variable GITHUB_TOKEN.
Edit: This is wrong, see below.

[ndench]

Even when I specify the GITHUB_TOKEN environment variable I still get the same issue as the OP. Is there something else you need to do as well?

[michaelmoussa]

I'm experiencing the same issue, but managed to work around it by adding the following to my aws_codepipeline resource:

  # Workaround for Terraform insisting on "updating" OAuthToken every run. In
  # the event that the OAuthToken actually needs to be updated, comment out
  # the `lifecycle` block, run `terraform apply`, and then restore the block.
  #
  # https://github.com/terraform-providers/terraform-provider-aws/issues/2854
  lifecycle {
    ignore_changes = [
      "stage.0.action.0.configuration.OAuthToken",
      "stage.0.action.0.configuration.%",
    ]
  }

The GitHub token isn't likely to change often in my use case, so the inconvenience of having to remove & restore that lifecycle block is not a big deal compared to having to confirm that I want to "change" the token on every single run (and having it displayed on the screen in plaintext each time, too).

It doesn't address the root cause, but hopefully someone else will find this workaround useful.

EDIT (2019-05-09): See my updated workaround below if you're experiencing this problem with Terraform 0.12.0-rc1 or newer.

[huksley]

The solution proposed by @michaelmoussa is good, but it is not applicable when you are using the module which, in turn, creates the aws_codepipeline resource.

It is very inconvenient to change the source code of that module to comment/uncomment lifecycle block all the time (if you have a group of infrastructure engineers). And downright impossible if you have it published in GitHub.

Another solution is to use conditional resources i.e. count={var.force_github_token ? 1 : 0} but the problem is I already have 6 different codepipeline resources and that, in turn, will lead to having 12 codepipeline resources.

• https://github.com/huksley/terraform-aws-cicd

[sunilkumarmohanty]

I had a look at the provider code and it seems that the OAuthToken is getting deleted from the state file.

if ok && actionProvider == "GitHub" {
				delete(config, "OAuthToken")
			}

I suspect this has been done to not store secrets in state file. However, in other resources like aws_db_instance, we store the passwords in state file. The state file always has been the single source of truth. The issue pointed out here violates that principal and kind of degrades the developer experience.

I suggest we change this behaviour and store the token in the state file and keep the experience consistent across resource.

Moreover, the OAuthToken value is taken from an environment variable, which is again not consistent with other resources.

[sunilkumarmohanty]

On further debugging, I found that the GetPipeline method of aws sdk for go returns **** instead of the actual OAuthToken, which means that the state file will always have **** in it instead of the actual OAuthToken. Hence, every time terraform plan is run, it will always state that the pipeline needs modification.

[cornfeedhobo]

@sunilkumarmohanty if that is the case, then let's just store the asterisk and move on. Who cares if it's not an absolute truth, as long as it stops breaking expectations.

[michaelmoussa]

Update for 0.12.0-rc1:

This is still broken in 0.12.0-rc1, but the workaround I posted a year ago (hacky birthday! 🎂) doesn't work anymore.

You'll first see an error saying "Dot must be followed by attribute name", which can be fixed by using stage[0].action[0] instead of stage.0.action.0. That will fix the .OAuthToken portion, but the .configuration.% portion will not work. I also tried .configuration[%] and even tried incorporating the splat operator, but no dice there ("Splat expressions (.*) may not be used here.").

The following approach will work in 0.12:

  # Workaround for Terraform insisting on "updating" OAuthToken every run. This
  # will prevent *any* updates to this CodePipeline resource while in place. In
  # the event that any real updates are needed, comment out the `lifecycle`
  # block, run `terraform apply`, and then restore the block.
  #
  # https://github.com/terraform-providers/terraform-provider-aws/issues/2854
  # https://www.terraform.io/docs/configuration/resources.html#ignore_changes
  lifecycle {
    ignore_changes = all
  }

NOTE: You could technically use ignore_changes = [stage] as well, which will allow you to update the CodePipeline resource itself as long as you don't modify the stages. I prefer the all approach, because it will make it more obvious that something is wrong if I try to modify the resource itself and the stages. Using [stage] would allow top-level attribute changes to take place, while ignoring the changes to the stage block, which could lead to unpredictable results and an all-around bad time.

[cryptiklemur]

You can't do ignore_changes = ["stage[0]"] either

[mattbryce93]

You can't do ignore_changes = ["stage[0]"] either

ignore_changes = [stage[0]] works though

ignore_changes = [stage[0].action[0]] works also to get one layer lower but anything I've tried to get into the configuration section has thus far failed 😞

[eshiri]

You can't do ignore_changes = ["stage[0]"] either

ignore_changes = [stage[0]] works though

ignore_changes = [stage[0].action[0]] works also to get one layer lower but anything I've tried to get into the configuration section has thus far failed 😞

I am facing the same issue. Couldn't ignore just the OAuthToken. Does anyone know of a solution?

[kevinburke1]

This is ugly but adding this in the lifecycle section worked for me

ignore_changes = [stage[0].action[0]]

[hlarsen]

looks like #2796 is related and #5764 would solve it - anyone have any thoughts?

[WGriffing]

I was able to get as far as:
ignore_changes = [stage[0].action[0].configuration]

However, I couldn't figure out how to specifically ignore one attribute of configuration such as OAuthToken either.

Ignoring the entire configuration won't work for my use case.

[nik-shornikov]

I was able to get as far as:
ignore_changes = [stage[0].action[0].configuration]

However, I couldn't figure out how to specifically ignore one attribute of configuration such as OAuthToken either.

Ignoring the entire configuration won't work for my use case.

i arrived at this too and its the superior workaround. however, like the doc says, maps cant be made to ignore newly added keys

so clearly the issue is that the read operation doesn't grab an empty or placeholder value for the token in the action configuration (which it obviously shouldnt be able to receive); instead it presents that key as missing

[cornfeedhobo]

looks like #2796 is related and #5764 would solve it - anyone have any thoughts?

@hlarsen i don't use this right now. hopefully someone else can respond.

[nl-brett-stime]

I had a look at the provider code and it seems that the OAuthToken is getting deleted from the state file.

I suspect this has been done to not store secrets in state file.

I suggest we change this behaviour and store the token in the state file and keep the experience consistent across resource.

@bflad @gdavison (please forward if someone else should be looking at the CodePipeline provider).

In the worst case, a hash of the OAuthToken could be stored in the state file so that we can do change-detection without having to expose the actual secret.

[Stretch96]

As @nl-brett-stime mentioned, if we could get the hashed password stored in the state file, it will allow to check for changes and also keep secrets secure(ish) - depends on the user to keep the state file private

We're experiencing this issue on the aws_codepipeline resource, OAuthToken in the source phase

Perhaps have it optional to store the hash

[bflad]

Hi folks 👋 This should be resolved, or at least now have different behavior with #14175 which was just merged and released with version 3.0.0 of the Terraform AWS Provider. If you are still having issues after upgrading to this release, please open a new issue and the maintainers will take a fresh look. 👍

[ghost]

This has been released in version 3.0.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!