Ability to setup aws_codepipeline to use webhooks instead of polling.

[jwhitcraft]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Support the new feature on aws_codepipeline of allowing webhooks to trigger the pipeline instead of polling github.

New or Affected Resource(s)

• aws_codepipeline

Potential Terraform Configuration

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

aws_codepipeline "default" {
  ...
  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["source"]

      configuration {
        ...
		PollForSourceChanges = false
	  }
    }
}

References

• https://aws.amazon.com/about-aws/whats-new/2018/05/aws-codepipeline-supports-push-events-from-github-via-webhooks/
• https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-webhooks-migration.html

[ryno75]

I think there is more to this than just adding the put/register webook capabilities to the provider.

I thought I would be crafty and just use the local-exec provisioner inside my aws_codepipeline resource to run the AWS CLI and perform the put-webhook and register-webhook-with-third-party commands as a workaround until this was supported. Unfortunately I always get a 401 error from github on the register-webhook-with-third-party call when doing this. I have figured out why this is happening. In short... aws_codepipeline does not appear to be registering CodePipeline as an authorized OAuth application in GitHub (under my personal settings in GitHub under "Applications" and "Authorized OAuth Apps" there is a missing entry for AWS CodePipeline (Oregon) after I run my terraform code and create my pipeline (which definitely has a Source stage referncing my Github OAuth token. this...

  stage {
    name = "Source"

    action {
      name             = "${local.src_action_name}"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      output_artifacts = ["gonzo-src"]
      version          = "1"

      configuration {
        Owner                = "${local.github_owner}"
        Repo                 = "${local.repo_name}"
        Branch               = "master"
        OAuthToken           = "${var.github_token}"
        PollForSourceChanges = "false"
      }
    }

Once my pipeline is created with Terraform I can go into the AWS CodePipeline console and see it has failed on sourcing my GitHub project. If I edit the source and click the Connect to GitHub button, go through the steps, select my repo and branch and save, it then starts working and I see the AWS CodePipeline (Oregon) entry show up under my Authorized OAuth Apps in GitHub. At this point I can go back to the aws console and run the exact same AWS CLI register-webhook-with-third-party command and it works flawlessly.

To summarize... I believe the aws_codepipeline resource code (or perhaps the go AWS SDK?) needs to be modified to reach out to GitHub and register AWS CodePipeline as an authorized OAuth application if a GitHub source with an OAuthToken is provided in the configuration. I am uncertain if this is a bug or a feature request though.

I can tell you this... if you define your pipeline using JSON and create it using the AWS CLI create-pipeline command it in fact does successfully register AWS CodePipeline as an authorized OAuth application.

[joestump]

I've started work on an aws_codepipeline_webhook resource. This API endpoint lets you create three kinds of webhooks: no auth, GitHub HMAC auth, or IP whitelisting. Technically, all three should work fine with GitHub. The OAuth flow is for automatically tying things together. If your CodeBuild/CodePipeline has an SSH key with GitHub access, I'm not sure the OAuth flow is required.

You'd then use the webhook URL it returns along with the secret you create for it when creating a github_repository_webhook.

You might use it kind of like this:

locals {
  webhook_secret = "a09s8df7asd0f97" # Can be anything
}

resource "aws_codepipeline_webhook" "github_to_foo" {
  name = "foo-pipeline-to-github"

  auth {
    type   = "GITHUB_HMAC"
    secret = "${local.webhook_secret}"
  } 

  target {
    action   = "CodeCheckoutActionNameHere"  
    pipeline = "foo-pipeline"
  }
}

resource "github_repository_webhook" "foo" {
  repository = "${github_repository.repo.name}"
  name    = "web"

  configuration {
    url    = "${aws_codepipeline_webhook.github_to_foo.url}"
    secret = "${local.webhook_secret}"
  }
}

[jwhitcraft]

@joestump what is the status of this, It would be really nice to get it in so we can use webhooks now.

[joestump]

@jwhitcraft the status is I need to get off my lazy butt and finish my PR already. 😄

[gregglowrimore]

We are anxiously awaiting this feature too.

[joestump]

I've got a WIP PR pushed to #5875. Hoping to finish testing soon.

[jpdoria]

I'm also very excited for this feature @joestump! I had to use local-exec to make Terraform do put and register API calls.

[joestump]

Spent some time today cleaning up PR #5875. It should be ready for final review.

[bflad]

The new aws_codepipeline_webhook resource has been merged and will release with version 1.41.0 of the AWS provider, likely later today. 🎉

[bflad]

This has been released in version 1.41.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[BouchaaraAdil]

does aws_codepipeline_webhook support github release Webhook event
i tried this configuration and it didn't work

resource "aws_codepipeline_webhook" "gh_webhook" {
    name            = "prod"
    authentication  = "GITHUB_HMAC"
    target_action   = "Source"
    target_pipeline = "${aws_codepipeline.cd.name}"

    authentication_configuration {
      secret_token = "${local.webhook_secret}"
    }

    filter {
      json_path    = "$.action"
      match_equals = "published"
    }
}

resource "github_repository_webhook" "gh_webhook" {
  repository = "${var.gh_repo}"

  name = "awscodepipeline"

  configuration {
    url          = "${aws_codepipeline_webhook.gh_webhook.url}"
    content_type = "json"
    insecure_ssl = true
    secret       = "${local.webhook_secret}"
  }
  events = ["release"]
}

when i release on github the payload then is delivered to codepipeline_webhook but is not consumed and the pipeline not triggered
Advice on that please.

[BouchaaraAdil]

could you please advice on that ⬆️
@jwhitcraft @ryno75 @joestump @jpdoria @gregglowrimore

[joestump]

I tested this live with the push event. It should work for any JSON payload that matches the filter block. Try changing name = "awscodepipeline" to name = "web". Does the hook show up and what happens when you test it?

[BouchaaraAdil]

sorry i already tried web before changed to awscodepipeline for testing purpose, is working now as expected with web, as i configured two aws_codepipeline_webhook , aws_codepipeline_webhook for stage/prod in parallel with different filters, but when i recreated resources last time it didn't work and i think is that sometimes aws can't register the webhook to a Pipeline instance ..

[ghost]

what is the difference when you want to use for an Organization ??
as we can not use webhook_secret ??
could you help me please with that

when i am use OAuthToken is working fine

aws_codepipeline.mypipeline: [ERROR] Error updating CodePipeline (codepipeline-dev-Terraform-development): InvalidActionDeclarationException: Action configuration for action 'Source' is missing required configuration 'OAuthToken'

• github_repository_webhook.bar: POST https://api.github.com/repos********/hooks: 401 Bad credentials

[nathanielks]

There's a caveat when using the GitHub provider where you have to specify the token to authenticate via a GITHUB_TOKEN environment variable. See the top of the docs here: https://www.terraform.io/docs/providers/aws/r/codepipeline.html

…

On Mon, Nov 12, 2018, 9:49 AM michaelm88 ***@***.*** wrote: what is the difference when you want to use for an Organization ?? as we can not use webhook_secret ?? could you help me please with that when i am use OAuthToken is working fine aws_codepipeline.mypipeline: [ERROR] Error updating CodePipeline (codepipeline-dev-Terraform-development): InvalidActionDeclarationException: Action configuration for action 'Source' is missing required configuration 'OAuthToken' - github_repository_webhook.bar: POST https://api.github.com/repos********/hooks: 401 Bad credentials — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4478 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATKmd2vIu8b5up1-6BRLUhKNtodT06Zks5uubSsgaJpZM4T2nhV> .

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!