dynamic-github-source

Problem

Current implementation of AWS CodeBuild (as of: 6/19/21) doesn't allow for dynamic repo and branch source. CodeBuild does allow for up to 12 secondary sources, although the buildspec would have to have additional logic to explicitly switch to the secondary source that was trigger via the CodeBuild Webhook. Another painful workaround is to create a Codebuild project for each repository. If each repo requires the same CodeBuild configurations, this can lead to multiple copies of the same CodeBuild project but with different sources. This can consequently clutter your AWS CodeBuild workspace especially if there are hundreds of repositories are included in this process.

Process

Steps

1. Github event is performed (e.g. user pushes new file, opens PR, etc.) that falls under one of the repository's event filters
2. Github webhook sends a POST HTTP method request to the API Gateway's (AGW) REST API
3. AGW request integration maps the request to a format friendly format {'headers': <Webhook headers>, 'body': <Webhook payload>} Processed request is passed to the request validator function.
4. Request validator function compares the sha256 value from the request header with the sha256 value created with the Github secret value and request payload. If the values are are not equal, then an error response is sent to the Github webhook. If the values are not equal, the Lambda function validates the payload against the filter groups defined for that repository. If the payload passes atleast one filter group, then the Lambda function succeeds invokes the next Lambda function asynchronously.
5. The second Lambda function triggers the Codebuild project with the repository's respective CodeBuild configurations ONLY for that build (after this build, CodeBuild project reverts to original configurations). The main attribute that is changed is the source configurations for the build which allow one CodeBuild to use a dynamic source.
6. CodeBuild performs the defined buildspec logic. User can define a global or repo-level buildspec that can be used for CI/CD, building AMIs, etc.

Non-Provider related Requirements

• The environment in which the Terraform module is applied must have pip within it's $PATH. The resource "null_resource" "lambda_pip_deps" {} installs the PyGithub package locally via pip and then is used within the payload validator function.

Usage

Minimal viable configuration:

module "dynamic_github_source" {
  source                         = "github.com/marshall7m/terraform-aws-dynamic-github-source"
  create_github_secret_ssm_param = true
  github_secret_ssm_value        = var.github_secret_ssm_value
  codebuild_buildspec            = file("buildspec.yaml")
  repos = {
    "test-repo" = {
      filter_groups = [
        [
          {
            type = "event"
            pattern = "push"
          }
        ]
      ]
    }
  }
}

Configure repo specific codebuild configurations via codebuild_cfg within repos:

module "dynamic_github_source" {
  source                         = "github.com/marshall7m/terraform-aws-dynamic-github-source"
  create_github_secret_ssm_param = true
  github_secret_ssm_value        = var.github_secret_ssm_value
  codebuild_name                 = "test-codebuild"
  codebuild_buildspec            = file("buildspec.yaml")
  repos = {
    "test-repo" = {
      codebuild_cfg = {
        environmentVariablesOverride = [
          {
            name  = "TEST"
            value = "FOO"
            type  = "PLAINTEXT"
          }
        ]
      }
      filter_groups = [
        [
          {
            type = "event"
            pattern = "push"
          },
          {
            type = "file_path"
            pattern = "CHANGELOG.md"
          }
        ],
        [
          {
            type = "event"
            pattern = "pull_request"
          },
          {
            type = "pr_action"
            pattern = "(opened|edited|synchronize)"
          },
          {
            type = "file_path"
            pattern = ".*\\.py$"
          },
          {
            type = "head_ref"
            pattern = "test-branch"
          }
        ]
      ]
    }
  }
}

Requirements

Name	Version
terraform	>=1.0.0
aws	>= 2.23
github	>= 4.4.0

Providers

Name	Version
aws	>= 2.23
local	n/a

Modules

Name	Source	Version
codebuild	github.com/marshall7m/terraform-aws-codebuild	v0.1.0
github_webhook_request_validator	github.com/marshall7m/terraform-aws-github-webhook	v0.1.4
lambda_trigger_codebuild	terraform-aws-modules/lambda/aws	3.3.1

Resources

Name	Type
aws_iam_policy.lambda	resource
local_file.repo_cfg	resource
aws_iam_policy_document.lambda	data source

Inputs

Name	Description	Type	Default	Required
api_description	Description for API-Gateway	string	null	no
api_name	Name of API-Gateway	string	"github-webhook"	no
codebuild_artifacts	Build project's primary output artifacts configuration see for more info: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/codebuild_project#argument-reference	object({ type = optional(string) artifact_identifier = optional(string) encryption_disabled = optional(bool) override_artifact_name = optional(bool) location = optional(string) name = optional(string) namespace_type = optional(string) packaging = optional(string) path = optional(string) })	{}	no
codebuild_assumable_role_arns	List of IAM role ARNS the Codebuild project can assume	list(string)	[]	no
codebuild_buildspec	Content of the default buildspec file	string	null	no
codebuild_cache	Cache configuration for Codebuild project	object({ type = optional(string) location = optional(string) modes = optional(list(string)) })	{}	no
codebuild_create_source_auth	Determines if a CodeBuild source credential resource should be created. Only one credential resource is needed/allowed per AWS account and region. See more at: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codebuild.GitHubSourceCredentials.html	bool	false	no
codebuild_cw_group_name	CloudWatch group name	string	null	no
codebuild_cw_stream_name	CloudWatch stream name	string	null	no
codebuild_description	CodeBuild project description	string	null	no
codebuild_environment	Codebuild environment configuration	object({ compute_type = optional(string) image = optional(string) type = optional(string) image_pull_credentials_type = optional(string) environment_variables = optional(list(object({ name = string value = string type = optional(string) }))) privileged_mode = optional(bool) certificate = optional(string) registry_credential = optional(object({ credential = optional(string) credential_provider = optional(string) })) })	{}	no
codebuild_name	Name of Codebuild project	string	"dynamic-github-source-build"	no
codebuild_role_arn	Existing IAM role ARN to attach to CodeBuild project	string	null	no
codebuild_role_policy_statements	IAM policy statements to add to CodeBuild project's role	list(object({ sid = optional(string) effect = string actions = list(string) resources = list(string) }))	[]	no
codebuild_s3_log_bucket	Name of S3 bucket where the build project's logs will be stored	string	null	no
codebuild_s3_log_encryption	Determines if encryption should be disabled for the build project's S3 logs	bool	false	no
codebuild_s3_log_key	Bucket path where the build project's logs will be stored (don't include bucket name)	string	null	no
codebuild_secondary_artifacts	Build project's secondary output artifacts configuration	object({ type = optional(string) artifact_identifier = optional(string) encryption_disabled = optional(bool) override_artifact_name = optional(bool) location = optional(string) name = optional(string) namespace_type = optional(string) packaging = optional(string) path = optional(string) })	{}	no
codebuild_source_auth_token	GitHub personal access token used to authorize CodeBuild projects to clone GitHub repos within the Terraform AWS provider's AWS account and region. If not specified, existing CodeBuild OAUTH or GitHub personal access token authorization is required beforehand.	string	null	no
codebuild_tags	Tags to attach to Codebuild project	map(string)	{}	no
codebuild_timeout	Minutes till build run is timed out	string	null	no
common_tags	Tags to add to all resources	map(string)	{}	no
enable_codebuild_cw_logs	Determines if CloudWatch logs should be enabled	bool	true	no
enable_codebuild_s3_logs	Determines if S3 logs should be enabled	bool	false	no
github_secret_ssm_description	Github secret SSM parameter description	string	"Secret value for Github Webhooks"	no
github_secret_ssm_key	SSM parameter store key for github webhook secret. Secret used within Lambda function for Github request validation.	string	"github-webhook-secret"	no
github_secret_ssm_tags	Tags for Github webhook secret SSM parameter	map(string)	{}	no
lambda_trigger_codebuild_function_name	Name of AWS Lambda function that will start the AWS CodeBuild with the override configurations	string	"dynamic-github-source-build-trigger"	no
repos	Map of keys with GitHub repo names and values representing their respective filter groups used to select what type of activities will trigger the associated Codebuild project. Params: { <repo full name (e.g user/repo)> : { filter_groups: List of filter groups that the Github activity has to meet. The activity has to meet all filters of atleast one group in order to succeed. [ [ (Filter Group) { type: The type of filter ( event - List of Github Webhook events that will invoke the API. Currently only supports: push and pull_request. pr_actions - List of pull request actions (e.g. opened, edited, reopened, closed). See more under the action key at: https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#pull_request base_refs - List of base refs head_refs - List of head refs actor_account_ids - List of Github user IDs commit_messages - List of commit messages file_paths - List of file paths ) pattern: Regex pattern that is searched for within the activity's related payload attribute. For type = event, use a single Github webhook event and not a regex pattern. exclude_matched_filter - If set to true, labels filter group as invalid if matched } ] ] codebuild_cfg: CodeBuild configurations used specifically for the repository. See AWS docs for details: https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StartBuild.html }	map(object({ filter_groups = optional(list(list(object({ type = string pattern = string exclude_matched_filter = optional(bool) })))) codebuild_cfg = optional(any) }))	{}	no

Outputs

Name	Description
api_deployment_invoke_url	API invoke URL the github webhook will ping
codebuild_arn	ARN of the CodeBuild project that will be triggered by the Lambda Function
codebuild_name	Name of the CodeBuild project that will be triggered by the Lambda Function
request_validator_agw_cw_log_group_name	Name of the CloudWatch log group associated with the API gateway
request_validator_cw_log_group_arn	ARN of the Cloudwatch log group associated with the Lambda function that validates the incoming requests
request_validator_function_arn	ARN of the Lambda function that validates incoming requests
request_validator_function_name	Name of the Cloudwatch log group associated with the Lambda function that validates the incoming requests
trigger_codebuild_cw_log_group_arn	ARN of the Cloudwatch log group associated with the Lambda function that triggers the downstream CodeBuild project
trigger_codebuild_cw_log_group_name	Name of the Cloudwatch log group associated with the Lambda function that triggers the downstream CodeBuild project
trigger_codebuild_function_arn	ARN of the Lambda function that triggers the downstream CodeBuild project with repo specific configurations
trigger_codebuild_function_name	Name of the Lambda function that triggers the downstream CodeBuild project with repo specific configurations