diff --git a/README.md b/README.md index 3f735c9..758e8cd 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,7 @@ -# quickstart-git2s3 -## Git webhooks with AWS services -### Linking your Git repository to Amazon S3 and AWS services for continuous code integration, testing, and deployment +## Git webhooks with AWS services - Quick Start -This Quick Start deploys HTTPS endpoints. AWS Lambda functions and AWS CodeBuild project for implementing webhooks, to enable event-driven integration between Git services and Amazon Web Services (AWS) on the AWS Cloud. +For architectural details, step-by-step instructions, and customization options, see the [deployment guide](https://fwd.aws/6e9Rd). -After you deploy the Quick Start, you can set up a webhook that uses the endpoints to create a bridge between your Git repository and AWS services like AWS CodePipeline and AWS CodeBuild. With this setup, builds and pipeline executions occur automatically when you commit your code to a Git repository, and your code can be continuously integrated, tested, built, and deployed on AWS with each change. +To post feedback, submit feature ideas, or report bugs, use the **Issues** section of this GitHub repo. -The Quick Start includes an AWS CloudFormation template that automates the deployment. You can also use the AWS CloudFormation template as a starting point for your own implementation. - - - -![Quick Start architecture for implementing webhooks on AWS](./docs/images/image3.png) - -For implementation details, deployment instructions, and customization options, see the [deployment guide](https://fwd.aws/QQBRr). - -To post feedback, submit feature ideas, or report bugs, use the **Issues** section of this GitHub repo. -If you'd like to submit code for this Quick Start, please review the [AWS Quick Start Contributor's Kit](https://aws-quickstart.github.io/). +To submit code for this Quick Start, see the [AWS Quick Start Contributor's Kit](https://aws-quickstart.github.io/). \ No newline at end of file diff --git a/docs/boilerplate b/docs/boilerplate index 6dad7ef..da2f08d 160000 --- a/docs/boilerplate +++ b/docs/boilerplate @@ -1 +1 @@ -Subproject commit 6dad7efceb9734dadee99b88ea55b95aeb8afc18 +Subproject commit da2f08d7a202c14815e67ad5bddb3c6c01c4bbbb diff --git a/docs/generated/parameters/git2s3.template.adoc b/docs/generated/parameters/git2s3.template.adoc new file mode 100644 index 0000000..8c0399d --- /dev/null +++ b/docs/generated/parameters/git2s3.template.adoc @@ -0,0 +1,33 @@ + +.General settings +[width="100%",cols="16%,11%,73%",options="header",] +|=== +|Parameter label (name) |Default value|Description|Output S3 bucket name +(`OutputBucketName`)|`**__Blank string__**`|(Optional) Name for the S3 bucket where the Git repository .zip file is stored. If left blank, the Quick Start creates one for you.|Custom domain name +(`CustomDomainName`)|`**__Blank string__**`|Domain name for the webhook endpoint. If left blank, API Gateway creates a domain name for you. +|=== +.Git pull settings +[width="100%",cols="16%,11%,73%",options="header",] +|=== +|Parameter label (name) |Default value|Description|API secret +(`ApiSecret`)|`**__Blank string__**`|API secret used to authenticate access to webhooks in GitHub Enterprise, GitLab, and other Git services. If a webhook payload header contains a matching secret, IP address authentication is bypassed. API secrets cannot contain commas (,), backward slashes (\), or quotes (").|Allowed IP addresses +(`AllowedIps`)|`18.205.93.0/25,18.234.32.128/25,13.52.5.0/25`|Comma-separated list of allowed IP CIDR blocks. The default addresses listed are BitBucket Cloud IP ranges.|Exclude .git directory +(`ExcludeGit`)|`True`|Choose False to omit the .git directory from the Git repository .zip file. +|=== +.AWS Quick Start configuration +[width="100%",cols="16%,11%,73%",options="header",] +|=== +|Parameter label (name) |Default value|Description|Quick Start S3 bucket name +(`QSS3BucketName`)|`aws-quickstart`|S3 bucket name for Quick Start assets. It can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).|Quick Start S3 bucket Region +(`QSS3BucketRegion`)|`us-east-1`|AWS Region where the Quick Start assets S3 bucket (QSS3BucketName) is hosted. Required when using your own S3 bucket.|Quick Start S3 key prefix +(`QSS3KeyPrefix`)|`quickstart-git2s3/`|Key prefix for the Quick Start assets S3 bucket. A key prefix is similar to a directory name that enables you to store similar data under the same directory in an S3 bucket. It can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). +|=== +.VPC configuration +[width="100%",cols="16%,11%,73%",options="header",] +|=== +|Parameter label (name) |Default value|Description|VPC ID +(`VPCId`)|`**__Blank string__**`|ID of the VPC in which the Lambda function runs.|VPC CIDR +(`VPCCidrRange`)|`**__Blank string__**`|CIDR range of the VPC.|Subnet IDs +(`SubnetIds`)|`**__Blank string__**`|SubnetIDs in which the Lambda function runs.|Hostname override +(`ScmHostnameOverride`)|`**__Blank string__**`|Name to override the hostname in the header of a webhook JSON payload. +|=== \ No newline at end of file diff --git a/docs/generated/parameters/index.adoc b/docs/generated/parameters/index.adoc index ff7bd09..fc1c39f 100644 --- a/docs/generated/parameters/index.adoc +++ b/docs/generated/parameters/index.adoc @@ -1 +1,3 @@ -// placeholder + +=== Parameters for deploying into your selected Region. +include::git2s3.template.adoc[] diff --git a/docs/images/image8.png b/docs/images/S3_new_object.png similarity index 100% rename from docs/images/image8.png rename to docs/images/S3_new_object.png diff --git a/docs/images/architecture_diagram.png b/docs/images/architecture_diagram.png index ad91da3..459938a 100644 Binary files a/docs/images/architecture_diagram.png and b/docs/images/architecture_diagram.png differ diff --git a/docs/images/image7.png b/docs/images/commit_push.png similarity index 100% rename from docs/images/image7.png rename to docs/images/commit_push.png diff --git a/docs/images/image5.png b/docs/images/outputs_tab.png similarity index 100% rename from docs/images/image5.png rename to docs/images/outputs_tab.png diff --git a/docs/images/image6.png b/docs/images/testfile.png similarity index 100% rename from docs/images/image6.png rename to docs/images/testfile.png diff --git a/docs/images/image2.png b/docs/images/using_webhooks.png similarity index 100% rename from docs/images/image2.png rename to docs/images/using_webhooks.png diff --git a/docs/partner_editable/_settings.adoc b/docs/partner_editable/_settings.adoc index b6aae21..241186a 100644 --- a/docs/partner_editable/_settings.adoc +++ b/docs/partner_editable/_settings.adoc @@ -1,20 +1,20 @@ // Do not change this first attribute. Do change the others. :quickstart-team-name: AWS Quick Start team :quickstart-project-name: quickstart-git2s3 -:partner-product-name: Git Webhooks +:partner-product-name: Git webhooks // For the following attribute, if you have no short name, enter the same name as partner-product-name. -:partner-product-short-name: Git Webhooks +:partner-product-short-name: Git webhooks // If there's no partner, comment partner-company-name and partner-contributors. // :partner-company-name: Example Company Name, Ltd. -:doc-month: October -:doc-year: 2020 +:doc-month: January +:doc-year: 2021 // For the following two "contributor" attributes, if the partner agrees to include names in the byline, // enter names for both partner-contributors and quickstart-contributors. // Otherwise, delete all placeholder names: everything preceding "{partner-company-name}" // and "{quickstart-team-name}". Use commas as shown in the placeholder text. // Use the comma before "and" only when three or more names. // :partner-contributors: Shuai Ye, Michael McConnell, and John Smith, {partner-company-name} -:quickstart-contributors: Kirankumar Chandrashekar and Jay McConnell, Amazon Web Services +:quickstart-contributors: Kirankumar Chandrashekar and Jay McConnell, AWS Quick Start team // For deployment_time, use minutes if deployment takes an hour or less, // for example, 30 minutes or 60 minutes. // Use hours for deployment times greater than 60 minutes (rounded to a quarter hour), @@ -28,3 +28,4 @@ :disable_licenses: :disable_regions: :disable_requirements: +:parameters_as_appendix: diff --git a/docs/partner_editable/additional_info.adoc b/docs/partner_editable/additional_info.adoc index 33f0a5d..bbe8b8b 100644 --- a/docs/partner_editable/additional_info.adoc +++ b/docs/partner_editable/additional_info.adoc @@ -1,47 +1,72 @@ -==== Adding an API Secret After Deployment +=== Configuring Git services -In some cases, your Git service may provide security mechanisms like API secrets when you create the webhook. In these cases, you can launch the Quick Start with a blank parameter value for the *API Secret* parameter, and then update the stack to provide the value of the parameter. Follow these steps: +After deploying the Quick Start, set up a webhook in your Git repository. -. In the https://console.aws.amazon.com/cloudformation[AWS Cloudformation console], select the stack. +To configure a webhook, you need *GitPullWebHookApi* and *PublicSSHKey*. You can find these on the *Outputs* tab on the AWS CloudFormation console after deploying the Quick Start. -[arabic, start=2] -. Choose *Actions*, and then choose *Update Stack*. -. Keep the default to use the current template. -. On the *Specify Details* page, change the *API Secret* parameter setting. -. Choose *Next* twice. +[#outputs_tab] +.Outputs tab on the AWS CloudFormation console +[link=images/outputs_tab.png] +image::../images/outputs_tab.png[outputs_tab,width=701,height=222] -[arabic, start=10] -. Under *Capabilities*, select the check box to acknowledge that the template will create IAM resources, and then choose *Update*. +* *GitPullWebHookApi* is the URL endpoint that receives the HTTP POST request from the Git service. +* *PublicSSHKey* is the public SSH key used to connect to your Git repository. This key can be configured as a read-only machine user or as a deployment key in your Git repository. -When the status is *UPDATE_COMPLETE*, the stack has been updated with the webhook secret you specified for security. +The instructions for setting up webhooks and deployment keys vary by Git service. For more information, see your Git service documentation. -== Test the deployment +=== Configuring AWS services -Before putting the webhook into production, you should test your deployment. +After deploying the Quick Start, configure the AWS services in your workload to use the Git repository S3 bucket as a source. -. Modify a file in your repository. +As shown in <>, the *Outputs* tab in the AWS CloudFormation console includes *OutputBucketName*. This output is an Amazon S3 key that forms the base of the path to the .zip file of your repository code. The S3 key has the following format: -image:../images/image6.png[../Desktop/Screen_Shot_2017-07-15_at_11_35_57_AM.png,width=529,height=21] +``` +S3://output-bucket-name/git-user/git-repository/git-user_git-repository.zip +``` +Here, `git-user` is the owner or path prefix of the repository. In some Git services, this may be an organization name. However, some Git services do not return a Git user or organization for a repository. In these cases, you can omit the `git-user` parts of the path. -. Commit and push the changes. +The instructions vary for linking an AWS service to an Amazon S3 object. For links to AWS service documentation, see link:#_aws_services[AWS services], later in this guide. + +== Adding an API secret after deployment + +You can launch this Quick Start without an *API Secret* parameter. If your Git service provides an API secret when you create a webhook, you can update the stack with the API secret later. + +To update the stack with an API secret, do the following: -image:../images/image7.png[Figure 4,width=648,height=198] +. In the https://console.aws.amazon.com/cloudformation[AWS CloudFormation console], select the stack you want to update. +. In the stack details pane, choose *Update*. +. Choose *Use current template*. +. On the *Specify stack details* page, change the *API Secret* parameter setting, then choose *Next*. +. On the *Configure stack options* page, choose *Next*. +. Select *I acknowledge that this template may create IAM resources*. +. Choose *Update stack*. When the status is *UPDATE_COMPLETE*, the stack is updated with the API secret. -[start=2] -. Wait a few minutes and check your S3 bucket for a new (or updated) object with a key that matches your repository path. +== Test the deployment + +Before putting a webhook into production, test your deployment by doing the following: -image:../images/image8.png[Figure 5,width=646,height=348] +. Modify a file in your repository. +. Commit and push the changes. +. Wait a few minutes, and then check the Git repository S3 bucket for a new (or updated) object with a key that matches your repository path. -*Figure 4: Checking for S3 bucket updates after a commit* +:xrefstyle: short +[#S3_new_object] +.Checking for a new or updated object in your S3 bucket after a commit +[link=images/S3_new_object.png] +image::../images/S3_new_object.png[S3_new_object,width=646,height=348] -=== Best Practices +== Best practices The architecture built by this Quick Start supports AWS best practices for security. -==== SSH Keys +=== SSH keys + +This Quick Start deploys a private SSH key pair that is encrypted with an AWS KMS key and uploaded to Amazon S3. AWS CodeBuild decrypts the private SSH key and uses it to authenticate your Git service before cloning the repository. + +We don’t recommend sharing SSH keys among multiple services, or launching another instance of this Quick Start to clone and store another repository in Amazon S3. Each repository should use unique SSH keys. -SSH keys are generated at stack creation, and are then encrypted using AWS KMS and the encrypted copy stored in Amazon S3. When you use the Git pull endpoint, the private key is fetched by the Lambda function, decrypted, and used to authenticate against your Git service to perform a clone of the repository. We don’t recommend (a) reusing SSH keys for multiple services, or (b) launching another instance of this Quick Start for each repository that you wish to clone to Amazon S3; this ensures that each repository uses unique keys. +=== Webhook security -==== Webhook Security +Git services provide different ways to authenticate an endpoint, such as webhook secrets, source-IP-address allow listing, personal access tokens, and OAuth2. We recommend that you set up at least one of these security mechanisms to protect your webhook API endpoint. -Different Git services provide varying ways to authenticate against an endpoint. The Git pull endpoint supports webhook secrets (used by GitHub Enterprise, GitLab, and other Git repository managers) as well as source IP address whitelisting. The zip download endpoint supports personal access tokens (as used by GitHub Enterprise and GitLab) and OAuth2 (used by Bitbucket). We recommend that you set up at least one of these security mechanisms to protect your webhook API endpoint. For more information about how this Quick Start utilizes these mechanisms, see the parameters in the link:#deployment-steps[Deployment Steps] section of this guide. For product-specific guidance on how to configure these security mechanisms, refer to your Git product’s documentation. +For more information about how this Quick Start uses endpoint security mechanisms, see the link:#_parameter_reference[Parameter reference] section of this guide. For specific guidance on how to configure security mechanisms for your Git service, refer to your Git service documentation. diff --git a/docs/partner_editable/architecture.adoc b/docs/partner_editable/architecture.adoc index 3e55d5b..2347538 100644 --- a/docs/partner_editable/architecture.adoc +++ b/docs/partner_editable/architecture.adoc @@ -1,27 +1,24 @@ Deploying this Quick Start builds the following environment in the AWS Cloud. // Replace this example diagram with your own. Send us your source PowerPoint file. Be sure to follow our guidelines here : http://(we should include these points on our contributors giude) + :xrefstyle: short -[#architecture1] -// .Quick Start architecture for {partner-product-short-name} on AWS +[#architecture_diagram] +.Git webhooks with AWS services Quick Start architecture [link=images/architecture_diagram.png] -image::../images/image3.png[Architecture,width=100%,height=100%] - - -*Figure 2: Webhook endpoint architecture on AWS* - - -The Quick Start deployment sets up a serverless AWS Cloud environment that includes the following components. - -* An API Gateway endpoint to accept the webhook requests from Git. -* Lambda function that processes the input git webhook payload received by the API Gateway and submits a build on AWS CodeBuild Project. -* An AWS CodeBuild project to connect to the Git service, either over SSH or through the Git service’s endpoint. This AWS CodeBuild project will zip the code and upload it to Amazon S3. - -*Important* The AWS CodeBuild project that is deployed by this Quick Start must be able to communicate with your Git repository. For example, you can use a SaaS-based Git service that the AWS CodeBuild can reach through the internet. - -* An AWS KMS key to encrypt the SSH private key used to connect to the repository over SSH. -* Two S3 buckets: One bucket stores the zipped contents of your Git repository, and the second bucket stores the AWS KMS-encrypted SSH private keys that are generated during stack creation. Note that the first bucket has versioning enabled, and all previous versions are retained indefinitely. If you’d like to manage the retention period for old versions, follow the instructions in the http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html[Amazon S3 documentation]. -* Several IAM roles required for the Lambda functions and API Gateway. The inline permissions attached to these roles are scoped using the http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege[least privilege] model. -* Two Lambda-backed AWS CloudFormation custom resources. One resource generates an SSH keypair, encrypts it using AWS KMS, and stores it in Amazon S3. The second resource deletes the content of the two S3 buckets on stack deletion. - -NOTE: If you need backups, make sure to copy the S3 bucket contents before you delete the CloudFormation stack. \ No newline at end of file +image::../images/architecture_diagram.png[architecture_diagram,width=100%,height=100%] + +As shown in <>, this Quick Start sets up a serverless AWS Cloud environment that includes the following components: + +* Amazon API Gateway to receive Git webhook requests and forward them to AWS Lambda. +* An AWS Lambda function to process Git webhook requests from API Gateway and invoke an AWS CodeBuild project. +* An AWS CodeBuild project to connect to your Git service, then retrieve, zip, and upload the latest version of your Git repository to Amazon S3. +* An AWS Key Management Service (AWS KMS) key to encrypt/decrypt the SSH (Secure Shell) keys used by AWS CodeBuild to connect to your Git repository using SSH. The SSH key pair is generated by a Lambda-backed AWS CloudFormation custom resource when the stack is deployed. +* Two Amazon S3 buckets: one for Git repository contents, and another for encrypted SSH keys. A Lambda-backed AWS CloudFormation custom resource deletes the contents of the S3 buckets when you delete the CloudFormation stack. If you need backups, copy the S3 buckets before deleting the stack. + +[NOTE] +======== +* The Quick Start deploys AWS Identity and Access Management (IAM) roles required by Lambda and API Gateway. The inline permissions attached to the roles are scoped using the http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege[least privilege] model. +* The AWS CodeBuild project must be able to communicate with your Git repository. For example, you can employ a SaaS-based Git service like GitHub to which CodeBuild can connect over the internet. +* The Git repository S3 bucket this Quick Start deploys has versioning enabled, and all previous versions are retained indefinitely. To modify the retention period, see http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html[How do I create a lifecycle rule for an S3 bucket?] +======== diff --git a/docs/partner_editable/deploy_steps.adoc b/docs/partner_editable/deploy_steps.adoc index 72efed0..13a16f7 100644 --- a/docs/partner_editable/deploy_steps.adoc +++ b/docs/partner_editable/deploy_steps.adoc @@ -8,49 +8,10 @@ . https://fwd.aws/mDMrd[Launch the AWS CloudFormation template] into your AWS account. -You can also https://fwd.aws/wr8Gg[download the template] to use it as a starting point for your own implementation. - -The stack takes approximately 15 minutes to create. - -NOTE: You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional cost for using this Quick Start. Prices are subject to change. See the pricing pages for each AWS service you will be using in this Quick Start for full details. +NOTE: You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional cost for using this Quick Start. Prices are subject to change. See the pricing pages for each AWS service you use in this Quick Start for full details. +:xrefstyle: short [start=2] -. Check the region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This is where the network infrastructure will be built. The template is launched in the US East (Ohio) Region by default. +. Check the Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This Region is where the Quick Start infrastructure is built. The template for this Quick Start is launched in the US East (Ohio) Region by default. . On the *Select Template* page, keep the default setting for the template URL, and then choose *Next*. -. On the *Specify Details* page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose *Next*. -. On the *Options* page, you can https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html[specify tags] (key-value pairs) for resources in your stack and https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html[set advanced options]. When you’re done, choose *Next*. -. On the *Review* page, review and confirm the template settings. Under *Capabilities*, select the check box to acknowledge that the template will create IAM resources. -. Choose *Create* to deploy the stack. -. Monitor the status of the stack. When the status is *CREATE_COMPLETE*, the webhook resources are ready. -. The *Outputs* tab for the stack contain the two webhook endpoint URLs, the output bucket name, and the public SSH key, as illustrated in Figure 4. - -=== Configure Your Git Repository - -After you have successfully deployed the Quick Start, you can configure the service that will use the S3 object as a source. Figure 4 shows the *Outputs* tab in the AWS CloudFormation console, which displays the outputs for configuring your Git webhook. - -image:../images/image5.png[Figure 4,width=701,height=222] - - -*Figure 3: Outputs tab after deployment* - - -* *GitPullWebHookApi* is the webhook endpoint to use if you opt for the Git pull method described in the link:#webhook-endpoints[Webhook Endpoints] section of this guide. -* *PublicSSHKey* is the public SSH key that you use to connect to your repository if you’re using the Git pull endpoint. This key can be configured as a read-only machine user or as a deployment key in your Git service. - -The exact process to set up webhooks differs from service to service. For step-by-step instructions, consult your Git service’s documentation. - -=== Configure an AWS Service to Connect to the S3 Object - -After you have successfully deployed the Quick Start, you can configure the desired service to use the S3 object as a source. As previously illustrated in Figure 4, the *Outputs* tab in the AWS CloudFormation console includes the *OutputBucketName* output. This output is an S3 key that forms the base of the path to your code zip file. The S3 key is in this format: - -S3://_output-bucket-name_/_git-user_/_git-repository_/_git-user_git-repository_.zip - -where: - -* _git-user_ is the owner or path prefix of the repository. In some Git services, this may be an organization name. -* Some Git services do not return a Git user or organization for a repository. In these cases, you can omit the _git-user_ parts of the path. - -The exact process for linking an AWS service differs from service to service. For step-by-step instructions, consult the service documentation. For easy reference, here are links for the two typical services: - -* http://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-s3.html[AWS CodePipeline – Simple pipeline tutorial] -* http://docs.aws.amazon.com/codebuild/latest/userguide/getting-started.html[AWS CodeBuild – Walkthrough for creating deployable source code] +. On the *Specify Details* page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. For details on each parameter, see the link:#_parameter_reference[Parameter reference] section of this guide. After reviewing and customizing the parameters, choose *Next*. \ No newline at end of file diff --git a/docs/partner_editable/faq_troubleshooting.adoc b/docs/partner_editable/faq_troubleshooting.adoc index dacb8ae..7330d30 100644 --- a/docs/partner_editable/faq_troubleshooting.adoc +++ b/docs/partner_editable/faq_troubleshooting.adoc @@ -4,45 +4,40 @@ *Q.* I encountered a *CREATE_FAILED* error when I launched the Quick Start. -*A.* If AWS CloudFormation fails to create the stack, we recommend that you relaunch the template with *Rollback on failure* set to *Disabled*. (This setting is under *Advanced* in the AWS CloudFormation console, *Options* page.) With this setting, the stack’s state is retained and the instance is left running, so you can troubleshoot the issue.) +*A.* If AWS CloudFormation fails to create the stack, we recommend that you relaunch the template with *Rollback on failure* set to *Disabled*. (This setting is under *Advanced* on the *Options* page of the AWS CloudFormation console.) With this setting, the stack’s state is retained and the instance remains running so you can troubleshoot the issue. -WARNING: When you set *Rollback on failure* to *Disabled*, you continue to incur AWS charges for this stack. Please make sure to delete the stack when you finish troubleshooting. +WARNING: When you set *Rollback on failure* to *Disabled*, you continue to incur AWS charges for this stack. Ensure that you delete the stack after troubleshooting. -For additional information, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html[Troubleshooting AWS CloudFormation^] on the AWS website. +For more information, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html[Troubleshooting AWS CloudFormation^]. == Troubleshooting -If your commits are not being pushed through to Amazon S3, check the following: +If commits to your repository do not show up in Amazon S3, do the following: -* In your Git service’s webhooks configuration, check that your configured security parameters and the endpoint are correct. Consult the Git service documentation for detailed guidance on configuration. -* Check the Lambda logs for errors. These are stored in Amazon CloudWatch Logs. To access the logs, open the endpoint’s Lambda function in the AWS console, navigate to the *Monitoring* tab, and then choose *View logs in CloudWatch*. +* Check the security parameters and endpoint in your Git webhook configuration. See link:#_configuring_git_services[Configuring Git services] earlier in this guide and consult your Git service documentation for help with configuring webhooks. +* Check the AWS Lambda logs for errors. These are stored in Amazon CloudWatch Logs. For help with accessing them, see https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html[Accessing Amazon CloudWatch logs for AWS Lambda]. +* Check the AWS CodeBuild project logs for errors. To access them, do the following: +. Open the AWS CodeBuild console. +. On the *Build history* page, choose the *Build run* link for the project. +. On the *Build status* page, see the *Build logs* tab. -For additional information, see https://docs.aws.amazon.com/lambda/latest/dg/lambda-troubleshooting.html[Troubleshooting issues in AWS Lambda^] on the AWS website. - -* Check the AWS CodeBuild project logs for errors. These are stored in Amazon CloudWatch Logs. To access the logs, open the endpoint’s CodeBuild Build Project in the AWS console, select a *Build run* and click on the hyperlink and navigate to the *Build logs* tab to see the build execution logs. - -For additional information, see https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html[Troubleshooting AWS CodeBuild^] on the AWS website. - -== Additional Resources +== Additional resources === AWS services -*AWS CloudFormation* https://aws.amazon.com/documentation/cloudformation/ - -*AWS Lambda* https://aws.amazon.com/documentation/lambda/ - -*Amazon API Gateway* https://aws.amazon.com/documentation/apigateway/ - -*Amazon S3* https://aws.amazon.com/documentation/s3/ - -*AWS CodePipeline* https://aws.amazon.com/documentation/codepipeline/ - -*AWS CodeBuild* https://aws.amazon.com/documentation/codebuild/ +* https://aws.amazon.com/documentation/cloudformation/[AWS CloudFormation] +* https://aws.amazon.com/documentation/lambda/[AWS Lambda] +** https://docs.aws.amazon.com/lambda/latest/dg/lambda-troubleshooting.html[Troubleshooting issues in AWS Lambda^] +* https://aws.amazon.com/documentation/apigateway/[Amazon API Gateway] +* https://aws.amazon.com/documentation/s3/[Amazon S3] +* https://aws.amazon.com/documentation/codepipeline/[AWS CodePipeline] +** http://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-s3.html[Tutorial: Create a simple pipeline (S3 bucket)] +* https://aws.amazon.com/documentation/codebuild/[AWS CodeBuild] +** https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html[Troubleshooting AWS CodeBuild^] +** http://docs.aws.amazon.com/codebuild/latest/userguide/getting-started.html[Getting started with AWS CodeBuild using the console] === Webhooks -*GitHub Developer Repository Webhooks API* https://developer.github.com/v3/repos/hooks/ - -*Atlassian Bitbucket Webhooks documentation* https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html - -*GitLab Webhooks documentation* https://docs.gitlab.com/ce/user/project/integrations/webhooks.html +* https://developer.github.com/v3/repos/hooks/[GitHub Docs: Webhooks] +* https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html[BitBucket Support: Manage webhooks] +* https://docs.gitlab.com/ce/user/project/integrations/webhooks.html[GitLab Docs: Webhooks] \ No newline at end of file diff --git a/docs/partner_editable/licenses.adoc b/docs/partner_editable/licenses.adoc index f88ecc0..5966250 100644 --- a/docs/partner_editable/licenses.adoc +++ b/docs/partner_editable/licenses.adoc @@ -1 +1,5 @@ -The Quick Start provides an Amazon API Gateway endpoint, several Lambda functions and an AWS CodeBuild project to handle the download, zipping, and deployment of code to Amazon S3. AWS CodePipeline carries a cost for each active pipeline; see https://aws.amazon.com/codepipeline/pricing/[AWS CodePipeline pricing]. Depending on your configuration, the Quick Start may deploy an AWS Key Management Service (AWS KMS) key; for pricing, see https://aws.amazon.com/kms/pricing/[AWS Key Management Service pricing]. API Gateway, Amazon S3, Lambda, and AWS CodeBuild costs vary depending on how often you commit code to your repository; each commit triggers a request to the Lambda execution in API Gateway; for details, see the pricing pages for https://aws.amazon.com/api-gateway/pricing/[API Gateway], https://aws.amazon.com/s3/pricing/[Amazon S3], https://aws.amazon.com/lambda/pricing/[Lambda] and https://aws.amazon.com/codebuild/pricing/[codebuild] . +The Quick Start provides an Amazon API Gateway endpoint, Lambda functions, and an AWS CodeBuild project to access, compress, and upload code to Amazon S3. AWS CodePipeline carries a cost for each active pipeline. (See https://aws.amazon.com/codepipeline/pricing/[AWS CodePipeline pricing].) + +Depending on your configuration, the Quick Start may deploy an AWS Key Management Service (AWS KMS) key which incurs a monthly cost for key storage and usage. (See https://aws.amazon.com/kms/pricing/[AWS Key Management Service pricing].) + +API Gateway, Amazon S3, Lambda, and AWS CodeBuild costs vary depending on how often you commit code to the connected Git repository. (See https://aws.amazon.com/api-gateway/pricing/[Amazon API Gateway pricing], https://aws.amazon.com/s3/pricing/[Amazon S3 pricing], https://aws.amazon.com/lambda/pricing/[AWS Lambda pricing] and https://aws.amazon.com/codebuild/pricing/[AWS CodeBuild pricing].) diff --git a/docs/partner_editable/overview_target_and_usage.adoc b/docs/partner_editable/overview_target_and_usage.adoc index 3a97fda..7f6ed60 100644 --- a/docs/partner_editable/overview_target_and_usage.adoc +++ b/docs/partner_editable/overview_target_and_usage.adoc @@ -1 +1,4 @@ -This Quick Start implements HTTPS endpoints and code that helps you link AWS services with webhooks. Git webhooks enable event-driven integration between Git services and external applications. This Quick Start uses this integration to receive an event when commits are pushed to a Git repository. +This Quick Start deploys the HTTPS endpoint you can use to configure a webhook to link your Git and AWS services. With a webhook in place, each time a Git user pushes a commit, your repository is automatically retrieved, zipped, and uploaded to an Amazon Simple Storage Service (Amazon S3) bucket. You can then configure AWS services such as AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy to use the S3 bucket as a source. + +This guide describes the components that are deployed by the Quick Start, and contains links to launch the AWS CloudFormation template that automates the deployment. + diff --git a/docs/partner_editable/product_description.adoc b/docs/partner_editable/product_description.adoc index c88dfc5..f8e30cc 100644 --- a/docs/partner_editable/product_description.adoc +++ b/docs/partner_editable/product_description.adoc @@ -1,10 +1,3 @@ +After deploying the Quick Start, you use the link:#_configuring_git_services[endpoint information] it provides to configure a webhook in your Git service. A webhook sends an HTTPS POST request to the endpoint in response to a push action. The HTTPS POST request contains JavaScript Object Notation (JSON) data about the push and repository. After the request is accepted by Amazon API Gateway, it is passed to an AWS Lambda function that triggers an AWS CodeBuild project. The AWS CodeBuild project uses the information in the HTTPS POST request to retrieve the latest version of your repository. -As Figure 1 illustrates, when code is pushed to your repository, the Git service sends an HTTPS POST to the endpoints configured by the Quick Start. The POST contains JSON data about the push operation, including the repository details that the Quick Start uses to fetch the latest version of your code. - -image:../images/image2.png[image,width=647,height=266] - -*Figure 1: Using webhooks for code commits* - -This Quick Start implements the required code to trigger a Lambda function and an AWS CodeBuild project working together that zips up the code in your repository and places the .zip file in Amazon S3. When this function is triggered by a Git webhook, it provides a convenient way to bridge Git services with AWS services like AWS CodePipeline and AWS CodeBuild, so that builds and pipeline executions occur automatically when you commit your code to a Git repository. Linking your existing code repositories to the AWS Cloud in this way enables your code to be continuously integrated, tested, built, and deployed on the AWS Cloud with each change. - -NOTE: The AWS CodeBuild Project deployed by this Quick Start must be able to communicate with your Git repository. For example, you can use a SaaS-based Git service that AWS CodeBuild can reach through the internet. +For more information about the components that this Quick Start deploys, see the link:#_architecture[Architecture] section later in this guide. \ No newline at end of file diff --git a/templates/git2s3.template.yaml b/templates/git2s3.template.yaml index baeb0f6..e7e7212 100644 --- a/templates/git2s3.template.yaml +++ b/templates/git2s3.template.yaml @@ -1,109 +1,113 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: Git Webhooks to clone repository contents to S3. For integrating 3rd - party git products with AWS Services like CodePipeline/CodeBuild (qs-1nfhrd9bh) +Description: Git webhooks to clone and store a Git repository in S3. Used to integrate Git services + with AWS services like AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. (qs-1nfhrd9bh) Metadata: QuickStartDocumentation: - EntrypointName: "Launch Quick Start" + EntrypointName: Parameters for deploying into your selected Region AWS::CloudFormation::Interface: ParameterGroups: - Label: - default: General Settings + default: General settings Parameters: - OutputBucketName - CustomDomainName - Label: - default: Git Pull Settings + default: Git pull settings Parameters: - ApiSecret - AllowedIps - ExcludeGit - Label: - default: AWS Quick Start Configuration + default: AWS Quick Start configuration Parameters: - QSS3BucketName - QSS3BucketRegion - QSS3KeyPrefix + - Label: + default: VPC configuration + Parameters: + - VPCId + - VPCCidrRange + - SubnetIds + - ScmHostnameOverride ParameterLabels: AllowedIps: - default: Allowed IPs + default: Allowed IP addresses ApiSecret: - default: API Secret + default: API secret CustomDomainName: - default: Custom Domain Name + default: Custom domain name OutputBucketName: - default: Output S3 Bucket Name + default: Output S3 bucket name QSS3BucketName: - default: Quick Start S3 Bucket Name + default: Quick Start S3 bucket name QSS3BucketRegion: - default: Quick Start S3 bucket region + default: Quick Start S3 bucket Region QSS3KeyPrefix: - default: Quick Start S3 Key Prefix + default: Quick Start S3 key prefix + VPCId: + default: VPC ID + VPCCidrRange: + default: VPC CIDR + SubnetIds: + default: Subnet IDs + ScmHostnameOverride: + default: Hostname override ExcludeGit: - default: Exclude Git + default: Exclude .git directory Parameters: AllowedIps: - Description: gitpull method only. Comma seperated list of IP CIDR blocks for source - IP authentication. The BitBucket Cloud IP ranges are provided as defaults. + Description: Comma-separated list of allowed IP CIDR blocks. The default addresses listed are BitBucket Cloud IP ranges. Type: String Default: 18.205.93.0/25,18.234.32.128/25,13.52.5.0/25 ApiSecret: - Description: 'gitpull method only. WebHook Secrets for use with GitHub Enterprise - and GitLab. If a secret is matched IP range authentication is bypassed. Cannot - contain: , \ "' + Description: API secret used to authenticate access to webhooks in GitHub Enterprise, GitLab, and other Git services. If a webhook payload header contains a matching secret, IP address authentication is bypassed. API secrets cannot contain commas (,), backward slashes (\), or quotes ("). Type: String Default: '' NoEcho: 'true' CustomDomainName: - Description: Use a custom domain name for the webhook endpoint, if left blank - API Gateway will create a domain name for you + Description: Domain name for the webhook endpoint. If left blank, API Gateway creates a domain name for you. Type: String Default: '' OutputBucketName: - Description: 'OPTIONAL: Bucket Name where the zip file output should be placed, - if left blank a bucket name will be automatically generated.' + Description: (Optional) Name for the S3 bucket where the Git repository .zip file is stored. If left blank, the Quick Start creates one for you. Type: String Default: '' QSS3BucketName: AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$ - ConstraintDescription: Quick Start bucket name can include numbers, lowercase - letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen + ConstraintDescription: Quick Start S3 bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Default: aws-quickstart - Description: S3 bucket name for the Quick Start assets. Quick Start bucket name - can include numbers, lowercase letters, uppercase letters, and hyphens (-). - It cannot start or end with a hyphen (-). + Description: S3 bucket name for Quick Start assets. It can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Type: String QSS3BucketRegion: Default: 'us-east-1' - Description: 'The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. When using your own bucket, you must specify this value.' + Description: AWS Region where the Quick Start assets S3 bucket (QSS3BucketName) is hosted. Required when using your own S3 bucket. Type: String QSS3KeyPrefix: AllowedPattern: ^[0-9a-zA-Z-/]*$ - ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters, - uppercase letters, hyphens (-), and forward slash (/). + ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). Default: quickstart-git2s3/ - Description: S3 key prefix for the Quick Start assets. Quick Start key prefix - can include numbers, lowercase letters, uppercase letters, hyphens (-), and - forward slash (/). + Description: Key prefix for the Quick Start assets S3 bucket. A key prefix is similar to a directory name that enables you to store similar data under the same directory in an S3 bucket. It can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). Type: String VPCId: - Description: Id of the VPC the DL Zip lambda should run in + Description: ID of the VPC in which the Lambda function runs. Type: String Default: '' VPCCidrRange: - Description: CIDR range of the VPC + Description: CIDR range of the VPC. Type: String Default: '' SubnetIds: - Description: SubnetIds the DL Zip lambda should run in + Description: SubnetIDs in which the Lambda function runs. Type: CommaDelimitedList Default: '' ScmHostnameOverride: - Description: Hostname to override the incoming json request with + Description: Name to override the hostname in the header of a webhook JSON payload. Type: String Default: '' ExcludeGit: - Description: Choose True to include the .git directory in the zip file or false to not include + Description: Choose False to omit the .git directory from the Git repository .zip file. Type: String Default: 'True' AllowedValues: ['True', 'False'] @@ -188,7 +192,7 @@ Resources: CopyZipsFunction: Type: AWS::Lambda::Function Properties: - Description: Copies objects from a source S3 bucket to a destination + Description: Copies objects from a source S3 bucket to a destination. Handler: index.handler Runtime: python3.7 Role: !GetAtt 'CopyZipsRole.Arn' @@ -272,8 +276,7 @@ Resources: KMSKey: Type: AWS::KMS::Key Properties: - Description: git CodePipeline integration, used to encrypt/decrypt ssh keys - stored in S3 + Description: AWS KWS key to encrypt and decrypt SSH keys stored in S3. KeyPolicy: Version: '2012-10-17' Statement: @@ -495,7 +498,7 @@ Resources: CodeBuildBasePolicy: Type: 'AWS::IAM::ManagedPolicy' Properties: - Description: Policy with base permissions for CodeBuild + Description: Policy with base permissions for CodeBuild. Path: / Roles: - !Ref CodeBuildServiceRole @@ -537,7 +540,7 @@ Resources: CodeBuildEndpointPolicy: Type: 'AWS::IAM::ManagedPolicy' Properties: - Description: Policy with permissions for codebuild to work with endpoints + Description: Policy with permissions enabling CodeBuild to work with endpoints. Path: / PolicyDocument: Version: "2012-10-17" @@ -718,7 +721,7 @@ Resources: Condition: ShouldRunInVPC Type: AWS::EC2::SecurityGroup Properties: - GroupDescription: Security Group to allow the lambda to access the git service + GroupDescription: Security Group to allow the Lambda function to access the Git service. SecurityGroupEgress: - CidrIp: !Ref 'VPCCidrRange' FromPort: 0