Skip to content
This repository was archived by the owner on Oct 4, 2024. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f4d3e8c
edits 120120
censullo Dec 2, 2020
f8e2e8d
edits part 1 120220
censullo Dec 2, 2020
cba8788
edits part 2 120220
censullo Dec 2, 2020
b9d0cee
edits part 1 120320
censullo Dec 3, 2020
99e7421
edits part 1 120420
censullo Dec 4, 2020
09327f5
edits part 2 120420
censullo Dec 4, 2020
4143cca
edits part 3 120420
censullo Dec 4, 2020
3d81d58
edits part 1 120720
censullo Dec 7, 2020
a0efc54
edits part 1 121420
censullo Dec 14, 2020
7bec22e
edits part 2 121420
censullo Dec 14, 2020
95a3975
edits part 3 121420
censullo Dec 14, 2020
6081387
edits part 1 121520
censullo Dec 15, 2020
b384b44
edits part 1 121720
censullo Dec 17, 2020
0b01eb2
edits part 2 121720
censullo Dec 17, 2020
6967363
edits part 3 121720
censullo Dec 17, 2020
6813cc7
edits part 4 121720
censullo Dec 17, 2020
d27fd55
edits part 1 121820
censullo Dec 18, 2020
5c68024
edits part 2 121820
censullo Dec 18, 2020
08e1675
edits part 1 122120
censullo Dec 21, 2020
b07fff9
edits part 2 122120
censullo Dec 21, 2020
4e10953
edits part 1 122220
censullo Dec 22, 2020
345199a
edits part 2 122220
censullo Dec 22, 2020
9286785
edits part 3 122220
censullo Dec 22, 2020
bf7ccb3
edits part 1 122320
censullo Dec 23, 2020
98ea5de
edits part 2 122320
censullo Dec 23, 2020
0a27ae2
edits part 3 122320
censullo Dec 23, 2020
7624527
edits part 4 122320
censullo Dec 23, 2020
ed49880
edits part 5 122320
censullo Dec 23, 2020
89eea9c
edits part 6 122320
censullo Dec 23, 2020
7214277
edits part 1 122920
censullo Dec 29, 2020
df37995
edits part 1 010421
censullo Jan 4, 2021
8610363
Update README.md
censullo Jan 4, 2021
3195e23
edits part 1 010521
censullo Jan 5, 2021
2598a75
Update boilerplate
censullo Jan 5, 2021
0aba138
edits part 2 010521
censullo Jan 5, 2021
e643411
Update git2s3.template.yaml
censullo Jan 6, 2021
61b8819
edits part 2 010621
censullo Jan 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 4 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,7 @@
# quickstart-git2s3
## Git webhooks with AWS services
### Linking your Git repository to Amazon S3 and AWS services for continuous code integration, testing, and deployment
## Git webhooks with AWS services - Quick Start

This Quick Start deploys HTTPS endpoints. AWS Lambda functions and AWS CodeBuild project for implementing webhooks, to enable event-driven integration between Git services and Amazon Web Services (AWS) on the AWS Cloud.
For architectural details, step-by-step instructions, and customization options, see the [deployment guide](https://fwd.aws/6e9Rd).

After you deploy the Quick Start, you can set up a webhook that uses the endpoints to create a bridge between your Git repository and AWS services like AWS CodePipeline and AWS CodeBuild. With this setup, builds and pipeline executions occur automatically when you commit your code to a Git repository, and your code can be continuously integrated, tested, built, and deployed on AWS with each change.
To post feedback, submit feature ideas, or report bugs, use the **Issues** section of this GitHub repo.

The Quick Start includes an AWS CloudFormation template that automates the deployment. You can also use the AWS CloudFormation template as a starting point for your own implementation.

<!-- ![Quick Start architecture for implementing webhooks on AWS](https://d0.awsstatic.com/partner-network/QuickStart/datasheets/git-to-s3-webhooks-architecture-on-aws.png) -->

![Quick Start architecture for implementing webhooks on AWS](./docs/images/image3.png)

For implementation details, deployment instructions, and customization options, see the [deployment guide](https://fwd.aws/QQBRr).

To post feedback, submit feature ideas, or report bugs, use the **Issues** section of this GitHub repo.
If you'd like to submit code for this Quick Start, please review the [AWS Quick Start Contributor's Kit](https://aws-quickstart.github.io/).
To submit code for this Quick Start, see the [AWS Quick Start Contributor's Kit](https://aws-quickstart.github.io/).
2 changes: 1 addition & 1 deletion docs/boilerplate
33 changes: 33 additions & 0 deletions docs/generated/parameters/git2s3.template.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@

.General settings
[width="100%",cols="16%,11%,73%",options="header",]
|===
|Parameter label (name) |Default value|Description|Output S3 bucket name
(`OutputBucketName`)|`**__Blank string__**`|(Optional) Name for the S3 bucket where the Git repository .zip file is stored. If left blank, the Quick Start creates one for you.|Custom domain name
(`CustomDomainName`)|`**__Blank string__**`|Domain name for the webhook endpoint. If left blank, API Gateway creates a domain name for you.
|===
.Git pull settings
[width="100%",cols="16%,11%,73%",options="header",]
|===
|Parameter label (name) |Default value|Description|API secret
(`ApiSecret`)|`**__Blank string__**`|API secret used to authenticate access to webhooks in GitHub Enterprise, GitLab, and other Git services. If a webhook payload header contains a matching secret, IP address authentication is bypassed. API secrets cannot contain commas (,), backward slashes (\), or quotes (").|Allowed IP addresses
(`AllowedIps`)|`18.205.93.0/25,18.234.32.128/25,13.52.5.0/25`|Comma-separated list of allowed IP CIDR blocks. The default addresses listed are BitBucket Cloud IP ranges.|Exclude .git directory
(`ExcludeGit`)|`True`|Choose False to omit the .git directory from the Git repository .zip file.
|===
.AWS Quick Start configuration
[width="100%",cols="16%,11%,73%",options="header",]
|===
|Parameter label (name) |Default value|Description|Quick Start S3 bucket name
(`QSS3BucketName`)|`aws-quickstart`|S3 bucket name for Quick Start assets. It can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).|Quick Start S3 bucket Region
(`QSS3BucketRegion`)|`us-east-1`|AWS Region where the Quick Start assets S3 bucket (QSS3BucketName) is hosted. Required when using your own S3 bucket.|Quick Start S3 key prefix
(`QSS3KeyPrefix`)|`quickstart-git2s3/`|Key prefix for the Quick Start assets S3 bucket. A key prefix is similar to a directory name that enables you to store similar data under the same directory in an S3 bucket. It can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/).
|===
.VPC configuration
[width="100%",cols="16%,11%,73%",options="header",]
|===
|Parameter label (name) |Default value|Description|VPC ID
(`VPCId`)|`**__Blank string__**`|ID of the VPC in which the Lambda function runs.|VPC CIDR
(`VPCCidrRange`)|`**__Blank string__**`|CIDR range of the VPC.|Subnet IDs
(`SubnetIds`)|`**__Blank string__**`|SubnetIDs in which the Lambda function runs.|Hostname override
(`ScmHostnameOverride`)|`**__Blank string__**`|Name to override the hostname in the header of a webhook JSON payload.
|===
4 changes: 3 additions & 1 deletion docs/generated/parameters/index.adoc
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
// placeholder

=== Parameters for deploying into your selected Region.
include::git2s3.template.adoc[]
File renamed without changes
Binary file modified docs/images/architecture_diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
11 changes: 6 additions & 5 deletions docs/partner_editable/_settings.adoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
// Do not change this first attribute. Do change the others.
:quickstart-team-name: AWS Quick Start team
:quickstart-project-name: quickstart-git2s3
:partner-product-name: Git Webhooks
:partner-product-name: Git webhooks
// For the following attribute, if you have no short name, enter the same name as partner-product-name.
:partner-product-short-name: Git Webhooks
:partner-product-short-name: Git webhooks
// If there's no partner, comment partner-company-name and partner-contributors.
// :partner-company-name: Example Company Name, Ltd.
:doc-month: October
:doc-year: 2020
:doc-month: January
:doc-year: 2021
// For the following two "contributor" attributes, if the partner agrees to include names in the byline,
// enter names for both partner-contributors and quickstart-contributors.
// Otherwise, delete all placeholder names: everything preceding "{partner-company-name}"
// and "{quickstart-team-name}". Use commas as shown in the placeholder text.
// Use the comma before "and" only when three or more names.
// :partner-contributors: Shuai Ye, Michael McConnell, and John Smith, {partner-company-name}
:quickstart-contributors: Kirankumar Chandrashekar and Jay McConnell, Amazon Web Services
:quickstart-contributors: Kirankumar Chandrashekar and Jay McConnell, AWS Quick Start team
// For deployment_time, use minutes if deployment takes an hour or less,
// for example, 30 minutes or 60 minutes.
// Use hours for deployment times greater than 60 minutes (rounded to a quarter hour),
Expand All @@ -28,3 +28,4 @@
:disable_licenses:
:disable_regions:
:disable_requirements:
:parameters_as_appendix:
77 changes: 51 additions & 26 deletions docs/partner_editable/additional_info.adoc
Original file line number Diff line number Diff line change
@@ -1,47 +1,72 @@
==== Adding an API Secret After Deployment
=== Configuring Git services

In some cases, your Git service may provide security mechanisms like API secrets when you create the webhook. In these cases, you can launch the Quick Start with a blank parameter value for the *API Secret* parameter, and then update the stack to provide the value of the parameter. Follow these steps:
After deploying the Quick Start, set up a webhook in your Git repository.

. In the https://console.aws.amazon.com/cloudformation[AWS Cloudformation console], select the stack.
To configure a webhook, you need *GitPullWebHookApi* and *PublicSSHKey*. You can find these on the *Outputs* tab on the AWS CloudFormation console after deploying the Quick Start.

[arabic, start=2]
. Choose *Actions*, and then choose *Update Stack*.
. Keep the default to use the current template.
. On the *Specify Details* page, change the *API Secret* parameter setting.
. Choose *Next* twice.
[#outputs_tab]
.Outputs tab on the AWS CloudFormation console
[link=images/outputs_tab.png]
image::../images/outputs_tab.png[outputs_tab,width=701,height=222]

[arabic, start=10]
. Under *Capabilities*, select the check box to acknowledge that the template will create IAM resources, and then choose *Update*.
* *GitPullWebHookApi* is the URL endpoint that receives the HTTP POST request from the Git service.
* *PublicSSHKey* is the public SSH key used to connect to your Git repository. This key can be configured as a read-only machine user or as a deployment key in your Git repository.

When the status is *UPDATE_COMPLETE*, the stack has been updated with the webhook secret you specified for security.
The instructions for setting up webhooks and deployment keys vary by Git service. For more information, see your Git service documentation.

== Test the deployment
=== Configuring AWS services

Before putting the webhook into production, you should test your deployment.
After deploying the Quick Start, configure the AWS services in your workload to use the Git repository S3 bucket as a source.

. Modify a file in your repository.
As shown in <<outputs_tab>>, the *Outputs* tab in the AWS CloudFormation console includes *OutputBucketName*. This output is an Amazon S3 key that forms the base of the path to the .zip file of your repository code. The S3 key has the following format:

image:../images/image6.png[../Desktop/Screen_Shot_2017-07-15_at_11_35_57_AM.png,width=529,height=21]
```
S3://output-bucket-name/git-user/git-repository/git-user_git-repository.zip
```
Here, `git-user` is the owner or path prefix of the repository. In some Git services, this may be an organization name. However, some Git services do not return a Git user or organization for a repository. In these cases, you can omit the `git-user` parts of the path.

. Commit and push the changes.
The instructions vary for linking an AWS service to an Amazon S3 object. For links to AWS service documentation, see link:#_aws_services[AWS services], later in this guide.

== Adding an API secret after deployment

You can launch this Quick Start without an *API Secret* parameter. If your Git service provides an API secret when you create a webhook, you can update the stack with the API secret later.

To update the stack with an API secret, do the following:

image:../images/image7.png[Figure 4,width=648,height=198]
. In the https://console.aws.amazon.com/cloudformation[AWS CloudFormation console], select the stack you want to update.
. In the stack details pane, choose *Update*.
. Choose *Use current template*.
. On the *Specify stack details* page, change the *API Secret* parameter setting, then choose *Next*.
. On the *Configure stack options* page, choose *Next*.
. Select *I acknowledge that this template may create IAM resources*.
. Choose *Update stack*. When the status is *UPDATE_COMPLETE*, the stack is updated with the API secret.

[start=2]
. Wait a few minutes and check your S3 bucket for a new (or updated) object with a key that matches your repository path.
== Test the deployment

Before putting a webhook into production, test your deployment by doing the following:

image:../images/image8.png[Figure 5,width=646,height=348]
. Modify a file in your repository.
. Commit and push the changes.
. Wait a few minutes, and then check the Git repository S3 bucket for a new (or updated) object with a key that matches your repository path.

*Figure 4: Checking for S3 bucket updates after a commit*
:xrefstyle: short
[#S3_new_object]
.Checking for a new or updated object in your S3 bucket after a commit
[link=images/S3_new_object.png]
image::../images/S3_new_object.png[S3_new_object,width=646,height=348]

=== Best Practices
== Best practices

The architecture built by this Quick Start supports AWS best practices for security.

==== SSH Keys
=== SSH keys

This Quick Start deploys a private SSH key pair that is encrypted with an AWS KMS key and uploaded to Amazon S3. AWS CodeBuild decrypts the private SSH key and uses it to authenticate your Git service before cloning the repository.

We don’t recommend sharing SSH keys among multiple services, or launching another instance of this Quick Start to clone and store another repository in Amazon S3. Each repository should use unique SSH keys.

SSH keys are generated at stack creation, and are then encrypted using AWS KMS and the encrypted copy stored in Amazon S3. When you use the Git pull endpoint, the private key is fetched by the Lambda function, decrypted, and used to authenticate against your Git service to perform a clone of the repository. We don’t recommend (a) reusing SSH keys for multiple services, or (b) launching another instance of this Quick Start for each repository that you wish to clone to Amazon S3; this ensures that each repository uses unique keys.
=== Webhook security

==== Webhook Security
Git services provide different ways to authenticate an endpoint, such as webhook secrets, source-IP-address allow listing, personal access tokens, and OAuth2. We recommend that you set up at least one of these security mechanisms to protect your webhook API endpoint.

Different Git services provide varying ways to authenticate against an endpoint. The Git pull endpoint supports webhook secrets (used by GitHub Enterprise, GitLab, and other Git repository managers) as well as source IP address whitelisting. The zip download endpoint supports personal access tokens (as used by GitHub Enterprise and GitLab) and OAuth2 (used by Bitbucket). We recommend that you set up at least one of these security mechanisms to protect your webhook API endpoint. For more information about how this Quick Start utilizes these mechanisms, see the parameters in the link:#deployment-steps[Deployment Steps] section of this guide. For product-specific guidance on how to configure these security mechanisms, refer to your Git product’s documentation.
For more information about how this Quick Start uses endpoint security mechanisms, see the link:#_parameter_reference[Parameter reference] section of this guide. For specific guidance on how to configure security mechanisms for your Git service, refer to your Git service documentation.
41 changes: 19 additions & 22 deletions docs/partner_editable/architecture.adoc
Original file line number Diff line number Diff line change
@@ -1,27 +1,24 @@
Deploying this Quick Start builds the following environment in the AWS Cloud.

// Replace this example diagram with your own. Send us your source PowerPoint file. Be sure to follow our guidelines here : http://(we should include these points on our contributors giude)

:xrefstyle: short
[#architecture1]
// .Quick Start architecture for {partner-product-short-name} on AWS
[#architecture_diagram]
.Git webhooks with AWS services Quick Start architecture
[link=images/architecture_diagram.png]
image::../images/image3.png[Architecture,width=100%,height=100%]


*Figure 2: Webhook endpoint architecture on AWS*


The Quick Start deployment sets up a serverless AWS Cloud environment that includes the following components.

* An API Gateway endpoint to accept the webhook requests from Git.
* Lambda function that processes the input git webhook payload received by the API Gateway and submits a build on AWS CodeBuild Project.
* An AWS CodeBuild project to connect to the Git service, either over SSH or through the Git service’s endpoint. This AWS CodeBuild project will zip the code and upload it to Amazon S3.

*Important* The AWS CodeBuild project that is deployed by this Quick Start must be able to communicate with your Git repository. For example, you can use a SaaS-based Git service that the AWS CodeBuild can reach through the internet.

* An AWS KMS key to encrypt the SSH private key used to connect to the repository over SSH.
* Two S3 buckets: One bucket stores the zipped contents of your Git repository, and the second bucket stores the AWS KMS-encrypted SSH private keys that are generated during stack creation. Note that the first bucket has versioning enabled, and all previous versions are retained indefinitely. If you’d like to manage the retention period for old versions, follow the instructions in the http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html[Amazon S3 documentation].
* Several IAM roles required for the Lambda functions and API Gateway. The inline permissions attached to these roles are scoped using the http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege[least privilege] model.
* Two Lambda-backed AWS CloudFormation custom resources. One resource generates an SSH keypair, encrypts it using AWS KMS, and stores it in Amazon S3. The second resource deletes the content of the two S3 buckets on stack deletion.

NOTE: If you need backups, make sure to copy the S3 bucket contents before you delete the CloudFormation stack.
image::../images/architecture_diagram.png[architecture_diagram,width=100%,height=100%]

As shown in <<architecture_diagram>>, this Quick Start sets up a serverless AWS Cloud environment that includes the following components:

* Amazon API Gateway to receive Git webhook requests and forward them to AWS Lambda.
* An AWS Lambda function to process Git webhook requests from API Gateway and invoke an AWS CodeBuild project.
* An AWS CodeBuild project to connect to your Git service, then retrieve, zip, and upload the latest version of your Git repository to Amazon S3.
* An AWS Key Management Service (AWS KMS) key to encrypt/decrypt the SSH (Secure Shell) keys used by AWS CodeBuild to connect to your Git repository using SSH. The SSH key pair is generated by a Lambda-backed AWS CloudFormation custom resource when the stack is deployed.
* Two Amazon S3 buckets: one for Git repository contents, and another for encrypted SSH keys. A Lambda-backed AWS CloudFormation custom resource deletes the contents of the S3 buckets when you delete the CloudFormation stack. If you need backups, copy the S3 buckets before deleting the stack.

[NOTE]
========
* The Quick Start deploys AWS Identity and Access Management (IAM) roles required by Lambda and API Gateway. The inline permissions attached to the roles are scoped using the http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege[least privilege] model.
* The AWS CodeBuild project must be able to communicate with your Git repository. For example, you can employ a SaaS-based Git service like GitHub to which CodeBuild can connect over the internet.
* The Git repository S3 bucket this Quick Start deploys has versioning enabled, and all previous versions are retained indefinitely. To modify the retention period, see http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html[How do I create a lifecycle rule for an S3 bucket?]
========
Loading