Skip to content

Commit

Permalink
Merge pull request #4 from mikegrima/docs
Browse files Browse the repository at this point in the history 
Added Docusaurus documentation
  • Loading branch information
@mikegrima
mikegrima committed Dec 25, 2017
2 parents c8b25d6 + bcbeea5 commit 282338d
Show file tree
Hide file tree
Showing 21 changed files with 2,700 additions and 213 deletions.
218 changes: 5 additions & 213 deletions README.md
View file

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions docs/GenerateDocs.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
How to generate the docs:
--------------------
1. Install Yarn
1. Navigate to the `website` dir.
1. Run `yarn build`
1. Copy `build/` to a different directory
1. Checkout the `gh-pages` branch: `git checkout gh-pages`
1. Copy and paste over the contents of `build/bucketsnake/*` over the contents of `/`
1. Commit it and push to the `gh-pages`
130 changes: 130 additions & 0 deletions docs/configuration.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
id: configuration
title: Configuring the Bucket Snake Lambda Function
sidebar_label: Configuration
---

Bucket Snake has a number of environment variables that it can be configured with. The table below outlines what they are and if they are required.

<table>
<col width="190px" />
<col width="120px" />
<thead>
<tr>
<th class="centerCell">Environment Variable</th>
<th class="centerCell">Default Value</th>
<th class="centerCell">Required</th>
<th class="centerCell">Environment-Variable Description</th>
<th class="centerCell">Example</th>
</tr>
</thead>
<tbody>
<tr>
<td class="centerCell"><code>APP_REPORTS_BUCKETS</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">This is a comma-separated list of S3 bucket names which contain the historical S3 report JSON. This is a list to permit applications to use per-region buckets.</td>
<td class="centerCell"><code>"historical-report-bucket-us-east-1,historical-report-bucket-us-west-2,historical-report-bucket-eu-west-1"</code><br />(Replace with your buckets)</td>
</tr>
<tr>
<td class="centerCell"><code>SWAG_BUCKET</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">This is the S3 bucket that contains the <a href="https://github.com/Netflix-Skunkworks/swag-api">SWAG</a> data set.</td>
<td class="centerCell"><code>"swag-data-set-bucket-here"</code><br />(Replace with your bucket)</td>
</tr>
<tr>
<td class="centerCell"><code>SWAG_REGION</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">The region for where the SWAG bucket lives.</td>
<td class="centerCell"><code>"us-east-1"</code><br />(Replace with your SWAG bucket region)</td>
</tr>
<tr>
<td class="centerCell"><code>SWAG_DATA_FILE</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">The prefix to where the accounts JSON lives in the SWAG bucket.</td>
<td class="centerCell"><code>"v2/accounts.json"</code><br />(Replace with your prefix)</td>
</tr>
<tr>
<td class="centerCell"><code>REPORTS_BUCKET</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">The S3 bucket that contains the Historical S3 report JSON that Bucket Snake will use. This is just 1 bucket vs. a list for what is granted to the application.</td>
<td class="centerCell"><code>"historical-report-bucket-us-east-1"</code><br />(Replace with your bucket)</td>
</tr>
<tr>
<td class="centerCell"><code>REPORTS_REGION</code></td>
<td class="centerCell">None</td>
<td class="centerCell"><strong>YES</strong></td>
<td class="nocenterCell">The region of the S3 bucket that contains the historical report.</td>
<td class="centerCell"><code>"us-east-1"</code><br />(Replace with your Historical report bucket region)</td>
</tr>
<tr>
<td class="centerCell"><code>REPORTS_PREFIX</code></td>
<td class="centerCell"><code>"historical-s3-report.json"</code></td>
<td class="centerCell"><strong>No</strong></td>
<td class="nocenterCell">The region of the S3 bucket that contains the historical report.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>BLACKLISTED_SOURCE_ACCOUNTS</code></td>
<td class="centerCell">None</td>
<td class="centerCell">No</td>
<td class="nocenterCell">A comma-separated list of AWS 12-digit account IDs for where source IAM roles are not permitted to use Bucket Snake for S3 access. Bucket Snake will not operate for source application IAM roles in these accounts.</td>
<td class="centerCell"><code>"0123456678910,012345678911"</code><br />(Replace with your account IDs)</td>
</tr>
<tr>
<td class="centerCell"><code>BLACKLISTED_BUCKET_ACCOUNTS</code></td>
<td class="centerCell">None</td>
<td class="centerCell">No</td>
<td class="nocenterCell">A comma-separated list of AWS 12-digit account IDs for accounts that Bucket Snake should not grant S3 access. I.e. a bucket in an account that is protected, and Bucket Snake should not be granting access to.</td>
<td class="centerCell"><code>"0123456678910,012345678911"</code><br />(Replace with your account IDs)</td>
</tr>
<tr>
<td class="centerCell"><code>BUCKET_SNAKE_POLICY_NAME</code></td>
<td class="centerCell"><code>"BucketSnake"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The IAM policy name on the IAM role that grants S3 access.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>STS_POLICY_NAME</code></td>
<td class="centerCell"><code>"BucketSnakeAssumeRole"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The IAM policy name on the source IAM role that grants <code>sts:AssumeRole</code> permissions to the destination AWS account S3 roles.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>DEST_ROLE_DESCRIPTION</code></td>
<td class="centerCell"><code>"Bucket Snake provisioned role"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The description to the destination S3 IAM roles provisioned by Bucket Snake.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>BUCKET_SNAKE_ROLE</code></td>
<td class="centerCell"><code>"BucketSnake"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The name of the IAM role that Bucket Snake needs to assume into to perform destination AWS account activities.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>BUCKET_SNAKE_SESSION_NAME</code></td>
<td class="centerCell"><code>"BucketSnake"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The name of the STS session name that Bucket Snake will use when it assumes to the destination AWS account IAM roles.</td>
<td class="centerCell">See Default</td>
</tr>
<tr>
<td class="centerCell"><code>IAM_REGION</code></td>
<td class="centerCell"><code>"us-east-1"</code></td>
<td class="centerCell">No</td>
<td class="nocenterCell">The AWS region for where IAM API commands are sent.</td>
<td class="centerCell">See Default</td>
</tr>
</tbody>
</table>

These variables can be supplied in the [serverless](https://github.com/Netflix-Skunkworks/bucketsnake/tree/master/docs/serverless-examples) configuration.
109 changes: 109 additions & 0 deletions docs/howitworks.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
id: howitworks
title: How Bucket Snake Works
sidebar_label: How it works
---

Bucket Snake works by creating and provisioning IAM roles in the accounts that own buckets for a given application.

The general flow works as follows:
1. Bucket Snake gets triggered with a payload describing:
- The name of the application
- The IAM role name that application uses
- The AWS account the application's IAM role resides in.
- A set of S3 buckets, and the corresponding permissions to grant
1. With the above information, Bucket Snake figures out where the buckets are located, creates the necessary roles with the access requested.
- If an S3 bucket resides in the same account as the source application, Bucket Snake simply grants permissions to the source application, so no role assumption is required.

## Sources of Truth
Bucket Snake depends on [SWAG](https://github.com/Netflix-Skunkworks/swag-api), [Historical](https://github.com/Netflix-Skunkworks/historical), and the [Historical S3 Report](https://github.com/Netflix-Skunkworks/historical-reports).

SWAG is a schema for the accounts in your infrastructure. It's a source of truth to know which AWS accounts you have, and some details about them.

Historical keeps track of all S3 buckets in your infrastructure, and the Historical S3 Report is a JSON file that is a dump of all S3 buckets currently in your infrastructure. This is used as a lookup-table to know which buckets you have, which region and AWS account they reside in.

### Historical Reports
The historical s3 report is needed because S3 ARNs don't provide enough detail on S3 buckets. For example, one cannot determine the account and region by just having an S3 ARN or bucket name.

The Historical S3 report solves this by providing a look-up table of S3 bucket, and the corresponding region and AWS account for the bucket. This is what Bucket Snake uses to determine if S3 access is cross-account.

A Bucket Snake aware client needs access to this data source to determine if role-assumption is required. It is assumed that this report is stored in an S3 bucket that permits multiple accounts in your infrastructure access to it (on the bucket policy). Bucket Snake will grant the application access to this JSON file.

## Triggering the Lambda
Bucket Snake is triggered from an AWS lambda function invocation that has a payload with this schema:

{
"role_name": "AppSourceIamRole",
"app_name": "nameOfAppWithSourceIAMRole",
"account_number": "The12digitAWSAccountIDWhereTheAppSourceIAMRoleLives",
"buckets": {
"name-of-s3-bucket": [
{
"prefix": "*",
"perms": [
"list"
]
},
{
"prefix": "some/prefix/here",
"perms": [
"get",
"put",
"delete"
]
}
]
"another-s3-bucket": [
"perms": [
"get"
],
"prefix: "*"
],
"and-another-s3-bucket": [
"perms": [
"put"
],
"prefix": "some/drop/location"
],
...
}
}


### Now what?
Bucket Snake would receive the JSON from the lambda invocation, and from that, would:
1. Verify that the source IAM role exists
1. Verify that the buckets exist, and are permitted by Bucket Snake (more on this in the configuration section)
1. Determine which S3 buckets are in the same account as the source application, and which are not
1. For buckets in the same account, Bucket Snake will add in the proper S3 permissions to the source app IAM role
1. For buckets that are not in the same account, Bucket Snake will create IAM roles in the destination accounts with access to the respective buckets
- Destination IAM role name follows the format: `AppName-12DigitSourceAccountNumber`.
- This role will have a trust policy that allows the source application `sts:AssumeRole` access to it.
1. If applicable, a policy will be added to the source IAM role to grant `sts:AssumeRole` access to those destination
IAM roles
1. And lastly, Bucket Snake will grant access to the Historical S3 report's JSON file so that application knows
which S3 buckets require the role assumption for access.

## How does my application make use of this?
At present no "Bucket Snake aware" client library exists. We are currently in the process of developing one for Python and Java.

This client would work by:
1. Fetch the Historical S3 JSON (access granted by Bucket Snake)
1. Check if the S3 bucket is in the same account (this information lives in the Historical report).

If it's in the same account, then the client directly access the bucket with the on-instance
IAM credentials.

If the bucket is in a _different account_:
1. Assume to the destination role (named `AppName-12DigitSourceAccountNumber`),
1. Cache the credentials for future use (re-assume when expired)
1. Use the assumed credentials to access the S3 bucket

## What about future access?
Simply pass in a new payload to the lambda with additional buckets to add access to. Bucket Snake is [idempotent](https://en.wikipedia.org/wiki/Idempotence).

The Bucket Snake policies should not be modified outside of Bucket Snake. You can modify any policy outside
of the Bucket Snake managed ones -- Bucket Snake will not alter or modify them. This is useful should you need to add additional permissions than what Bucket Snake would provide.

## Which permissions are created?
See the next section for details.
118 changes: 118 additions & 0 deletions docs/installation.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
---
id: installation
title: Deploying the Bucket Snake Lambda Function
sidebar_label: Deployment
---

Bucket Snake can be deployed with [Serverless](https://serverless.com). See the [docs/serverless-examples](https://github.com/Netflix-Skunkworks/bucketsnake/tree/master/docs/serverless-examples) for an example.

## What is required?
Bucket Snake depends on [SWAG](https://github.com/Netflix-Skunkworks/swag-api), [Historical](https://github.com/Netflix-Skunkworks/historical), and the [Historical S3 Report](https://github.com/Netflix-Skunkworks/historical-reports).

For a summary of these tools, please visit their respective pages and review the documentation.

### Historical S3 Report
Bucket Snake has a hard dependency on the Historical S3 Report, which is provided as part of the [Historical-Reports](https://github.com/Netflix-Skunkworks/historical-reports).

Bucket Snake doesn't require much from the report. It requires:
1. The bucket name
1. The bucket region
1. The bucket account

This can be achieved with a Historical S3 report that has the `EXCLUDE_FIELDS` environmental variable configured with the value:

Name,_version,Grants,LifecycleRules,Logging,Policy,Tags,Versioning,Website,Cors,Notifications,Acceleration,Replication,CreationDate,AnalyticsConfigurations,MetricsConfigurations,InventoryConfigurations

A slim report is not required -- but it makes for a smaller JSON file for client applications to fetch from S3 (or have included with their deployment).

#### Historical Report Bucket
An S3 Bucket that contains this report must be available for the Bucket Snake lambda function. This is required to be set in the configuration. (See the next section for details)

The Historical S3 report should live in a bucket that all applications in your infrastructure can access. This will allow applications to fetch this file to make a determination if cross-account access is required.

### SWAG
SWAG is a hard requirement for Bucket Snake. The JSON must be accessible in S3 and is configurable. (See the next section for details)

## IAM Roles
Bucket Snake operates from a hub-spoke type of model. The lambda function itself requires an IAM role, which then assumes into other account IAM roles to provision the S3 access for a given application.

Please use your favorite tool to sync these roles out across your environment.

#### Bucket Snake Lambda Function IAM Role

The trust policy must be similar to:

{
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
]
}

The inline-polices must be similar to:

{
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Sid": "HistoricalS3",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::historical-s3-report-bucket/prefix/to/historical-s3-reports.json"
},
{
"Sid": "AssumeToRoles",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::*:role/BucketSnake"
}
]
}


#### In-account Bucket Snake IAM Role (Destination Roles)

The trust policy must be similar to:

{
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::SOURCE-BUCKET-SNAKE-LAMBDA-ACCOUNT-HERE:role/BucketSnakeLambdaProfile"
}
}
]
}

The inline-polices must be similar to:

{
"Statement": [
{
"Action": [
"iam:CreateRole",
"iam:GetRole",
"iam:PutRolePolicy",
"iam:UpdateAssumeRolePolicy"
],
"Resource": "*",
"Effect": "Allow"
}
]
}

0 comments on commit 282338d

Please sign in to comment.