From b14f2b6dfbeec6cdbdccee829cac372d4a1abe21 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Tue, 16 Nov 2021 11:26:46 -0500 Subject: [PATCH 1/8] Adding information about creating an IAM role using the CLI --- .../storage/catalog/aws-s3/index.md | 100 +++++++++++++++++- 1 file changed, 98 insertions(+), 2 deletions(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 8a3ba1ceae..115c105169 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -34,9 +34,9 @@ The Segment Tracking API processes data from your sources, and collects the Even ## Create a new destination -Complete the following steps to configure the AWS S3 Destination with IAM Role Support. +Complete either [Create an IAM role in the AWS console](#create-an-iam-role-in-the-aws-console) or [Create an IAM role using the AWS CLI](#create-an-aws-role-using-the-aws-cli) to configure the AWS S3 Destination with IAM Role Support. -### Create an IAM role in AWS +### Create an IAM role in the AWS console To complete this section, you need access to your AWS dashboard. @@ -100,6 +100,102 @@ To complete this section, you need access to your AWS dashboard. If you have server-side encryption enabled, see the [required configuration](#encryption). +### Create an IAM role using the AWS CLI + +To create an IAM role with external ID and with S3 permissions using the AWS CLI, follow the steps below. + +#### Prerequisites +To create an S3 IAM role, you must first install and configure the AWS CLI on your local machine and create an S3 bucket. Refer to Amazon's documentation, [Getting started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) for more information. + +#### Procedure +1. Copy the following code snippet and save it as a file on your local machine titled `trust-relationship-policy.json`. Replace `` with your Segment workspace ID. + +```json + +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::595280932656:role/segment-s3-integration-production-access" + }, + "Action": "sts:AssumeRole", + "Condition": { + "StringEquals": { + "sts:ExternalId": "" + } + } + } + ] +} +``` + +2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give this IAM role: + +``` python +aws iam create-role --role-name --assume-role-policy-document file://trust-relationship-policy.json --description "IAM role for Segment to assume (AWS S3 destination)" +``` + +>info "" +> To verify that you successfully created an IAM role, log into your AWS console and open the IAM Management Console. Under the Trust Relationship tab, there should be a key-value pair: a `sts:ExternalID` key with a value of `your Segment workspace ID`. + +3. Copy the following IAM policy, replacing `` with the name of your S3 bucket, and save it as a file on your local machine titled `iam-policy.json`. + +```json + +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "PutObjectsInBucket", + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": "arn:aws:s3:::/segment-logs/*" + } + ] +} + +``` + +5. Navigate to the folder containing `iam-policy.json`, and run the following command to create the IAM policy: + +``` python +aws iam create-policy --policy-name segment-s3-putobject --policy-document file://iam-policy.json --description "Allow Segment to PutObject into S3 destination bucket" +``` +6. A successful output has the following format. Take note of the `Arn,` as you'll need it in the next step. + +``` json + +{ + "Policy": { + "PolicyName": "segment-s3-putobject", + "PolicyId": "AABC1DE2F34GG567H", + "Arn": "arn:aws:iam::012345678912:policy/segment-s3-putobject", + "Path": "/", + "DefaultVersionId": "v1", + "AttachmentCount": 0, + "PermissionsBoundaryUsageCount": 0, + "IsAttachable": true, + "CreateDate": "2021-11-11T01:21:00+00:00", + "UpdateDate": "2021-11-11T01:21:00+00:00" + } +} + +``` +7. Run the following command to attach the IAM policy to the IAM role, replacing `` with the name of your role and `` with the Arn output from the last step: + +``` python +aws iam attach-role-policy --role-name --policy-arn +``` + +> info "" +> To verify that you have successfully created your IAM role, navigate to your AWS console and open the IAM Management Console. On the Permissions tab, verify that there is a `segment-s3-putobject` Permissions policy. + ### Add the AWS S3 with IAM Role Support Destination From cae2e3f1c2dd7b243ae53385912c2a691e620d63 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Tue, 16 Nov 2021 14:56:36 -0500 Subject: [PATCH 2/8] Fixing broken links, spelling/casing errors --- src/connections/storage/catalog/aws-s3/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 115c105169..ef3c677367 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -34,7 +34,7 @@ The Segment Tracking API processes data from your sources, and collects the Even ## Create a new destination -Complete either [Create an IAM role in the AWS console](#create-an-iam-role-in-the-aws-console) or [Create an IAM role using the AWS CLI](#create-an-aws-role-using-the-aws-cli) to configure the AWS S3 Destination with IAM Role Support. +Complete either [Create an IAM role in the AWS console](#create-an-iam-role-in-the-aws-console) or [Create an IAM role using the AWS CLI](#create-an-iam-role-using-the-aws-cli) to configure the AWS S3 Destination with IAM Role Support. ### Create an IAM role in the AWS console @@ -105,7 +105,7 @@ If you have server-side encryption enabled, see the [required configuration](#en To create an IAM role with external ID and with S3 permissions using the AWS CLI, follow the steps below. #### Prerequisites -To create an S3 IAM role, you must first install and configure the AWS CLI on your local machine and create an S3 bucket. Refer to Amazon's documentation, [Getting started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) for more information. +To create an S3 IAM role, you must first install and configure the AWS CLI on your local machine and create an S3 bucket. Refer to Amazon's documentation, [Getting started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html){:target="_blank"} for more information. #### Procedure 1. Copy the following code snippet and save it as a file on your local machine titled `trust-relationship-policy.json`. Replace `` with your Segment workspace ID. @@ -222,7 +222,7 @@ To finish configuration, enable the AWS S3 Destination with IAM Role Support des To migrate an existing Amazon S3 destination to the AWS S3 with IAM Role Support Destination: -1. Configure the IAM role and IAM policy permissions as described in steps 2 - 4 [above](#create-an-iam-role-in-aws). +1. Configure the IAM role and IAM policy permissions as described in steps 2 - 4 [above](#create-an-iam-role-in-the-aws-console). 2. Add the AWS S3 with IAM Role Support Destination and add the AWS Region and IAM role ARN. For the bucket name, enter `/segment-logs/test`. Enable the destination, and verify data is received at `/segment-logs/test/segment-logs`. If the folder receives data, continue to the next step. If you don't see log entries, check the trust relationship document and IAM policy attached to the role. 3. Update the bucket name in the new destination to ``. 4. After 1 hour, disable the original Amazon S3 destination. @@ -254,7 +254,7 @@ Segment groups logs by day, and names them using the following format: s3://{bucket}/segment-logs/{source-id}/{received-day}/filename.gz -The received-day refers to the UTC date unix timestamp, that the API receives the file, which makes it easy to find all calls received within a certain timeframe. +The received-day refers to the UTC date Unix timestamp, that the API receives the file, which makes it easy to find all calls received within a certain timeframe. ## Encryption From 4a233087f450605cacbbd544219d688558544ca6 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 09:24:07 -0500 Subject: [PATCH 3/8] Making changes requested in code review --- .../storage/catalog/aws-s3/index.md | 123 +++++++++--------- 1 file changed, 60 insertions(+), 63 deletions(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index ef3c677367..1bb69ae4ec 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -110,91 +110,88 @@ To create an S3 IAM role, you must first install and configure the AWS CLI on yo #### Procedure 1. Copy the following code snippet and save it as a file on your local machine titled `trust-relationship-policy.json`. Replace `` with your Segment workspace ID. -```json - -{ - "Version": "2012-10-17", - "Statement": [ + ```json { - "Sid": "", - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam::595280932656:role/segment-s3-integration-production-access" - }, - "Action": "sts:AssumeRole", - "Condition": { - "StringEquals": { - "sts:ExternalId": "" + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::595280932656:role/segment-s3-integration-production-access" + }, + "Action": "sts:AssumeRole", + "Condition": { + "StringEquals": { + "sts:ExternalId": "" + } + } } - } + ] } - ] -} -``` + ``` 2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give this IAM role: -``` python -aws iam create-role --role-name --assume-role-policy-document file://trust-relationship-policy.json --description "IAM role for Segment to assume (AWS S3 destination)" -``` - ->info "" -> To verify that you successfully created an IAM role, log into your AWS console and open the IAM Management Console. Under the Trust Relationship tab, there should be a key-value pair: a `sts:ExternalID` key with a value of `your Segment workspace ID`. + ``` python + aws iam create-role --role-name --assume-role-policy-document file://trust-relationship-policy.json --description "IAM role for Segment to assume (AWS S3 destination)" + ``` -3. Copy the following IAM policy, replacing `` with the name of your S3 bucket, and save it as a file on your local machine titled `iam-policy.json`. +3. To verify that the IAM role is created, log into the AWS console and open the IAM Management Console. Under the Trust Relationship tab, there should be a key-value pair: a `sts:ExternalID` key with a value of `your Segment workspace ID`. -```json +4. Copy the following IAM policy, replacing `` with the name of your S3 bucket, and save it as a file on your local machine titled `iam-policy.json`. -{ - "Version": "2012-10-17", - "Statement": [ + ```json { - "Sid": "PutObjectsInBucket", - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:PutObjectAcl" - ], - "Resource": "arn:aws:s3:::/segment-logs/*" + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "PutObjectsInBucket", + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": "arn:aws:s3:::/segment-logs/*" + } + ] } - ] -} - -``` + ``` 5. Navigate to the folder containing `iam-policy.json`, and run the following command to create the IAM policy: -``` python -aws iam create-policy --policy-name segment-s3-putobject --policy-document file://iam-policy.json --description "Allow Segment to PutObject into S3 destination bucket" -``` + ``` python + aws iam create-policy --policy-name segment-s3-putobject --policy-document file://iam-policy.json --description "Allow Segment to PutObject into S3 destination bucket" + ``` + 6. A successful output has the following format. Take note of the `Arn,` as you'll need it in the next step. -``` json - -{ - "Policy": { - "PolicyName": "segment-s3-putobject", - "PolicyId": "AABC1DE2F34GG567H", - "Arn": "arn:aws:iam::012345678912:policy/segment-s3-putobject", - "Path": "/", - "DefaultVersionId": "v1", - "AttachmentCount": 0, - "PermissionsBoundaryUsageCount": 0, - "IsAttachable": true, - "CreateDate": "2021-11-11T01:21:00+00:00", - "UpdateDate": "2021-11-11T01:21:00+00:00" + ``` json + { + "Policy": { + "PolicyName": "segment-s3-putobject", + "PolicyId": "AABC1DE2F34GG567H", + "Arn": "arn:aws:iam::012345678912:policy/segment-s3-putobject", + "Path": "/", + "DefaultVersionId": "v1", + "AttachmentCount": 0, + "PermissionsBoundaryUsageCount": 0, + "IsAttachable": true, + "CreateDate": "2021-11-11T01:21:00+00:00", + "UpdateDate": "2021-11-11T01:21:00+00:00" + } } -} -``` + ``` + 7. Run the following command to attach the IAM policy to the IAM role, replacing `` with the name of your role and `` with the Arn output from the last step: -``` python -aws iam attach-role-policy --role-name --policy-arn -``` + ``` python + aws iam attach-role-policy --role-name --policy-arn + ``` > info "" -> To verify that you have successfully created your IAM role, navigate to your AWS console and open the IAM Management Console. On the Permissions tab, verify that there is a `segment-s3-putobject` Permissions policy. +> To verify that the IAM role is created, navigate to the AWS console and open the IAM Management Console. On the Permissions tab, verify that there is a `segment-s3-putobject` Permissions policy. ### Add the AWS S3 with IAM Role Support Destination From 93a726622639c63d7a17bf215ce4b16569a85bfe Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 09:50:34 -0500 Subject: [PATCH 4/8] Fixed code snippet error --- src/connections/storage/catalog/aws-s3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 1bb69ae4ec..1f81ee37c9 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -164,7 +164,7 @@ To create an S3 IAM role, you must first install and configure the AWS CLI on yo aws iam create-policy --policy-name segment-s3-putobject --policy-document file://iam-policy.json --description "Allow Segment to PutObject into S3 destination bucket" ``` -6. A successful output has the following format. Take note of the `Arn,` as you'll need it in the next step. +6. A successful output has the following format. Take note of the `Arn`, as you'll need it in the next step. ``` json { From 31ef9634dc55f404b089bdb768b562fe7e8e1a37 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 12:39:26 -0500 Subject: [PATCH 5/8] Removed beta flag from documentation, as per 11/17 data lakes and blob storage meeting --- src/connections/storage/catalog/aws-s3/index.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 1f81ee37c9..5d6cb9a56b 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -5,9 +5,6 @@ redirect_from: hide-personas-partial: true --- -> info "This document is about a destination which is in beta" -> This means that the AWS S3 with IAM Role Support destination is in active development, and some functionality may change before it becomes generally available. - ## Differences between the Amazon S3 destination and the AWS S3 destination From 2085a660ceda556e2231e36d56cb58f6dd9eba19 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 13:15:46 -0500 Subject: [PATCH 6/8] [netlify-build] --- src/connections/storage/catalog/aws-s3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 5d6cb9a56b..562bc95bc8 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -128,7 +128,7 @@ To create an S3 IAM role, you must first install and configure the AWS CLI on yo } ``` -2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give this IAM role: +2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give the IAM role: ``` python aws iam create-role --role-name --assume-role-policy-document file://trust-relationship-policy.json --description "IAM role for Segment to assume (AWS S3 destination)" From f2adfa512ddaf8c2e4f9b38a47e5b1943476c212 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 13:17:03 -0500 Subject: [PATCH 7/8] the -> this --- src/connections/storage/catalog/aws-s3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 562bc95bc8..5d6cb9a56b 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -128,7 +128,7 @@ To create an S3 IAM role, you must first install and configure the AWS CLI on yo } ``` -2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give the IAM role: +2. Navigate to the folder containing `trust-relationship-policy.json` and run the following command to create your IAM role and attach the trust relationship document, replacing `` with the name you want to give this IAM role: ``` python aws iam create-role --role-name --assume-role-policy-document file://trust-relationship-policy.json --description "IAM role for Segment to assume (AWS S3 destination)" From ad0dcddae2b56fb8712680cb0bb12f51261b5aa7 Mon Sep 17 00:00:00 2001 From: forstisabella <92472883+forstisabella@users.noreply.github.com> Date: Wed, 17 Nov 2021 13:31:13 -0500 Subject: [PATCH 8/8] Fixing json spacing in AWS console procedure --- src/connections/storage/catalog/aws-s3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/storage/catalog/aws-s3/index.md b/src/connections/storage/catalog/aws-s3/index.md index 5d6cb9a56b..5b19e97f83 100644 --- a/src/connections/storage/catalog/aws-s3/index.md +++ b/src/connections/storage/catalog/aws-s3/index.md @@ -91,7 +91,7 @@ To complete this section, you need access to your AWS dashboard. ], "Resource": "" } - ] + ] } ```