Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove IAM Role Setup instructions from s3.md #35190

Merged
merged 2 commits into from
Feb 13, 2024
Merged

Remove IAM Role Setup instructions from s3.md #35190

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae2346d
Remove IAM Role Setup instructions from s3.md
tybernstein Feb 12, 2024
105393e
Merge branch 'master' into tybernstein-patch-1
tybernstein Feb 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 7 additions & 68 deletions docs/integrations/sources/s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Please note that using cloud storage may incur egress costs. Egress refers to da

### Step 1: Set up Amazon S3

**If you are syncing from a private bucket**, you need to authenticate the connection. This can be done either by using an `IAM User` (with `AWS Access Key ID` and `Secret Access Key`) or an `IAM Role` (with `Role ARN`). Begin by creating a policy with the necessary permissions:
**If you are syncing from a private bucket**, you will need to provide both an `AWS Access Key ID` and `AWS Secret Access Key` to authenticate the connection. The IAM user associated with the credentials must be granted `read` and `list` permissions for the bucket and its objects. If you are unfamiliar with configuring AWS permissions, you can follow these steps to obtain the necessary permissions and credentials:

#### Create a Policy

Expand Down Expand Up @@ -47,70 +47,11 @@ At this time, object-level permissions alone are not sufficient to successfully
:::

4. Give your policy a descriptive name, then click **Create policy**.

#### Option 1: Using an IAM Role (Most secure)

<!-- env:cloud -->
:::note
This authentication method is currently in the testing phase. To enable it for your workspace, please contact our Support Team.
:::
<!-- /env:cloud -->

1. In the IAM dashboard, click **Roles**, then **Create role**. <!-- env:oss -->
2. Choose the appropriate trust entity and attach the policy you created.
3. Set up a trust relationship for the role. For example for **AWS account** trusted entity use default AWS account on your instance (it will be used to assume role). To use **External ID** set it to environment variables as `export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}"`. Edit the trust relationship policy to reflect this:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-external-id}"
}
}
}
]
}
```
<!-- /env:oss -->
<!-- env:cloud -->
2. Choose the **AWS account** trusted entity type.
3. Set up a trust relationship for the role. This allows the Airbyte instance's AWS account to assume this role. You will also need to specify an external ID, which is a secret key that the trusting service (Airbyte) and the trusted role (the role you're creating) both know. This ID is used to prevent the "confused deputy" problem. The External ID should be your Airbyte workspace ID, which can be found in the URL of your workspace page. Edit the trust relationship policy to include the external ID:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-airbyte-workspace-id}"
}
}
}
]
}
```
<!-- /env:cloud -->
4. Complete the role creation and note the Role ARN.

#### Option 2: Using an IAM User

1. In the IAM dashboard, click **Users**. Select an existing IAM user or create a new one by clicking **Add users**.
2. If you are using an _existing_ IAM user, click the **Add permissions** dropdown menu and select **Add permissions**. If you are creating a _new_ user, you will be taken to the Permissions screen after selecting a name.
3. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.
4. After successfully creating your user, select the **Security credentials** tab and click **Create access key**. You will be prompted to select a use case and add optional tags to your access key. Click **Create access key** to generate the keys.

5. In the IAM dashboard, click **Users**. Select an existing IAM user or create a new one by clicking **Add users**.
6. If you are using an _existing_ IAM user, click the **Add permissions** dropdown menu and select **Add permissions**. If you are creating a _new_ user, you will be taken to the Permissions screen after selecting a name.
7. Select **Attach policies directly**, then find and check the box for your new policy. Click **Next**, then **Add permissions**.
8. After successfully creating your user, select the **Security credentials** tab and click **Create access key**. You will be prompted to select a use case and add optional tags to your access key. Click **Create access key** to generate the keys.

:::caution
Your `Secret Access Key` will only be visible once upon creation. Be sure to copy and store it securely for future use.
:::
Expand All @@ -130,9 +71,7 @@ For more information on managing your access keys, please refer to the
3. Give a **Name** to the stream
4. (Optional) - If you want to enforce a specific schema, you can enter a **Input schema**. By default, this value is set to `{}` and will automatically infer the schema from the file\(s\) you are replicating. For details on providing a custom schema, refer to the [User Schema section](#user-schema).
5. Optionally, enter the **Globs** which dictates which files to be synced. This is a regular expression that allows Airbyte to pattern match the specific files to replicate. If you are replicating all the files within your bucket, use `**` as the pattern. For more precise pattern matching options, refer to the [Path Patterns section](#path-patterns) below.
6. **To authenticate your private bucket**:
- If using an IAM role, enter the **AWS Role ARN**.
- If using IAM user credentials, fill the **AWS Access Key ID** and **AWS Secret Access Key** fields with the appropriate credentials.
6. **If you are syncing from a private bucket**, you must fill the **AWS Access Key ID** and **AWS Secret Access Key** fields with the appropriate credentials to authenticate the connection. All other fields are optional and can be left empty. Refer to the [S3 Provider Settings section](#s3-provider-settings) below for more information on each field.

All other fields are optional and can be left empty. Refer to the [S3 Provider Settings section](#s3-provider-settings) below for more information on each field.

Expand Down
Loading