Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions api-reference/workflow/destinations/delta-table.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import DeltaTablePrerequisites from '/snippets/general-shared-text/delta-table.m

<DeltaTablePrerequisites />

## Create the destination connector

To create a Delta Tables in Amazon S3 destination connector, see the following examples.

import DeltaTableSDK from '/snippets/destination_connectors/delta_table_sdk.mdx';
Expand Down
24 changes: 14 additions & 10 deletions snippets/general-shared-text/delta-table.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ For more information about requirements, see the following:
allowfullscreen
></iframe>

- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
Additional approaches are in the following video and in the how-to sections at the end of this page.
- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

<iframe
width="560"
Expand All @@ -39,8 +39,15 @@ For more information about requirements, see the following:
allowfullscreen
></iframe>

- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).
- For authenticated bucket write access or both, you should first
[block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions,
attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

<iframe
width="560"
height="315"
Expand All @@ -51,11 +58,8 @@ For more information about requirements, see the following:
allowfullscreen
></iframe>

- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
[Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).
- Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account.
Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

<iframe
width="560"
Expand All @@ -67,10 +71,10 @@ For more information about requirements, see the following:
allowfullscreen
></iframe>

- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
- If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).
- If the target files are in a folder, make sure the authenticated AWS IAM user has
authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

Expand Down
111 changes: 53 additions & 58 deletions snippets/general-shared-text/s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ allowfullscreen
></iframe>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

- An AWS account. [Create an AWS account](https://aws.amazon.com/free).
Expand All @@ -26,8 +26,8 @@ requirements. For more information about requirements, see the following:
allowfullscreen
></iframe>

- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
Additional approaches are in the following video and in the how-to sections at the end of this page.
- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

<iframe
width="560"
Expand All @@ -39,61 +39,44 @@ requirements. For more information about requirements, see the following:
allowfullscreen
></iframe>

- Anonymous (supported but not recommended) or authenticated access to the bucket.
- Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
[in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

- [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).
![Enable anonymous bucket access](/img/connectors/s3-anon-access.gif)

![Enable anonymous bucket access](/img/connectors/s3-anon-access.gif)
- For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
[block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

- [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

![Enable authenticated bucket access](/img/connectors/s3-auth-access.gif)

- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/y4SfQoJpipo"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
[Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/MoFTaGJE65Q"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/4_csSXc_GNU"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>
After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
can be granted in one of the following ways:

- Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/y4SfQoJpipo"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
[create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
[have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

<Warning>
AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

<Warning>
AWS STS credentials can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
generated. After the expiry time, the credentials are no longer valid will no longer work with the corresponding S3 connector.
You must get a new set of credentials to replace the expired ones by calling
[GetSessionToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) in the AWS STS API.
To overwrite the expired credentials with the new set:

- For the Unstructured user interface (UI), manually update the AWS Key, AWS Secret Key, and STS Token fields in the Unstructured UI
Expand All @@ -105,10 +88,22 @@ requirements. For more information about requirements, see the following:
[destination](/api-reference/workflow/destinations/s3) connector.
- For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
</Warning>

</Warning>

- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
- If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/MoFTaGJE65Q"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).
- If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
2 changes: 2 additions & 0 deletions ui/destinations/delta-table.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import DeltaTablePrerequisites from '/snippets/general-shared-text/delta-table.m

<DeltaTablePrerequisites />

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
Expand Down