From 21a93c83557c711a5d74c4d5329d336474d275e3 Mon Sep 17 00:00:00 2001 From: Paul Cornell Date: Mon, 4 Aug 2025 13:11:53 -0700 Subject: [PATCH 1/2] S3 connectors and Delta Tables in S3 destination connector: simplify steps, how to attach access role --- .../workflow/destinations/delta-table.mdx | 2 + snippets/general-shared-text/delta-table.mdx | 24 ++-- snippets/general-shared-text/s3.mdx | 111 +++++++++--------- ui/destinations/delta-table.mdx | 2 + 4 files changed, 71 insertions(+), 68 deletions(-) diff --git a/api-reference/workflow/destinations/delta-table.mdx b/api-reference/workflow/destinations/delta-table.mdx index 51680074..954dbc63 100644 --- a/api-reference/workflow/destinations/delta-table.mdx +++ b/api-reference/workflow/destinations/delta-table.mdx @@ -20,6 +20,8 @@ import DeltaTablePrerequisites from '/snippets/general-shared-text/delta-table.m +## Create the destination connector + To create a Delta Tables in Amazon S3 destination connector, see the following examples. import DeltaTableSDK from '/snippets/destination_connectors/delta_table_sdk.mdx'; diff --git a/snippets/general-shared-text/delta-table.mdx b/snippets/general-shared-text/delta-table.mdx index bb2f7c7d..f6a67908 100644 --- a/snippets/general-shared-text/delta-table.mdx +++ b/snippets/general-shared-text/delta-table.mdx @@ -26,8 +26,8 @@ For more information about requirements, see the following: allowfullscreen > -- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). - Additional approaches are in the following video and in the how-to sections at the end of this page. +- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video. + Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page. -- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html). +- For authenticated bucket write access or both, you should first + [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html). + After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. + For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions, + attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket. + To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video. + Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page. + -- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at - minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html). - -- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account. - [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). +- Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account. + Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video. -- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`). +- If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`). If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`). - If the target files are in a folder, make sure the authenticated AWS IAM user has - authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders). + authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders). ## Add an access policy to an existing bucket diff --git a/snippets/general-shared-text/s3.mdx b/snippets/general-shared-text/s3.mdx index e402de82..a9f7fbc4 100644 --- a/snippets/general-shared-text/s3.mdx +++ b/snippets/general-shared-text/s3.mdx @@ -11,7 +11,7 @@ allowfullscreen > The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but -not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security +not recommended); or generate AWS STS temporary access credentials if required by your organization's security requirements. For more information about requirements, see the following: - An AWS account. [Create an AWS account](https://aws.amazon.com/free). @@ -26,8 +26,8 @@ requirements. For more information about requirements, see the following: allowfullscreen > -- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). - Additional approaches are in the following video and in the how-to sections at the end of this page. +- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video. + Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page. -- Anonymous (supported but not recommended) or authenticated access to the bucket. +- Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps + [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation. - - [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user). + ![Enable anonymous bucket access](/img/connectors/s3-anon-access.gif) - ![Enable anonymous bucket access](/img/connectors/s3-anon-access.gif) +- For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first + [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html). - - [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html). - - ![Enable authenticated bucket access](/img/connectors/s3-auth-access.gif) - -- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html). - - - -- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at - minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html). - -- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account. - [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). - - - -- For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken). - - + After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. + For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions + can be granted in one of the following ways: + + - Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket. + To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video. + Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page. + + + + - Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires. + Learn how to use the IAM console to [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) and then + [grant the user permission to assume the role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html). Finally, + [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces + a temporary AWS access key, AWS secret access key, and AWS STS session token. + + + AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially + generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector. + You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces + a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token. - - AWS STS credentials can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially - generated. After the expiry time, the credentials are no longer valid will no longer work with the corresponding S3 connector. - You must get a new set of credentials to replace the expired ones by calling - [GetSessionToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) in the AWS STS API. To overwrite the expired credentials with the new set: - For the Unstructured user interface (UI), manually update the AWS Key, AWS Secret Key, and STS Token fields in the Unstructured UI @@ -105,10 +88,22 @@ requirements. For more information about requirements, see the following: [destination](/api-reference/workflow/destinations/s3) connector. - For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector. + - - -- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`). +- If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account. + Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video. + + + +- If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`). If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`). - If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has - authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders). \ No newline at end of file + authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders). \ No newline at end of file diff --git a/ui/destinations/delta-table.mdx b/ui/destinations/delta-table.mdx index 0313f38b..888c88d0 100644 --- a/ui/destinations/delta-table.mdx +++ b/ui/destinations/delta-table.mdx @@ -20,6 +20,8 @@ import DeltaTablePrerequisites from '/snippets/general-shared-text/delta-table.m +## Create the destination connector + To create the destination connector: 1. On the sidebar, click **Connectors**. From 5e1f036fc904650d5d850e5ad6781d17bac0cad5 Mon Sep 17 00:00:00 2001 From: Paul Cornell Date: Mon, 4 Aug 2025 15:31:09 -0700 Subject: [PATCH 2/2] More clarification on what AssumeRole produces --- snippets/general-shared-text/s3.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/snippets/general-shared-text/s3.mdx b/snippets/general-shared-text/s3.mdx index a9f7fbc4..0f864273 100644 --- a/snippets/general-shared-text/s3.mdx +++ b/snippets/general-shared-text/s3.mdx @@ -66,10 +66,10 @@ requirements. For more information about requirements, see the following: > - Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires. - Learn how to use the IAM console to [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) and then - [grant the user permission to assume the role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html). Finally, + Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html), + [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces - a temporary AWS access key, AWS secret access key, and AWS STS session token. + a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`). AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially