Skip to content

✨ Source S3: Add IAM Role Authentication - #33818

Merged
Anatolii Yatsuk (tolik0) merged 10 commits into
masterfrom
tolik0/source-s3/add-irsa
Jan 17, 2024
Merged

Anatolii Yatsuk (tolik0) merged 10 commits into
masterfrom
tolik0/source-s3/add-irsa

Conversation

@tolik0

@tolik0 Anatolii Yatsuk (tolik0) commented Dec 27, 2023 •

Copy link
Copy Markdown
Contributor

What

Enhance the S3 connector source to support IAM role-based authentication, allowing users to utilize IAM roles for more granular control over permissions and to eliminate the need for managing static access keys. Implements: #31865

How

Boto3 class RefreshableCredentials is used to refresh credentials automatically during the sync.

@vercel

vercel Bot commented Dec 27, 2023 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 17, 2024 2:10pm

@github-actions

github-actions Bot commented Dec 27, 2023 •

Copy link
Copy Markdown
Contributor

Warning

🚨 Connector code freeze is in effect until 2024-01-02. This PR is changing connector code. Please contact the current OC engineers if you want to merge this change to master.

@github-actions

github-actions Bot commented Dec 27, 2023 •

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.


return self._s3_client

def _get_iam_s3_client(self, client_kv_args: dict) -> BaseClient:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice solution. Can you link any documentation you used for this? I found this article that gave a nice description: https://medium.com/@li.chastina/auto-refresh-aws-tokens-using-iam-role-and-boto3-afd3c52fd8c7

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also used the same Medium article but didn't find any additional official documentation to link except for the source code, so I will not add any links.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add docstring for this method.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring

Comment thread docs/integrations/sources/s3.md Outdated
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{airbyte-aws-account-id}:user/{your-username}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should say: arn:aws:iam::094410056844:user/delegated_access_user

"role_arn": {
"title": "AWS Role ARN (External ID is '8eee93517501b43a')",
"description": "Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set External ID as '8eee93517501b43a'.",
"airbyte_secret": true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spoke about this privately, but I am in favor of not setting this to secret. This value is not sensitive and I think it is helpful to be able to see it. Very open to push back here though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you, as role arn cannot be used outside the users' Airbyte workspace, it should not be secret. I fixed it in the latest commit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!

Need add small description to docstring.


return self._s3_client

def _get_iam_s3_client(self, client_kv_args: dict) -> BaseClient:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add docstring for this method.

This branch was successfully deployed

1 active deployment
Preview — dc17cfca Deployed Jan 17, 2024 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/s3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants