Repository navigation
✨ Source S3: Add IAM Role Authentication - #33818
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
Warning 🚨 Connector code freeze is in effect until 2024-01-02. This PR is changing connector code. Please contact the current OC engineers if you want to merge this change to master. |
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
|
|
||
| return self._s3_client | ||
|
|
||
| def _get_iam_s3_client(self, client_kv_args: dict) -> BaseClient: |
There was a problem hiding this comment.
This is a nice solution. Can you link any documentation you used for this? I found this article that gave a nice description: https://medium.com/@li.chastina/auto-refresh-aws-tokens-using-iam-role-and-boto3-afd3c52fd8c7
There was a problem hiding this comment.
I also used the same Medium article but didn't find any additional official documentation to link except for the source code, so I will not add any links.
There was a problem hiding this comment.
Can you add docstring for this method.
There was a problem hiding this comment.
Added docstring
| { | ||
| "Effect": "Allow", | ||
| "Principal": { | ||
| "AWS": "arn:aws:iam::{airbyte-aws-account-id}:user/{your-username}" |
There was a problem hiding this comment.
This should say: arn:aws:iam::094410056844:user/delegated_access_user
014b7e1 to
e0d86b7
Compare
| "role_arn": { | ||
| "title": "AWS Role ARN (External ID is '8eee93517501b43a')", | ||
| "description": "Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set External ID as '8eee93517501b43a'.", | ||
| "airbyte_secret": true, |
There was a problem hiding this comment.
Spoke about this privately, but I am in favor of not setting this to secret. This value is not sensitive and I think it is helpful to be able to see it. Very open to push back here though.
There was a problem hiding this comment.
I agree with you, as role arn cannot be used outside the users' Airbyte workspace, it should not be secret. I fixed it in the latest commit.
d5d849d to
68e2c5e
Compare
Serhii Lazebnyi (lazebnyi)
left a comment
There was a problem hiding this comment.
Nice job!
Need add small description to docstring.
|
|
||
| return self._s3_client | ||
|
|
||
| def _get_iam_s3_client(self, client_kv_args: dict) -> BaseClient: |
There was a problem hiding this comment.
Can you add docstring for this method.
What
Enhance the S3 connector source to support IAM role-based authentication, allowing users to utilize IAM roles for more granular control over permissions and to eliminate the need for managing static access keys. Implements: #31865
How
Boto3 class
RefreshableCredentialsis used to refresh credentials automatically during the sync.