-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-10675: [C++][Python] Support AWS S3 web identity credentials. #8977
ARROW-10675: [C++][Python] Support AWS S3 web identity credentials. #8977
Conversation
27a43e9
to
4cf3abd
Compare
4cf3abd
to
a168ca4
Compare
a168ca4
to
374f47d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for posting this PR! The basic approach looks ok to me, just two comments below.
@@ -82,6 +82,8 @@ cdef class S3FileSystem(FileSystem): | |||
external_id: str, default None | |||
An optional unique identifier that might be required when you assume | |||
a role in another account. | |||
use_web_identity: bool, default False | |||
... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add the argument description here?
@@ -149,6 +157,8 @@ cdef class S3FileSystem(FileSystem): | |||
) | |||
elif anonymous: | |||
options = CS3Options.Anonymous() | |||
elif use_web_identity: | |||
options = CS3Options.FromAssumeRoleWithWebIdentity() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If use_web_identity
, anonymous
, role_arn
and (access_key
or secret_key
) are all exclusive, we should probably test for such cases and raise an error.
@pitrou Thanks for checking the MR! I'll push some modifications following your comments, and look more closely at the CI failures in the next couple of days. |
@balancap do you have time to update this? |
@jorgecarleitao @pitrou I am really sorry I did not take the time to update this PR. I'll get that done for the end of the week. |
No problem, take your time! (but a reminder can sometimes help ;)) |
d4608a9
to
356c300
Compare
@sahil1105 Perhaps you would be interested in this PR? |
In reviewing it? |
@sahil1105 Reviewing, and potentially suggesting updates (or submitting a fresh new PR) if you are motivated enough :-) |
Sure. I'll take a crack at it this weekend (or early next week) and submit a PR. |
Closing this, since it was superseded by #10088. |
Add support for AWS STS web identity.
The AWS SDK has made the strange choice to force users to pass web identity arguments through environment variables (see https://github.com/aws/aws-sdk-cpp/blob/2be13177875e944151132d90305dbc46e80bf8e3/aws-cpp-sdk-core/source/auth/STSCredentialsProvider.cpp), and does not provide a way of directly passing these values.
I am not sure the design choice made in this PR is so great, but I have failed to find a nicer way to integrate, considering the limitations of the AWS sdk.