ARROW-10675: [C++][Python] Support AWS S3 web identity credentials. #8977
Conversation
balancap
force-pushed
the
ARROW-10675-support-aws-s3-sts-web-identity-credentials
branch
from
27a43e9
to
4cf3abd
Compare
balancap
force-pushed
the
ARROW-10675-support-aws-s3-sts-web-identity-credentials
branch
from
4cf3abd
to
a168ca4
Compare
balancap
force-pushed
the
ARROW-10675-support-aws-s3-sts-web-identity-credentials
branch
from
a168ca4
to
374f47d
Compare
| @@ -82,6 +82,8 @@ cdef class S3FileSystem(FileSystem): | |||
| external_id: str, default None | |||
| An optional unique identifier that might be required when you assume | |||
| a role in another account. | |||
| use_web_identity: bool, default False | |||
| ... | |||
There was a problem hiding this comment.
Can you add the argument description here?
| @@ -149,6 +157,8 @@ cdef class S3FileSystem(FileSystem): | |||
| ) | |||
| elif anonymous: | |||
| options = CS3Options.Anonymous() | |||
| elif use_web_identity: | |||
| options = CS3Options.FromAssumeRoleWithWebIdentity() | |||
There was a problem hiding this comment.
If use_web_identity, anonymous, role_arn and (access_key or secret_key) are all exclusive, we should probably test for such cases and raise an error.
|
@pitrou Thanks for checking the MR! I'll push some modifications following your comments, and look more closely at the CI failures in the next couple of days. |
|
@balancap do you have time to update this? |
|
@jorgecarleitao @pitrou I am really sorry I did not take the time to update this PR. I'll get that done for the end of the week. |
|
No problem, take your time! (but a reminder can sometimes help ;)) |
jorgecarleitao
force-pushed
the
master
branch
from
d4608a9
to
356c300
Compare
|
@sahil1105 Perhaps you would be interested in this PR? |
In reviewing it? |
|
@sahil1105 Reviewing, and potentially suggesting updates (or submitting a fresh new PR) if you are motivated enough :-) |
Sure. I'll take a crack at it this weekend (or early next week) and submit a PR. |
|
Closing this, since it was superseded by #10088. |
Add support for AWS STS web identity.
The AWS SDK has made the strange choice to force users to pass web identity arguments through environment variables (see https://github.com/aws/aws-sdk-cpp/blob/2be13177875e944151132d90305dbc46e80bf8e3/aws-cpp-sdk-core/source/auth/STSCredentialsProvider.cpp), and does not provide a way of directly passing these values.
I am not sure the design choice made in this PR is so great, but I have failed to find a nicer way to integrate, considering the limitations of the AWS sdk.