[C++] Support AWS S3 Web identity credentials

[asfimport]

It seems to me that Arrow only supports at the moment the "AssumeRole" AWS STS API, but not the other options offered:

• https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison

• https://sdk.amazonaws.com/cpp/api/LATEST/class_aws_1_1_auth_1_1_s_t_s_assume_role_web_identity_credentials_provider.html

  I am clearly no security/infra expert, but it seems that the configuration "AssumeRoleWithWebIdentity" is used commonly in Kubernetes setups, and I believe it would be beneficial for Arrow C++ & Python library to support.

  At the moment, a workaround is to call directly aws sts to generate a temporary session, but it is a fairly painful solution as the session expires, all PyArrow objects with an S3 filesystem (datasets, ...) need to be re-built with new credentials. 

Reporter: Paul Balanca / @balancap
Assignee: Sahil Gupta / @sahil1105

Related issues:

• [Python] Support AWS S3 Web identity credentials (is required by)

PRs and other links:

• GitHub Pull Request #10088
• GitHub Pull Request #8977

_{Note: This issue was originally created as ARROW-10675. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Thank you for the report. Would you like to provide a pull request, at least on the C++ side?

[asfimport]

Paul Balanca / @balancap:
I should be able to draft something next week. I may need a bit of guidance on the testing strategy you're adopting in the project of this kind of authentication code.

[asfimport]

Antoine Pitrou / @pitrou:
Feel free to ask any questions. For testing, though, my intuition is that we can't do much for integration testing of such a feature, at least not with our current CI.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 10088
#10088

[asfimport]

Tobias Ullrich:
Hi @pitrou, 

we just tried to use this (as we need it in a k8s setup) and it does not work in pyarrow. Looking at the code and both PRs, it seems  like PR #10088 is actually not adding the capabilities in Python. PR #8977 seems to would have added this.

Please let me know if I am missing something here.

Thank you.

-Tobias

[asfimport]

Antoine Pitrou / @pitrou:
[~tobias_ullrich] Sorry, you are right. Python support was deferred to a later JIRA/PR and this was forgotten when closing this JIRA. Will open another one.

[asfimport]

Antoine Pitrou / @pitrou:
I opened ARROW-16457 for the Python side.