Map ProfileCredentialsProvider to profile credential chain

[karuppayya]

Which issue does this PR close?

Closes #4212

Rationale for this change

Support AWS Profile Credential provider

What changes are included in this PR?

Support AWS Profile Credential provider

How are these changes tested?

Units tests + on a real AWS cluster setup

[karuppayya]

cc @mbutrovich @parthchandra @andygrove
Can any you of you help kick-off CI

[andygrove]

Thanks @karuppayya. Could you also update docs/source/user-guide/latest/datasources.md section Supported Credential Providers.

[andygrove]

Current test failures are unrelated to this PR.

[parthchandra]

Will this support specifying fs.s3a.profile.name ?

[karuppayya]

@parthchandra Did some investigation on this before this change, No this cannot support the param.
org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider has both fs.s3a.auth.profile.name and fs.s3a.auth.profile.file
The above ws introduced in Hadoop 3.4.2

Spark version	Default hadoop.version
3.5.0 – 3.5.6	3.3.4
4.0.0 / 4.0.1	3.4.1
master (future release)	3.5.0

4.1.0 is the first version which might be affected?
While this is default hadoop version compiled against, user could potentially use their own hadoop libs.

[andygrove]

4.0.2 failures are unrelated and due to flaky test - seems to be happening a lot - #4221

[andygrove]

LGTM. Thanks @karuppayya!

[parthchandra]

@parthchandra Did some investigation on this before this change, No this cannot support the param. org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider has both fs.s3a.auth.profile.name and fs.s3a.auth.profile.file The above ws introduced in Hadoop 3.4.2

Spark version Default hadoop.version
3.5.0 – 3.5.6 3.3.4
4.0.0 / 4.0.1 3.4.1
master (future release) 3.5.0
4.1.0 is the first version which might be affected? While this is default hadoop version compiled against, user could potentially use their own hadoop libs.

The native side should be independent of the hadoop versions. The fs.s3a.* configs are already forwarded to the native side and in s3.rs get_config_trimmed(configs, bucket, "auth.profile.name") will get you the value of the config. This can be passed in to ProfileFileCredentialsProvider::builder()

Maybe you can add that in a follow up?

[karuppayya]

The native side should be independent of the hadoop versions

Makes sense and also would be consistent with other providers in AssumedRoleCredentialProvider, SimpleAWSCredentialsProvider.
Will do a follow up after this.