Describe the bug
Comet's native S3 object store does not map profile credential provider class names declared to Rust AWS profile credential providers. It currently handles static keys, IMDS, environment variables, web identity, assumed role, and anonymous, but does not recognize AWS's ProfileCredentialsProvider.
Steps to reproduce
Set spark.hadoop.fs.s3a.aws.credentials.provider to com.amazonaws.auth.profile.ProfileCredentialsProvider, it throws
Unsupported credential provider: com.amazonaws.auth.profile.ProfileCredentialsProvider
Expected behavior
Should map the configured Profile Credential Provider to corresponding AWS SDK Rust impl, and also profiles to be used
Additional context
No response
Describe the bug
Comet's native S3 object store does not map profile credential provider class names declared to Rust AWS profile credential providers. It currently handles static keys, IMDS, environment variables, web identity, assumed role, and anonymous, but does not recognize AWS's ProfileCredentialsProvider.
Steps to reproduce
Set
spark.hadoop.fs.s3a.aws.credentials.providertocom.amazonaws.auth.profile.ProfileCredentialsProvider, it throwsExpected behavior
Should map the configured Profile Credential Provider to corresponding AWS SDK Rust impl, and also profiles to be used
Additional context
No response