Skip to content

fix: cache CLI AWS credentials until expiry#23325

Open
ametel01 wants to merge 4 commits into
apache:mainfrom
ametel01:fix-cli-aws-credential-cache
Open

fix: cache CLI AWS credentials until expiry#23325
ametel01 wants to merge 4 commits into
apache:mainfrom
ametel01:fix-cli-aws-credential-cache

Conversation

@ametel01

@ametel01 ametel01 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The CLI's S3 credential provider wrapped AWS SDK credentials but called SharedCredentialsProvider::provide_credentials() for every object-store credential request. That bypasses the AWS SDK request pipeline identity cache and can repeatedly invoke expensive providers such as credential_process or SSO-backed providers even when the returned credentials have not expired.

What changes are included in this PR?

  • Cache the converted object_store::aws::AwsCredential inside S3CredentialProvider.
  • Seed that cache from the initial credential fetch already performed while building the S3 object store.
  • Reuse cached credentials until they are expired or within a small refresh buffer.
  • Add regression coverage for seeded credentials, non-expiring credentials, expired and near-expiry credentials, and an end-to-end credential_process flow that fails on main without this fix.

Are these changes tested?

Yes:

  • cargo test -p datafusion-cli s3_object_store_reuses_fetched_credentials_until_expiry -- --nocapture
  • cargo test -p datafusion-cli object_storage
  • cargo test -p datafusion-cli
  • cargo clippy --all-targets --all-features -- -D warnings

I also verified the new public-flow regression test against detached main; it fails there because the credential process runs multiple times instead of once.

Are there any user-facing changes?

No API or configuration changes. The CLI should make fewer redundant AWS credential-provider calls for S3 object stores while still refreshing expiring credentials.

@ametel01 ametel01 marked this pull request as ready for review July 5, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI utility does not check expiration of AWS credentials

1 participant