[Bug] BE loses S3 access once the EKS Pod Identity token rotates; token file is never re-read

[Owen-CH-Leung]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master (also 4.1.x)

What's Wrong?

Under EKS Pod Identity, the BE cannot reach s3 once its credentials expire, because it cannot refresh them.

The CustomAwsCredentialsProviderChain only reads the credential endpoint's token from AWS_CONTAINER_AUTHORIZATION_TOKEN, once at construction. EKS Pod Identity supplies that token as a file stored in AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and rotates it, which the chain never reads.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html

So the BE keeps presenting the token it captured at startup. After first rotation, the agent will reject it and therefore every s3 request from BE fails.

What You Expected?

The BE reads AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and re-reads it on each refresh, so BE can hit s3 across token rotations.

How to Reproduce?

1. Run the BE in EKS with a Pod Identity association and no static credentials
2. Query a s3-backed catalog, and then wait for the token file to rotate
3. Subsequent queries will fail

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_fc3e45704055

Initial triage

Judgment: valid BE credential-provider compatibility bug; high confidence. The issue is open with no labels, assignee, milestone, or earlier comments, and the author is willing to submit a PR. The unsupported behavior is present at the current remote heads of both master (8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323) and branch-4.1 (e48fc7e97b98909199d27bfd3aba3924f5588c51).

Verified facts

• Both branches still vendor AWS SDK for C++ 1.11.219. AWS lists 1.11.263 as the minimum C++ SDK version for EKS Pod Identity.
• EKS Pod Identity injects AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE; the projected token file is refreshed by EKS. This is the documented Pod Identity environment and volume setup.
• Doris's CustomAwsCredentialsProviderChain recognizes the full URI but only reads AWS_CONTAINER_AUTHORIZATION_TOKEN, once while constructing TaskRoleCredentialsProvider. It never reads AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE.
• In SDK 1.11.219, TaskRoleCredentialsProvider::Reload() refreshes the returned temporary AWS credentials, but its ECSCredentialsClient retains the authorization token supplied at construction. Therefore a token copied into AWS_CONTAINER_AUTHORIZATION_TOKEN at BE startup becomes stale after the projected file rotates.
• Upstream SDK 1.11.263 added the intended mechanism: the default chain passes the token-file path into GeneralHTTPCredentialsProvider, and Reload() reads that file again before each credential-endpoint request.
• There is a second affected route: an explicitly selected CredProviderType::Container reads only AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. EKS supplies the full URI, so this route also needs coverage. The cloud recycler has the same construction pattern.

One detail of the reported timeline is not yet explained. With the standard EKS injection there is no raw AWS_CONTAINER_AUTHORIZATION_TOKEN, so the current code can fail on the first credential fetch, not only after rotation. The stated "works initially, then fails" sequence is consistent if an entrypoint or another component copied the initial file contents into the raw-token environment variable, or if another credential source was initially selected. This does not weaken the confirmed missing token-file support, but it should be clarified before treating the exact timeline as reproduced.

Information requested from the reporter

1. Exact Doris build/commit for each tested version, plus the BE value of aws_credentials_provider_version and the catalog's s3.credentials_provider_type (or confirmation that it is unset/default).
2. Presence and paths of the AWS_CONTAINER_* variables inside the BE container. Please redact the authorization token itself; only indicate whether the raw-token variable is present and how it is populated.
3. The exact first failure and surrounding BE credential-provider/S3 log lines, including timestamps relative to the credential expiry and projected-token rotation. Please redact credentials, account IDs, bucket names, and other sensitive values.
4. Confirmation that the first S3 query really used the Pod Identity role rather than a node role, profile, or another earlier provider in the chain.

These details are useful to reconcile the timing, but they are not blockers to fixing the verified incompatibility.

Recommended implementation and validation

• Prefer upgrading the C++ SDK to at least 1.11.263 (or a newer vetted 1.11.x release) and porting the upstream general/container HTTP provider logic into CustomAwsCredentialsProviderChain while preserving Doris's intentional provider ordering. An SDK bump alone is insufficient because the custom chain currently bypasses the new default-chain token-file logic.
• Apply the same helper/provider construction to explicit CredProviderType::Container paths in BE and cloud code. Audit other C++ container-credential call sites so behavior is consistent.
• Add a focused unit test with a loopback credential endpoint and a temporary authorization-token file: accept token A for the first short-lived credential response, replace the file with token B, force/await provider refresh, and verify the next request sends B and returns refreshed credentials. Also retain coverage for ECS relative URI and the direct AWS_CONTAINER_AUTHORIZATION_TOKEN form.
• After the master fix is verified, backport it to 4.1. Restarting the BE pod is only a temporary operational workaround because it reloads the initial token but does not make rotation safe.

Suggested triage: kind/fix plus the repository's BE/object-storage label, and assign the S3 credential-provider owner.