[FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers

[michal-kuzak]

Is your feature request related to a problem? Please describe.

Polaris currently supports S3 access via STS AssumeRole (role ARN + optional external ID). Please add first‑class support for STS AssumeRoleWithWebIdentity (ARWI). Many enterprise S3‑compatible object stores deployed on‑prem rely on OIDC federation and expose an STS endpoint that accepts ARWI, rather than operating a native IAM service. Examples include Pure Storage FlashBlade, or Dell S3 storage among others. Adding ARWI unlocks secure, short‑lived, IdP‑federated access to S3‑compatible stores without long‑lived keys.

On‑prem S3 providers often delegate authentication to external IdPs (Okta, Microsoft Entra ID, etc.) and expect clients to obtain credentials via AssumeRoleWithWebIdentity to their STS endpoint. Polaris today documents role‑arn/region/external‑id and custom S3/STS endpoints, but not ARWI. Supporting ARWI would let Polaris run in OIDC‑first environments (Kubernetes, corporate SSO) and still vend short‑lived credentials to engines.

Describe the solution you'd like

Add a storage auth selector for S3 catalogs, e.g. stsAuthMethod with values:

• assumeRole (current default/behavior)
• webIdentity (new)

When stsAuthMethod=webIdentity:

• Use AWS SDK’s ARWI credentials provider (e.g., StsAssumeRoleWithWebIdentity… in AWS SDK v2) against the configured sts-endpoint (already supported). (polaris.incubator.apache.org)
• Accept inputs via CLI and REST model:
  • role-arn (unchanged)
  • role-session-name (optional)
  • web-identity-token-file (path to OIDC/JWT file)
  • region (optional; also allow via env)
  • sts-endpoint (already documented)

Describe alternatives you've considered

No response

Additional context

Default remains stsAuthMethod=assumeRole; existing catalogs and clients are unaffected.
IMPLICIT mode continues to work; ARWI simply adds another supported path to obtain base credentials.

Ref:
https://support.purestorage.com/bundle/FlashBlade_Admin_Guide_4.6.4/resource/FlashBlade_Admin_Guide_4.6.4.pdf (p.462)

https://support.purestorage.com/bundle/m_purityfb_rest_api/page/FlashBlade/Purity_FB/PurityFB_REST_API/S3_Object_Store_REST_API/topics/concept/c_flashblade_object_store_documentation_s3_api.html

[dimas-b]

@michal-kuzak : Do you have capacity to make a PR for this feature? Contributions are welcome 😉

[cccs-cat001]

I just came to the same conclusion that we need this! I'd be willing to make a PR if someone can help with the design 😅 I've found right here is where we need to make this change: https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L107

But I'm not sure how we can go about getting the user token into this class. From what I can tell, the AwsCredentislStorageIntegration is an application scoped class, and we'd need to somehow give it the users request in there to be able to pass in the web token.

[dimas-b]

Prior art for getting the initial token is mostly around local files, AFAIK:

polaris/extensions/auth/opa/impl/src/main/java/org/apache/polaris/extension/auth/opa/token/FileBearerTokenProvider.java

Line 58 in 3882c03

public class FileBearerTokenProvider implements BearerTokenProvider {

If that works for you @cccs-cat001 , I suppose the class could be moved to a sharable module and reused for AwsCredentislStorageIntegration.

[dimas-b]

I suppose we could also support implicit (workload identity) authentication too, in case Polaris runs in EC2.

Cross-cloud AssumeRoleWithWebIdentity may be a bit too complicated for the initial PR, although it is certainly possible with some extra code. I'd propose to handle that as a follow-up.

[cccs-cat001]

I'm not sure I understand, how does that get a token from the user?

[dimas-b]

The user who deploys Polaris would have to provide the initial token in a file accessible to Polaris servers. It is going to be a "service" account effectively.

@cccs-cat001 : by "user" do you mean a user of the Polaris API? Do you mean that user's identity to carry forward to the S3 requests made as a result of the Polaris REST API request? That would be an interesting feature, doable, I think, but it would require a lot of preparatory work, I'm sure 😅

[dparent1]

Hi @dimas-b, appreciate your participation here, I'm working with @cccs-cat001 on this. Based on your last response:

Would that not mean that all access to the s3 buckets are essentially done by the service account? I'd like to pass the jwt token provided to polaris to the s3 appliance so it can allow / deny access based on the claims in the jwt token. It would also allow the s3 appliance to log which user is trying to interact with the buckets and the objects in the buckets. Would that be at all possible or would that be a big break from how polaris was designed to function?

[dimas-b]

@dparent1 :

The current state of the server codebase is that Polaris uses fixed base credentials for its own access to storage in all requests. It can use external IdP for authenticating API requests, but will switch to the single "service" credential for storage access.

I'd like to pass the jwt token provided to polaris to the s3 appliance so it can allow / deny access based on the claims in the jwt token.

I fully support implementing this use case... however, I do believe it will require a lot of code changes in Polaris 😅 just a fair warning 😉 Ultimately, this is about feeding the right input credentials into AwsCredentialsStorageIntegration.

I'd propose to move this discussion to the dev ML for visibility and because I believe email is a better tool than GH comments for design discussions :) As for me, I do not have a solid plan for this in my mind ATM, so I suppose it will have to be a collaborative and iterative process... as it's supposed to be at the ASF :)

[cccs-cat001]

I'll write that up :)