[Cherry-pick to branch-1.3] [#10990] feat(iceberg-rest): Vended credential refresh for S3, GCS, OSS, and ADLS (#11294) - #11449
Merged
Conversation
…SS, and ADLS (#11294) ### What changes were proposed in this pull request? This PR adds **Iceberg 1.11 vended credential refresh** for object storage in the Iceberg REST Catalog (IRC), and includes the **server-side Iceberg 1.11.0 upgrade** required for the feature. **Vended credential refresh (IRC)** - Add `IcebergRESTUtils.toRESTCredential(...)` to map Gravitino storage tokens to Iceberg 1.11 client properties, table-scoped refresh paths (`GET .../tables/{table}/credentials`), and storage `prefix` from table location. - Extend `CredentialPropertyUtils.toIcebergProperties` with token expiry metadata for S3/IRSA, OSS, and ADLS (GCS expiry already supported). - Wire credential refresh into IRC load-table, credentials API, and scan-plan responses in `CatalogWrapperForREST`. - For `catalog-backend=rest`, proxy `/credentials` to the upstream IRC via `getRESTTableCredentials()` (uses `UserPrincipalForwardingAuthManager` from `RESTCatalog` properties); other backends use local `getLocalTableCredentials()` via `catalogCredentialManager`. **Iceberg 1.11 upgrade (included in this branch)** - Upgrade server-side Iceberg to **1.11.0** for IRC and lakehouse catalog. - Adapt IRC REST handlers, plan-scan response building, and related Spark/Flink IT classpath expectations. ### Why are the changes needed? Iceberg 1.11 clients can refresh vended object-storage credentials before they expire during long-running jobs. Gravitino IRC must return expiry fields and table-scoped refresh endpoints for S3, GCS, OSS, and ADLS when temporary credentials are enabled. When IRC proxies another IRC (`catalog-backend=rest`), `/credentials` must be fetched from the upstream catalog instead of being generated locally on the proxy. The server must run Iceberg 1.11 to align with the REST spec and client refresh behavior. Fix: #10990 ### Does this PR introduce _any_ user-facing change? - When `vended-credentials` is enabled, load-table / credentials / scan-plan responses may include cloud-specific expiry fields and refresh-endpoint properties (for example `client.refresh-credentials-endpoint`, `gcs.oauth2.refresh-credentials-endpoint`, `adls.refresh-credentials-endpoint`). - Refresh paths use the relative form without a leading slash: `v1/{catalog}/namespaces/.../tables/.../credentials`. - IRC and lakehouse Iceberg catalog documentation reflect **Iceberg 1.11.0** server support. ### How was this patch tested? - [x] `./gradlew :common:test --tests org.apache.gravitino.credential.TestCredentialPropertiesUtils -PskipITs` - [x] `./gradlew :iceberg:iceberg-rest-server:test --tests org.apache.gravitino.iceberg.service.TestIcebergRESTUtils --tests org.apache.gravitino.iceberg.service.TestCatalogWrapperForREST -PskipITs` - [x] `./gradlew :iceberg:iceberg-rest-server:compileJava :iceberg:iceberg-rest-server:compileTestJava -PskipITs`
Author
Code Coverage Report
Files
|
mchades
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick Information:
branch-1.3