[feat](vault) Support keyless GCS storage vaults via Application Default Credentials#65433
Open
cyriltovena wants to merge 4 commits into
Open
[feat](vault) Support keyless GCS storage vaults via Application Default Credentials#65433cyriltovena wants to merge 4 commits into
cyriltovena wants to merge 4 commits into
Conversation
…ult Credentials
Storage vaults on GCS currently require static HMAC keys: the only
credential fields an S3-type vault accepts for provider GCP are
s3.access_key/s3.secret_key. Every other credential mode
(role_arn, s3.credentials_provider_type) is AWS-only, so clusters
running on GKE cannot use Workload Identity and must mint and
distribute long-lived HMAC key pairs.
This adds a new credentials provider type, GCP_ADC, valid only for
provider GCP vaults:
CREATE STORAGE VAULT gcs_vault PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);
The GCS XML API accepts OAuth2 bearer tokens as an alternative to
HMAC signatures, so no new SDK dependency is needed:
- A shared GcpAdcTokenProvider (common/cpp) resolves tokens following
the Application Default Credentials order: a service account json
pointed to by GOOGLE_APPLICATION_CREDENTIALS (JWT assertion signed
with OpenSSL, exchanged at the OAuth2 token endpoint), else the
GCE/GKE metadata server (i.e. Workload Identity). Tokens are cached
and refreshed ahead of expiry. Built on the AWS SDK's own HTTP/JSON
utilities.
- BE S3ObjStorageClient and recycler S3ObjClient attach the token as
an Authorization: Bearer header per request; the S3 client is
created with anonymous credentials so SigV4 signing is skipped
(mirroring how the existing keyless credential providers reuse the
provider seam added for vaults without role ARN).
- FE validates the property combination (provider GCP only, mutually
exclusive with ak/sk and role_arn) and skips its static-credential
connectivity probe; the BE probes the vault at startup instead,
like assumed-role vaults.
- Meta-service accepts keyless GCP vaults on ADD_S3_VAULT and
ALTER_S3_VAULT (switching an existing HMAC vault to GCP_ADC clears
the stored keys, symmetric with the ak/sk <-> role_arn switches).
Presigned URLs are not supported in this mode (a bearer token cannot
be embedded in a URL) and return an empty string with a warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cyriltovena
requested review from
Gabriel39,
gavinchou,
liaoxin01,
luwei16 and
morningman
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
/review |
Author
|
run buildall |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29324 ms |
Author
|
run feut |
Contributor
TPC-DS: Total hot run time: 180486 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
ClickBench: Total hot run time: 25.61 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29454 ms |
Contributor
TPC-DS: Total hot run time: 180054 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Author
|
run p0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65432
Related PR: #62788, #64766 (the vault credentials-provider plumbing this builds on), #62987 (Google auth for Iceberg REST catalogs, FE-side)
Problem Summary:
Storage vaults with
provider = GCPcan only authenticate with static HMAC keys today — the existing keyless mechanisms (s3.role_arn, thes3.credentials_provider_typevalues) are all AWS-semantic. On GKE, where workloads authenticate via Workload Identity / Application Default Credentials, this forces users to mint and distribute long-lived HMAC key pairs.This PR adds a
GCP_ADCcredentials provider type for GCP vaults. The GCS XML API acceptsAuthorization: Bearer <oauth2-token>as an alternative to HMAC signatures, so no new SDK dependency is introduced:CREATE STORAGE VAULT gcs_vault PROPERTIES ( "type" = "S3", "s3.endpoint" = "storage.googleapis.com", "s3.region" = "us-central1", "s3.bucket" = "my-bucket", "s3.root.path" = "doris", "provider" = "GCP", "s3.credentials_provider_type" = "gcp_adc" );Design (following the seams introduced by #64766 as closely as I could):
CredProviderTypePB::GCP_ADC = 9/TCredProviderType::GCP_ADC.GcpAdcTokenProviderresolves OAuth2 tokens in ADC order —GOOGLE_APPLICATION_CREDENTIALSservice account json (RS256 JWT assertion via OpenSSL, exchanged at the token endpoint), else the GCE/GKE metadata server (Workload Identity). Tokens are cached and refreshed ahead of expiry; built on the AWS SDK's own HTTP/JSON utilities so both BE and the meta-service recycler can use it.GCP_ADCthe S3 client is created withAnonymousAWSCredentialsProvider(SigV4 signing is skipped on empty credentials) andS3ObjStorageClient/S3ObjClientattach the bearer token per request viaSetAdditionalCustomHeaderValue. Re-applied per page in paginated listings so token rotation mid-listing is safe.s3.credentials_provider_type = "gcp_adc"is validated (provider GCP only; mutually exclusive with ak/sk and role_arn) and mapped inS3Properties.getObjStoreInfoPB. The FE connectivity probe is skipped for this mode (its Java client signs with static credentials); the BE probes the vault at startup instead, same as assumed-role vaults.ADD_S3_VAULTaccepts keyless GCP vaults withGCP_ADC;ALTER_S3_VAULTsupports switching an HMAC vault toGCP_ADC(clearing stored keys) and back, symmetric with the existing ak/sk ⇄ role_arn switches.Known limitation: presigned URLs are not supported in this mode (a short-lived bearer token cannot be embedded in a URL);
generate_presigned_urllogs a warning and returns an empty string.This is my first contribution to Doris, so I fully expect there are conventions I've missed — very happy to rework any part of the approach (naming, where the token provider lives, whether a native GCS client would be preferred longer-term) based on your feedback.
Release note
Support keyless GCS authentication for storage vaults via Google Application Default Credentials / Workload Identity (
"s3.credentials_provider_type" = "gcp_adc").Check List (For Author)
Test
gcp_adc) is planned on our side; I will report the results in this PR. Unit tests cover the FE property validation/proto mapping, the meta-service ADD/ALTER paths, and the BE credentials-provider selection.Behavior changed:
Does this need documentation?
gcp_adcvalue on the CREATE/ALTER STORAGE VAULT pages once the property surface is settled in review.Check List (For Reviewer who merge this PR)