Search before asking
Description
Today, an S3-type storage vault with provider = GCP can only authenticate with static HMAC keys (s3.access_key / s3.secret_key). The other credential mechanisms available to vaults (s3.role_arn, s3.credentials_provider_type) are AWS-specific, so on GCP there is currently no way to run a compute-storage decoupled cluster without minting and distributing long-lived HMAC key pairs.
For deployments on GKE this is a real gap: the platform-native (and increasingly organization-mandated) way to authorize workloads is Workload Identity / Application Default Credentials, where pods obtain short-lived OAuth2 tokens from the metadata server and no static secret exists at all. AWS deployments already enjoy the equivalent via WEB_IDENTITY/instance-profile credentials (thanks to the great work in #62788 and #64766) — it would be lovely for GCP users to have parity.
The GCS XML API (which Doris already uses for GCP vaults through the S3-compatible endpoint) accepts Authorization: Bearer <oauth2-token> as a first-class alternative to HMAC signatures, so this doesn't require a new SDK dependency — "only" a way to resolve ADC tokens and attach them to vault I/O requests across FE/BE/meta-service.
Use case
CREATE STORAGE VAULT on a GKE cluster with Workload Identity, with no static credential anywhere:
CREATE STORAGE VAULT IF NOT EXISTS gcs_vault
PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);Benefits: no key minting/distribution/rotation, tokens are short-lived, and access control lives entirely in IAM bindings on the bucket — consistent with the keyless direction the vault credential work has been taking on AWS.
Related issues
#62987 added Google authentication for Iceberg REST catalogs (FE-side); this proposes the analogous capability for storage vaults (data path).
I have a working implementation following the pattern of #64766 (new CredProviderTypePB value, shared token provider built on the AWS SDK's HTTP/JSON utilities, bearer header on the S3 client with anonymous credentials) and will open a PR shortly — very happy to adjust the approach based on maintainer feedback.
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
Today, an S3-type storage vault with
provider = GCPcan only authenticate with static HMAC keys (s3.access_key/s3.secret_key). The other credential mechanisms available to vaults (s3.role_arn,s3.credentials_provider_type) are AWS-specific, so on GCP there is currently no way to run a compute-storage decoupled cluster without minting and distributing long-lived HMAC key pairs.For deployments on GKE this is a real gap: the platform-native (and increasingly organization-mandated) way to authorize workloads is Workload Identity / Application Default Credentials, where pods obtain short-lived OAuth2 tokens from the metadata server and no static secret exists at all. AWS deployments already enjoy the equivalent via
WEB_IDENTITY/instance-profile credentials (thanks to the great work in #62788 and #64766) — it would be lovely for GCP users to have parity.The GCS XML API (which Doris already uses for GCP vaults through the S3-compatible endpoint) accepts
Authorization: Bearer <oauth2-token>as a first-class alternative to HMAC signatures, so this doesn't require a new SDK dependency — "only" a way to resolve ADC tokens and attach them to vault I/O requests across FE/BE/meta-service.Use case
CREATE STORAGE VAULTon a GKE cluster with Workload Identity, with no static credential anywhere:CREATE STORAGE VAULT IF NOT EXISTS gcs_vault PROPERTIES ( "type" = "S3", "s3.endpoint" = "storage.googleapis.com", "s3.region" = "us-central1", "s3.bucket" = "my-bucket", "s3.root.path" = "doris", "provider" = "GCP", "s3.credentials_provider_type" = "gcp_adc" );Benefits: no key minting/distribution/rotation, tokens are short-lived, and access control lives entirely in IAM bindings on the bucket — consistent with the keyless direction the vault credential work has been taking on AWS.
Related issues
#62987 added Google authentication for Iceberg REST catalogs (FE-side); this proposes the analogous capability for storage vaults (data path).
I have a working implementation following the pattern of #64766 (new
CredProviderTypePBvalue, shared token provider built on the AWS SDK's HTTP/JSON utilities, bearer header on the S3 client with anonymous credentials) and will open a PR shortly — very happy to adjust the approach based on maintainer feedback.Are you willing to submit PR?
Code of Conduct