Skip to content

[Hashicorp] Fix Vault GCP auth for GCE metadata credentials#68069

Open
fpopic wants to merge 1 commit into
apache:mainfrom
fpopic:fix_vault_gcp_auth_requirements
Open

[Hashicorp] Fix Vault GCP auth for GCE metadata credentials#68069
fpopic wants to merge 1 commit into
apache:mainfrom
fpopic:fix_vault_gcp_auth_requirements

Conversation

@fpopic
Copy link
Copy Markdown
Contributor

@fpopic fpopic commented Jun 5, 2026

What

Fix HashiCorp Vault GCP authentication when Application Default Credentials come from Compute Engine metadata credentials and initially expose the service account email as default.

Why

In GCE-based environments such as Cloud Composer, google.auth.compute_engine.Credentials may start with service_account_email == "default" until the credentials are refreshed from the metadata server. The Vault client used that value directly in the IAM signJwt request, producing an invalid resource name like projects/<project>/serviceAccounts/default.

Sanitized task log excerpt:

{connection.py:531} ERROR - Unable to retrieve connection from secrets backend (VaultBackend). Checking subsequent secrets backend.
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/<project-id>/serviceAccounts/default:signJwt?alt=json returned "Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account". Details: "Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account">
airflow.exceptions.AirflowNotFoundException: The conn_id `<conn-id>` isn't defined

Context

How

  • Resolve GCP service account email before building the IAM signJwt request.
  • Treat missing or default service account email values as unresolved.
  • Refresh Compute Engine credentials so the metadata server populates the real service account email.
  • Preserve key-file behavior by using client_email when available.
  • Add a regression test for the Compute Engine/Composer ADC case.

Tests

.venv/bin/python -m pytest providers/hashicorp/tests/unit/hashicorp/_internal_client/test_vault_client.py -q --with-db-init

AI assistance

This PR was prepared with help from GPT-5 / Codex.

@fpopic fpopic requested a review from hussein-awala as a code owner June 5, 2026 09:14
@boring-cyborg boring-cyborg Bot added area:providers provider:hashicorp Hashicorp provider related issues labels Jun 5, 2026
@fpopic fpopic force-pushed the fix_vault_gcp_auth_requirements branch from fddeb58 to ac64fd6 Compare June 5, 2026 09:25
@fpopic fpopic marked this pull request as draft June 5, 2026 09:39
@fpopic fpopic force-pushed the fix_vault_gcp_auth_requirements branch from ac64fd6 to 8513abb Compare June 5, 2026 09:47
@fpopic fpopic marked this pull request as ready for review June 5, 2026 09:54
@fpopic
Copy link
Copy Markdown
Contributor Author

fpopic commented Jun 5, 2026

@potiuk it is ready for review, once released as rc2 will do real intergration test.

@fpopic fpopic changed the title [Hashicorp] Fix Vault GCP auth for Compute Engine metadata credentials [Hashicorp] Fix Vault GCP auth for GCE metadata credentials Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:hashicorp Hashicorp provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants