Skip to content

Fix error messages in PythonVirtualenvOperator when Azure Key Vault secret backend is configured#67157

Merged
jscheffl merged 6 commits into
apache:mainfrom
boschglobal:bugfix/fix-error-messages-in-pythonvirtualenvoperator-with-azure-key-vault-secret-backend
May 19, 2026
Merged

Fix error messages in PythonVirtualenvOperator when Azure Key Vault secret backend is configured#67157
jscheffl merged 6 commits into
apache:mainfrom
boschglobal:bugfix/fix-error-messages-in-pythonvirtualenvoperator-with-azure-key-vault-secret-backend

Conversation

@wolfdn
Copy link
Copy Markdown
Contributor

@wolfdn wolfdn commented May 19, 2026

Problem

The Azure Key Vault secrets backend raises an HttpResponseError (BadParameter) when Airflow requests a variable whose key contains a dot. Azure Key Vault secret names only allow alphanumeric characters and dashes ([a-zA-Z0-9-]), so dots in the variable key cause the API call to fail.

This affects:

  • Any user-defined variable with a dot in its name — e.g. a variable named my.app.setting will fail when looked up through this backend.
  • The PythonVirtualenvOperator specifically — it calls Variable.get("PythonVirtualenvOperator.cache_key", "") on every execution to compute its virtualenv cache hash. This means every DAG using PythonVirtualenvOperator produces an ERROR on every run when the Azure Key Vault backend is configured, even though the variable not existing is perfectly fine (it defaults to "").

The resulting error floods the logs:

ERROR - Unable to retrieve variable from secrets backend (AzureKeyVaultBackend).
HttpResponseError: (BadParameter) The request URI contains an invalid name: airflow-variables-PythonVirtualenvOperator.cache-key

Fix

  1. Rename the PythonVirtualenvOperator cache variable from PythonVirtualenvOperator.cache_key to python_virtualenv_operator_cache_key. This avoids the dot that is incompatible with secrets backends that restrict allowed characters (like Azure Key Vault). The underscore-only name is properly normalized by the existing build_path logic.

  2. Add a validation guard in the Azure Key Vault backend_get_secret_value now validates the constructed secret name against Azure Key Vault's naming rules (1-127 chars, alphanumeric and dashes only) before making the API call. Invalid names return None with a WARNING log, alerting the user that their variable cannot be resolved via Key Vault. This replaces the previous behavior of letting Azure throw an unhandled HttpResponseError at ERROR level.

Migration note

The cache hash computation for PythonVirtualenvOperator now reads from python_virtualenv_operator_cache_key instead of PythonVirtualenvOperator.cache_key. Since most users never set this variable (it's an optional manual cache-busting override), the only effect is a one-time rebuild of cached virtual environments due to the changed hash input.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    GitHub Copilot - Claude Opus 4.6

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/standard/docs/operators/python.rst
Comment thread providers/standard/docs/changelog.rst Outdated
@jscheffl jscheffl merged commit 5a0ff83 into apache:main May 19, 2026
276 of 277 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants