Skip to content

Update SecretsManagerBackend to support multi-team #65372

@vincbeck

Description

@vincbeck

Description

In multi-team mode, Airflow passes team_name to the secrets backend when retrieving connections and variables (via get_conn_value, get_variable, and get_connection in BaseSecretsBackend). This allows backends to scope secrets per team — for example, by looking up secrets under a team-specific path prefix.

Currently, SecretsManagerBackend accepts team_name in its get_conn_value and get_variable signatures but completely ignores it. The team_name is never forwarded to _get_secret, so all teams resolve secrets from the same path regardless of team context.

Expected behavior

When team_name is provided, SecretsManagerBackend should incorporate it into the secret lookup path. For example, with connections_prefix="airflow/connections" and team_name="analytics", the backend could look up airflow/connections/analytics/smtp_default instead of airflow/connections/smtp_default.

The exact path convention is an implementation decision, but a reasonable approach would be to insert the team name between the prefix and the secret ID:

{prefix}/{team_name}/{secret_id}

When team_name is None (single-team mode or no team context), the behavior should remain unchanged ({prefix}/{secret_id}).

Current behavior

team_name is accepted in the method signatures but never used:

def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None:
    # team_name is ignored
    secret = self._get_secret(self.connections_prefix, conn_id, self.connections_lookup_pattern)
    ...

def get_variable(self, key: str, team_name: str | None = None) -> str | None:
    # team_name is ignored
    return self._get_secret(self.variables_prefix, key, self.variables_lookup_pattern)

What needs to change

  1. Pass team_name to _get_secretget_conn_value and get_variable should forward team_name to _get_secret.

  2. Incorporate team_name into the secret path_get_secret (or build_path) should use team_name to construct a team-scoped path when provided.

  3. Update tests to cover team-scoped secret lookups.

Files likely to be modified

  • providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py
  • providers/amazon/tests/unit/amazon/aws/secrets/test_secrets_manager.py

Note

The same gap exists in SystemsManagerParameterStoreBackend (providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py) and in other provider secrets backends (Google Secret Manager, Azure Key Vault, HashiCorp Vault, Yandex Lockbox, Kubernetes). Those could be addressed in follow-up issues.

Use case

Operators running Airflow in multi-team mode with AWS Secrets Manager cannot isolate secrets per team. All teams currently share the same secret namespace, which prevents proper secret scoping in multi-team deployments.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions