Skip to content

Add team_name to Multi-Team metrics #68108

Open
ferruzzi wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ferruzzi/multi-team/metrics
Open

Add team_name to Multi-Team metrics #68108
ferruzzi wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ferruzzi/multi-team/metrics

Conversation

@ferruzzi
Copy link
Copy Markdown
Contributor

@ferruzzi ferruzzi commented Jun 5, 2026

Add team_name to ~35 metrics (DagRun, TaskInstance, pool slots) for multi-team Airflow deployments. When multi_team is disabled or a DAG has no team, the tag is omitted entirely (no change to existing systems).

Changes:

  • Add team_name to DagRun.stats_tags and TaskInstance.stats_tags
  • Set _team_name on DagRun/TI objects in 4 scheduler injection points
  • Add persistent _dag_id_to_team_name cache on SchedulerJobRunner
  • Add stats_tags property to RuntimeTaskInstance (Task SDK)
  • Add team_name to pool slot gauge metrics via pool.get_name_to_team_name_mapping

Remaining metrics will be done in (6 or 7?) parallel chunks after this lands.

Co-authored-by: Niko Oliveira onikolas@amazon.com


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Add `team_name` to ~35 metrics (DagRun, TaskInstance, pool slots) for multi-team Airflow deployments. When multi_team is disabled or a DAG has no team, the tag is omitted entirely (no change to existing systems).

Changes:
- Add `team_name` to `DagRun.stats_tags` and `TaskInstance.stats_tags`
- Set `_team_name` on `DagRun`/`TI` objects in 4 scheduler injection points
- Add persistent `_dag_id_to_team_name` cache on `SchedulerJobRunner`
- Add `stats_tags` property to `RuntimeTaskInstance` (Task SDK)
- Add `team_name` to pool slot gauge metrics via `pool.get_name_to_team_name_mapping`

Co-authored-by: Niko Oliveira <onikolas@amazon.com>
@boring-cyborg boring-cyborg Bot added area:Scheduler including HA (high availability) scheduler area:task-sdk labels Jun 5, 2026
@ferruzzi ferruzzi requested review from o-nikolas and vincbeck June 5, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant