Skip to content

fix(metrics): emit queued_duration metric in execution API ti_run endpoint#63522

Draft
YoannAbriel wants to merge 1 commit intoapache:mainfrom
YoannAbriel:fix/issue-63503
Draft

fix(metrics): emit queued_duration metric in execution API ti_run endpoint#63522
YoannAbriel wants to merge 1 commit intoapache:mainfrom
YoannAbriel:fix/issue-63503

Conversation

@YoannAbriel
Copy link
Contributor

In Airflow 3, tasks transition from QUEUED to RUNNING via the execution API (ti_run endpoint), not the legacy TaskInstance._run_raw_task path. The task.queued_duration metric (dag.{dag_id}.{task_id}.queued_duration in legacy format) is only emitted in the old path via emit_state_change_metric(), so it's never recorded for tasks started through the task SDK.

This adds the metric emission to ti_run() — selects queued_dttm, end_date, and queue from the TI row and calls DualStatsManager.timing() after the state update. Same first-attempt guard as the original (skips when end_date is set).

Closes: #63503


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4, claude-opus-4-6)

Generated-by: Claude Code (Opus 4, claude-opus-4-6) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:task-sdk labels Mar 13, 2026
@potiuk potiuk marked this pull request as draft March 13, 2026 19:46
@potiuk
Copy link
Member

potiuk commented Mar 13, 2026

@YoannAbriel This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Other failing CI checks: Failing: Postgres tests: core / DB-core:Postgres:14:3.10:API...Serialization, MySQL tests: core / DB-core:MySQL:8.0:3.10:API...Serialization, Sqlite tests: core / DB-core:Sqlite:3.10:API...Serialization, Special tests / Min SQLAlchemy test: core / DB-core:MinSQLAlchemy-Postgres:14:3.10:API...Serialization, Special tests / Latest SQLAlchemy test: core / DB-core:LatestSQLAlchemy-Postgres:14:3.10:API...Serialization (+3 more). Run prek run --from-ref main locally to reproduce. See static checks docs.

Note: Your branch is 1 commit behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

…point

The task.queued_duration metric was not being emitted when tasks
transitioned from QUEUED to RUNNING via the execution API (used by
the task SDK in Airflow 3). The metric was only emitted in the legacy
TaskInstance._run_raw_task path.

Add queued_dttm, end_date, and queue to the ti_run select query and
emit the metric after the state update, matching the behavior in
TaskInstance.emit_state_change_metric.

Closes: apache#63503
@YoannAbriel YoannAbriel marked this pull request as ready for review March 15, 2026 22:03
@potiuk
Copy link
Member

potiuk commented Mar 16, 2026

@YoannAbriel This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Other failing CI checks: Failing: Postgres tests: core / DB-core:Postgres:14:3.10:API...Serialization, MySQL tests: core / DB-core:MySQL:8.0:3.10:API...Serialization, Sqlite tests: core / DB-core:Sqlite:3.10:API...Serialization, Special tests / Min SQLAlchemy test: core / DB-core:MinSQLAlchemy-Postgres:14:3.10:API...Serialization, Special tests / Latest SQLAlchemy test: core / DB-core:LatestSQLAlchemy-Postgres:14:3.10:API...Serialization (+2 more). Run prek run --from-ref main locally to reproduce. See static checks docs.

Note: Your branch is 11 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
  • Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

@potiuk potiuk marked this pull request as draft March 16, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metric dag.dag_id.task_id.queued_duration missing

2 participants