Skip to content

Conversation

@xBis7
Copy link
Contributor

@xBis7 xBis7 commented Feb 10, 2026

While running some manual tests with the metrics enabled, I noticed that the scheduler started exiting after triggering a few dags.

This is the stacktrace that I'm seeing in the logs

2026-02-10T10:21:07.655259Z [error    ] Exception when executing SchedulerJob._run_scheduler_loop
p [airflow.jobs.scheduler_job_runner.SchedulerJobRunner] loc=scheduler_job_runner.py:1324
Traceback (most recent call last):
  File "/opt/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1319, in _execute
    self._run_scheduler_loop()
  File "/opt/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1666, in _run_scheduler_loop
    next_event = timers.run(blocking=False)
  File "/usr/python/lib/python3.10/sched.py", line 151, in run
    action(*argument, **kwargs)
  File "/opt/airflow/airflow-core/src/airflow/utils/event_scheduler.py", line 40, in repeat
    action(*args, **kwargs)
  File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)  # type: ignore[arg-type]
  File "/opt/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 2647, in _emit_ti_metrics
    extra_tags={"queue": queue, "dag_id": dag_id, "task_id": task_id},
UnboundLocalError: local variable 'queue' referenced before assignment
2026-02-10T10:21:07.699406Z [info     ] Shutting down LocalExecutor; waiting for running tasks t

This is actually my fault because in a previous PR of mine, #53722, I accidentally removed the variable assignment

This is the exact diff from the PR

-  dag_id, task_id, queue = prev_key
-  Stats.gauge(f"ti.{state}.{queue}.{dag_id}.{task_id}", 0)
-  Stats.gauge(f"ti.{state}", 0, tags={"queue": queue, "dag_id": dag_id, "task_id": task_id})
+  DualStatsManager.gauge(
+          f"ti.{state}",
+          0,
+          tags={},
+          extra_tags={"queue": queue, "dag_id": dag_id, "task_id": task_id},
+  )

This is the line that shouldn't have been removed

dag_id, task_id, queue = prev_key

I tested it manually and the error is gone.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@xBis7 xBis7 requested review from XD-DENG and ashb as code owners February 10, 2026 10:46
@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Feb 10, 2026
@potiuk potiuk merged commit 4b4e66d into apache:main Feb 10, 2026
72 checks passed
@xBis7
Copy link
Contributor Author

xBis7 commented Feb 10, 2026

@shahar1 @potiuk Thank you!

@xBis7 xBis7 deleted the dual-stats-manager-bug branch February 10, 2026 12:12
Alok-kumar-priyadarshi pushed a commit to Alok-kumar-priyadarshi/airflow that referenced this pull request Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants