Surface worker DAG parse duration in task log#66138
Merged
kaxil merged 2 commits intoMay 1, 2026
Merged
Conversation
Customers see an opaque 'queued -> running -> first log line' delay and cannot tell whether slow task startup is due to their DAG file complexity or Airflow-side bundle operations (e.g. git fetch). The worker's parse() call happens AFTER the TI state transitions to RUNNING (supervisor.py:1031 -> api_fastapi/.../task_instances.py:108 sets state=RUNNING and writes start_date before StartupDetails is sent to the worker). Today that post-RUNNING parse phase is invisible in the UI and appears as a gap between start_date and the first task log line. This change times the two phases inside parse() and emits a single structured "Worker startup parse complete" log event in startup() after the run_as_user re-exec check (so it fires exactly once per task, never twice on impersonated runs): - bundle_prepare_ms: bundle fetch/verify (Airflow overhead; near-zero for warm GitDagBundle, non-trivial on first task after a version change) - dag_file_parse_ms: DAG file import (user code) The log line surfaces directly in the customer's task log file. Support engineers can now answer "is it them or us?" from the first log line. This is a partial closure of PM-2780. Full closure requires persisting parse duration on the TaskInstance model and surfacing it in the UI (separate, larger PR). Emitting as Stats.timing metrics was considered and deferred -- worker log is the customer-facing surface; Stats goes to ops dashboards and requires metrics registry + naming convention work (shared/observability/.../metrics_template.yaml).
Reverts the parse() signature change and the test/fixture adaptations from the prior commit. parse() again returns just RuntimeTaskInstance. The "Worker startup parse complete" log event is emitted from inside parse() right before return, keeping the public surface of the function stable for any downstream callers (pytest fixtures, etc.). Under `run_as_user` impersonation, parse() runs twice (pre-sudo and post-sudo), so this event fires twice for those tasks. The two entries expose impersonation re-parse overhead, which is informative (tasks with run_as_user pay for parse twice) rather than a defect.
jscheffl
approved these changes
May 1, 2026
eladkal
approved these changes
May 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Customers and support engineers cannot tell whether a task's apparent slow startup (opaque gap between
start_dateand the first task log line) is due to their DAG file complexity or to Airflow-side bundle operations (git fetch, etc.).Background
The worker's DAG parse happens after the TI state transitions to
RUNNING:client.task_instances.start(...)which hits theti_runendpoint -- this setsTI.state=RUNNINGand writesTI.start_date.StartupDetailsto the worker subprocess.parse()-- bundle fetch + DAG file import happen here.So the existing
queued_durationmetric captures queued_dttm → RUNNING (pre-parse: pod boot, worker fork, supervisor handshake, API roundtrip). The parse phase itself is invisible in the UI today and shows up only as a gap betweenstart_dateand the first task log line.What this PR does
Times the two phases inside
parse()and emits a single structured log event just before return:The event lands in the customer's task log file (structlog kwargs, JSON-serialized by the supervisor's log channel). Support engineers can now answer "is it them or us?" from the first log line.
parse()'s public signature is unchanged (RuntimeTaskInstance); no test or pytest-fixture updates needed. Single file, ~18 lines added.Design rationale
Why a log line and not a
Stats.timingmetric? Stats goes to StatsD/Prometheus dashboards, which most customers don't own. The task log file is what customers read when debugging slow tasks -- that's where the answer needs to be. AddingStats.timingalso requires registering metrics inshared/observability/src/airflow_shared/observability/metrics/metrics_template.yamland following the_durationnaming convention -- deferred as a separate PR if ops wants dashboard integration.Why split
bundle_preparevsdag_file_parse? Different failure modes, different owners.bundle_prepare_msis bimodal by nature: near-zero for warmGitDagBundle(cached worktree), non-trivial on first task after a bundle version change.dag_file_parse_msattributes to user DAG code (heavy imports, top-leveltime.sleep, etc.).Why emit from inside
parse()instead of fromstartup()? Keepingparse()'s signature stable avoids breaking downstream callers (pytest fixtures, tests, any unknown use). Underrun_as_userimpersonation,parse()runs twice (pre-sudo and post-sudo), so this event fires twice for those tasks. The two entries expose impersonation re-parse overhead, which is informative rather than a defect.Gotchas / known limitations
StartupDetailsfetch can dominate on cold K8sExecutor pods.queued_durationalready measures that full pre-RUNNING gap end-to-end; this PR adds the post-RUNNING phase.GitDagBundleshows near-zerobundle_prepare_ms. Expected -- the cached worktree check short-circuits. Only cold pulls (first task after a version change) emit non-trivial values, and that latency is real.run_as_userimpersonation emits the event twice per task. Once in the pre-sudo process, once in the sudo'd process. Customers reading the task log see both -- the second one is the measurement that matters for their task's real startup.