Skip to content

Clarify logs when a DAG is being processed in DPM#65196

Merged
amoghrajesh merged 2 commits intoapache:mainfrom
astronomer:better-log-for-dpm
Apr 15, 2026
Merged

Clarify logs when a DAG is being processed in DPM#65196
amoghrajesh merged 2 commits intoapache:mainfrom
astronomer:better-log-for-dpm

Conversation

@amoghrajesh
Copy link
Copy Markdown
Contributor


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

The "still being processed" debug log in the DAG file processor manager was logging
processor.start_time which is a time.monotonic() value, and honestly that is confusing while debugging

For example, it could lead to misreading 2637 as
"this file has been processing for 2637 seconds" when it is actually the monotonic clock value at process start).

I propose to add the elapsed duration (now - processor.start_time) alongside the start time (not removing it if someone has processes that rely on it), so the log now reads:

File path my_dag.py is still being processed (started: 2637.46, duration: 12.34s)


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the debug logging in the DAG file processor manager to make it clearer how long a DAG file has been processing, avoiding confusion caused by logging a raw time.monotonic() start value.

Changes:

  • Updates the “still being processed” debug log message to include an elapsed duration (time.monotonic() - processor.start_time).
  • Switches log formatting to fixed-precision floats for start and duration.

@amoghrajesh amoghrajesh self-assigned this Apr 15, 2026
@amoghrajesh amoghrajesh merged commit 96b5be9 into apache:main Apr 15, 2026
79 checks passed
@amoghrajesh amoghrajesh deleted the better-log-for-dpm branch April 15, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants