Make RollupMapper calendar windows DST-aware#68150
Open
Vamsi-klu wants to merge 3 commits into
Open
Conversation
Vamsi-klu
pushed a commit
to Vamsi-klu/airflow
that referenced
this pull request
Jun 7, 2026
The check-newsfragment-pr-number CI check requires the newsfragment file to be named after the PR number, not the issue number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The check-newsfragment-pr-number CI check requires the newsfragment file to be named after the PR number, not the issue number.
2309bee to
98106c6
Compare
The check-newsfragments-are-valid hook requires non-significant newsfragments to be a single line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make
DayWindow(the calendar window used byRollupMapper) DST-aware. With a UTC or naive upstream mapper a day is still exactly 24 hourly steps, so nothing changes. When paired with a local-timezone upstream mapper (e.g.StartOfDayMapper(timezone="America/New_York")) the window now enumerates the real local hours of the calendar day — 23 on a spring-forward day, 25 on a fall-back day — instead of a fixed 24.The timezone is read from the upstream mapper (a
tzinfoproperty added to the temporal mapper) and threaded intoWindow.to_upstream(..., tz)at call time, so windows stay stateless and serialization-free.Why
DayWindowassumed every day has 24 hours. With a local-timezone upstream mapper, a spring-forward day has only 23 real hours, so one expected hourly key (e.g.2024-03-10T02) never occurs — the required upstream key set could never be fully satisfied and the asset-partitioned rollup run was held forever. See #68004.Impact
01:00collapses to a single expected key unless the upstream mapper'sinput_formatcarries the UTC offset (%z) — so without%zit yields 24 distinct keys. The window no longer hangs; full 25-key coverage is left as follow-up work that also needs anencode_upstreamchange.Testing
Reworked
airflow-core/tests/unit/partition_mappers/test_window.py(TestDayWindow):test_day_window_spring_forward_local_tz_yields_23_keys— replaces a formerxfailtest; asserts a spring-forward rollup (2024-03-10,America/New_York,%Y-%m-%dT%H) now expects exactly 23 upstream keys and that2024-03-10T02is absent.test_day_window_fall_back_local_tz_does_not_hang— fall-back day (2024-11-03) yields 24 distinct keys (the repeated local01:00collapses to one key without%z); crucially none is unsatisfiable, so the rollup no longer hangs.test_day_window_non_dst_local_day_yields_24_keys— a non-DST local day (2024-06-10) still yields 24 keys.test_day_window_utc_yields_24_keys_on_dst_day— a UTC upstream mapper yields 24 keys even on the spring-forward date.test_day_window_tz_none_yields_24_naive_steps(renamed) — withtz=None,DayWindowstill yields a fixed 24 naive hourly steps on both DST dates.Also updates the
assets.rstdocs and theexample_asset_partition.pydocstring to describe the DST-aware behavior, syncs the Task SDKDayWindowdocstring (declaration only; the enumeration runs in airflow-core), and adds the68150.bugfix.rstnewsfragment.closes: #68004
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines