Ensure `DateTimeTrigger` receives a datetime object #17747

kaxil · 2021-08-20T01:32:54Z

While using the following example DAG, the task failed with moment does not have tzinfo attribute. This happened because a string was passed from DateTimeSensor to DateTimeTrigger. This PR ensures that a datetime object is passed and fixed logic in DateTimeSensor too so that self.target_time is always a datetime object.

Stacktrace:

[2021-08-19, 21:57:17 UTC] {taskinstance.py:1657} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1296, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1415, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1471, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/date_time.py", line 87, in execute
    self.defer(trigger=DateTimeTrigger(moment=self.target_time), method_name="execute_complete")
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/triggers/temporal.py", line 37, in __init__
    if moment.tzinfo is None:
AttributeError: 'str' object has no attribute 'tzinfo'

Example DAG:

from datetime import timedelta

from airflow import DAG
from airflow.sensors.date_time import DateTimeSensorAsync
from airflow.utils import dates, timezone

with DAG(
    dag_id='example_date_time_async_operator',
    schedule_interval='0 0 * * *',
    start_date=dates.days_ago(2),
    dagrun_timeout=timedelta(minutes=60),
    tags=['example', 'example2', 'async'],
) as dag:

    DateTimeSensorAsync(task_id="test", target_time=timezone.datetime(2021, 8, 19, 23, 15, 0))

cc @andrewgodwin

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

andrewgodwin

Can't believe I missed that str inputs were allowed!

uranusjr

Come on CI…

github-actions · 2021-08-20T04:17:06Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

While using the following example DAG, the task failed with `moment` does not have `tzinfo` attribute. This happened because a string was passed from `DateTimeSensor` to `DateTimeTrigger`. This PR ensures that a datetime object is passed and fixed logic in `DateTimeSensor` too so that `self.target_time` is always a datetime object. ```python from datetime import timedelta from airflow import DAG from airflow.sensors.date_time import DateTimeSensorAsync from airflow.utils import dates, timezone with DAG( dag_id='example_date_time_async_operator', schedule_interval='0 0 * * *', start_date=dates.days_ago(2), dagrun_timeout=timedelta(minutes=60), tags=['example', 'example2', 'async'], ) as dag: DateTimeSensorAsync(task_id="test", target_time=timezone.datetime(2021, 8, 19, 23, 15, 0)) ```

While fixing ``DateTimeSensorAsync`` in apache#17747 -- I broke ``DateTimeSensor``. As `target_time` is a template_field for `DateTimeSensor`, Jinja tries to render it which does not work if the input is a datetime object or if someone passes just a template field like ``{{ execution_date }} `` it throws an error: ``` DateTimeSensor(task_id="foo", target_time="{{ execution_time }}" ) ' Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 131, in _parse dt = parser.parse( File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: {{ execution_time }} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 3, in <module> File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 186, in apply_defaults result = func(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airflow/sensors/date_time.py", line 66, in __init__ self.target_time = timezone.parse(target_time) File "/usr/local/lib/python3.9/site-packages/airflow/utils/timezone.py", line 175, in parse return pendulum.parse(string, tz=timezone or TIMEZONE, strict=False) # type: ignore File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 29, in parse return _parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 45, in _parse parsed = base_parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 74, in parse return _normalize(_parse(text, **_options), **_options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 135, in _parse raise ParserError("Invalid date string: {}".format(text)) pendulum.parsing.exceptions.ParserError: Invalid date string: {{ execution_time }} ``` This PR fixes it by reverting change in `DateTimeSensor` and parses the string to datetime in `DateTimeSensorAsync.execute`

While fixing ``DateTimeSensorAsync`` in #17747 -- I broke ``DateTimeSensor``. As `target_time` is a template_field for `DateTimeSensor`, Jinja tries to render it which does not work if the input is a datetime object or if someone passes just a template field like ``{{ execution_date }} `` it throws an error: ``` DateTimeSensor(task_id="foo", target_time="{{ execution_time }}" ) ' Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 131, in _parse dt = parser.parse( File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: {{ execution_time }} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 3, in <module> File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 186, in apply_defaults result = func(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airflow/sensors/date_time.py", line 66, in __init__ self.target_time = timezone.parse(target_time) File "/usr/local/lib/python3.9/site-packages/airflow/utils/timezone.py", line 175, in parse return pendulum.parse(string, tz=timezone or TIMEZONE, strict=False) # type: ignore File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 29, in parse return _parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 45, in _parse parsed = base_parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 74, in parse return _normalize(_parse(text, **_options), **_options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 135, in _parse raise ParserError("Invalid date string: {}".format(text)) pendulum.parsing.exceptions.ParserError: Invalid date string: {{ execution_time }} ``` This PR fixes it by reverting change in `DateTimeSensor` and parses the string to datetime in `DateTimeSensorAsync.execute`

While fixing ``DateTimeSensorAsync`` in apache/airflow#17747 -- I broke ``DateTimeSensor``. As `target_time` is a template_field for `DateTimeSensor`, Jinja tries to render it which does not work if the input is a datetime object or if someone passes just a template field like ``{{ execution_date }} `` it throws an error: ``` DateTimeSensor(task_id="foo", target_time="{{ execution_time }}" ) ' Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 131, in _parse dt = parser.parse( File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: {{ execution_time }} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 3, in <module> File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 186, in apply_defaults result = func(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airflow/sensors/date_time.py", line 66, in __init__ self.target_time = timezone.parse(target_time) File "/usr/local/lib/python3.9/site-packages/airflow/utils/timezone.py", line 175, in parse return pendulum.parse(string, tz=timezone or TIMEZONE, strict=False) # type: ignore File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 29, in parse return _parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parser.py", line 45, in _parse parsed = base_parse(text, **options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 74, in parse return _normalize(_parse(text, **_options), **_options) File "/usr/local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 135, in _parse raise ParserError("Invalid date string: {}".format(text)) pendulum.parsing.exceptions.ParserError: Invalid date string: {{ execution_time }} ``` This PR fixes it by reverting change in `DateTimeSensor` and parses the string to datetime in `DateTimeSensorAsync.execute` GitOrigin-RevId: fbdb6882e4789c55c751b8be466eed89945d3241

kaxil requested a review from jedcunningham August 20, 2021 01:32

boring-cyborg bot added the area:core-operators Operators, Sensors and hooks within Core Airflow label Aug 20, 2021

kaxil force-pushed the fix-date-time-trigger branch from a1e53cd to 7aea47c Compare August 20, 2021 01:43

kaxil requested a review from uranusjr August 20, 2021 01:43

andrewgodwin approved these changes Aug 20, 2021

View reviewed changes

uranusjr approved these changes Aug 20, 2021

View reviewed changes

github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Aug 20, 2021

kaxil closed this Aug 20, 2021

kaxil reopened this Aug 20, 2021

kaxil removed the full tests needed We need to run full set of tests for this PR to merge label Aug 20, 2021

kaxil force-pushed the fix-date-time-trigger branch from 7aea47c to 51686d1 Compare August 20, 2021 16:04

kaxil merged commit e21b54a into apache:main Aug 20, 2021

kaxil deleted the fix-date-time-trigger branch August 20, 2021 18:42

kaxil mentioned this pull request Sep 1, 2021

Fix passing Jinja templates in DateTimeSensor #17959

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure `DateTimeTrigger` receives a datetime object #17747

Ensure `DateTimeTrigger` receives a datetime object #17747

kaxil commented Aug 20, 2021 •

edited

andrewgodwin left a comment

uranusjr left a comment

github-actions bot commented Aug 20, 2021

Ensure DateTimeTrigger receives a datetime object #17747

Ensure DateTimeTrigger receives a datetime object #17747

Conversation

kaxil commented Aug 20, 2021 • edited

andrewgodwin left a comment

Choose a reason for hiding this comment

uranusjr left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 20, 2021

Ensure `DateTimeTrigger` receives a datetime object #17747

Ensure `DateTimeTrigger` receives a datetime object #17747

kaxil commented Aug 20, 2021 •

edited