Skip to content

Commit

Permalink
Move docs for cron basics to Authoring and Scheduling section (#37049)
Browse files Browse the repository at this point in the history
(cherry picked from commit 4ca8d4e)
  • Loading branch information
ketozhang authored and potiuk committed Feb 13, 2024
1 parent 0fc8838 commit c6c9007
Show file tree
Hide file tree
Showing 4 changed files with 84 additions and 42 deletions.
70 changes: 70 additions & 0 deletions docs/apache-airflow/authoring-and-scheduling/cron.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Cron & Time Intervals
======================
You may set your DAG to run on a simple schedule by setting its ``schedule`` argument to either a
`cron expression <https://en.wikipedia.org/wiki/Cron#CRON_expression>`_, a ``datetime.timedelta`` object,
or one of the :ref:`cron-presets`.

.. code-block:: python
from airflow.models.dag import DAG
import datetime
dag = DAG("regular_interval_cron_example", schedule="0 0 * * *", ...)
dag = DAG("regular_interval_cron_preset_example", schedule="@daily", ...)
dag = DAG("regular_interval_timedelta_example", schedule=datetime.timedelta(days=1), ...)
.. _cron-presets:

Cron Presets
''''''''''''
For more elaborate scheduling requirements, you can implement a :doc:`custom timetable <../authoring-and-scheduling/timetable>`.
Note that Airflow parses cron expressions with the croniter library which supports an extended syntax for cron strings. See their documentation `in github <https://github.com/kiorky/croniter>`_.
For example, you can create a DAG schedule to run at 12AM on the first Monday of the month with their extended cron syntax: ``0 0 * * MON#1``.

.. tip::
You can use an online editor for CRON expressions such as `Crontab guru <https://crontab.guru/>`_

+----------------+--------------------------------------------------------------------+-----------------+
| preset | meaning | cron |
+================+====================================================================+=================+
| ``None`` | Don't schedule, use for exclusively "externally triggered" DAGs | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@once`` | Schedule once and only once | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@continuous``| Run as soon as the previous run finishes | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@hourly`` | Run once an hour at the end of the hour | ``0 * * * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@daily`` | Run once a day at midnight (24:00) | ``0 0 * * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@weekly`` | Run once a week at midnight (24:00) on Sunday | ``0 0 * * 0`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@monthly`` | Run once a month at midnight (24:00) of the first day of the month | ``0 0 1 * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@quarterly`` | Run once a quarter at midnight (24:00) on the first day | ``0 0 1 */3 *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@yearly`` | Run once a year at midnight (24:00) of January 1 | ``0 0 1 1 *`` |
+----------------+--------------------------------------------------------------------+-----------------+

Your DAG will be instantiated for each schedule along with a corresponding
DAG Run entry in the database backend.
1 change: 1 addition & 0 deletions docs/apache-airflow/authoring-and-scheduling/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ It's recommended that you first review the pages in :doc:`core concepts </core-c
.. toctree::
:maxdepth: 2

cron
timezone
datasets
timetable
36 changes: 0 additions & 36 deletions docs/apache-airflow/core-concepts/dag-run.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,42 +44,6 @@ There are two possible terminal states for the DAG Run:

DAGs that have a currently running DAG run can be shown on the UI dashboard in the "Running" tab. Similarly, DAGs whose latest DAG run is marked as failed can be found on the "Failed" tab.

Cron Presets
''''''''''''

You may set your DAG to run on a simple schedule by setting its ``schedule`` argument to either a
`cron expression <https://en.wikipedia.org/wiki/Cron#CRON_expression>`_, a ``datetime.timedelta`` object,
or one of the following cron "presets". For more elaborate scheduling requirements, you can implement a :doc:`custom timetable <../authoring-and-scheduling/timetable>`. Note that Airflow parses cron expressions with the croniter library which supports an extended syntax for cron strings. See their documentation `in github <https://github.com/kiorky/croniter>`_. For example, you can create a DAG schedule to run at 12AM on the first Monday of the month with their extended cron syntax: ``0 0 * * MON#1``.

.. tip::
You can use an online editor for CRON expressions such as `Crontab guru <https://crontab.guru/>`_

+----------------+--------------------------------------------------------------------+-----------------+
| preset | meaning | cron |
+================+====================================================================+=================+
| ``None`` | Don't schedule, use for exclusively "externally triggered" DAGs | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@once`` | Schedule once and only once | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@continuous``| Run as soon as the previous run finishes | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@hourly`` | Run once an hour at the end of the hour | ``0 * * * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@daily`` | Run once a day at midnight (24:00) | ``0 0 * * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@weekly`` | Run once a week at midnight (24:00) on Sunday | ``0 0 * * 0`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@monthly`` | Run once a month at midnight (24:00) of the first day of the month | ``0 0 1 * *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@quarterly`` | Run once a quarter at midnight (24:00) on the first day | ``0 0 1 */3 *`` |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@yearly`` | Run once a year at midnight (24:00) of January 1 | ``0 0 1 1 *`` |
+----------------+--------------------------------------------------------------------+-----------------+

Your DAG will be instantiated for each schedule along with a corresponding
DAG Run entry in the database backend.


.. _data-interval:

Data Interval
Expand Down
19 changes: 13 additions & 6 deletions docs/apache-airflow/core-concepts/dags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -190,18 +190,20 @@ DAGs do not *require* a schedule, but it's very common to define one. You define
with DAG("my_daily_dag", schedule="@daily"):
...

The ``schedule`` argument takes any value that is a valid `Crontab <https://en.wikipedia.org/wiki/Cron>`_ schedule value, so you could also do::
There are various valid values for the ``schedule`` argument::

with DAG("my_daily_dag", schedule="0 0 * * *"):
...

.. tip::
with DAG("my_one_time_dag", schedule="@once"):
...

For more information on ``schedule`` values, see :doc:`DAG Run <dag-run>`.
with DAG("my_continuous_dag", schedule="@continuous"):
...

If ``schedule`` is not enough to express the DAG's schedule, see :doc:`Timetables </howto/timetable>`.
For more information on ``logical date``, see :ref:`data-interval` and
:ref:`faq:what-does-execution-date-mean`.
.. tip::

For more information different types of scheduling, see :doc:`/authoring-and-scheduling/index`.

Every time you run a DAG, you are creating a new instance of that DAG which
Airflow calls a :doc:`DAG Run <dag-run>`. DAG Runs can run in parallel for the
Expand Down Expand Up @@ -237,6 +239,11 @@ schedule interval put in place, the logical date is going to indicate the time
at which it marks the start of the data interval, where the DAG run's start
date would then be the logical date + scheduled interval.

.. tip::

For more information on ``logical date``, see :ref:`data-interval` and
:ref:`faq:what-does-execution-date-mean`.

DAG Assignment
--------------

Expand Down

0 comments on commit c6c9007

Please sign in to comment.