Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ This section contains information about deploying DAGs into production and the a
dag-serialization
modules_management
scheduler
dagfile-processing
pools
cluster-policies
priority-weight
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,6 @@ In the UI, it appears as if Airflow is running your tasks a day **late**
For more read about that you can reference `this GitHub discussion <https://github.com/apache/airflow/discussions/28809>`__.


DAG File Processing
-------------------

You can have the Airflow Scheduler be responsible for starting the process that turns the Python files contained in the DAGs folder into DAG objects
that contain tasks to be scheduled.

Refer to :doc:`../authoring-and-scheduling/dagfile-processing` for details on how this can be achieved


Triggering DAG with Future Date
-------------------------------

Expand Down
1 change: 0 additions & 1 deletion docs/apache-airflow/authoring-and-scheduling/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ It's recommended that you first review the pages in :doc:`core concepts </core-c

plugins
deferring
dagfile-processing
serializers
connections
dynamic-task-mapping
Expand Down
10 changes: 4 additions & 6 deletions docs/apache-airflow/core-concepts/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ A minimal Airflow installation consists of the following components:
a configuration property of the *scheduler*, not a separate component and runs within the scheduler
process. There are several executors available out of the box, and you can also write your own.

* A *dag processor*, which parses DAG files and serializes them into the
*metadata database*. More about processing DAG files can be found in
:doc:`/administration-and-deployment/dagfile-processing`

* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
DAGs and tasks.

Expand All @@ -74,12 +78,6 @@ performance in your Airflow:
where deferred tasks are not used, a triggerer is not necessary. More about deferring tasks can be
found in :doc:`/authoring-and-scheduling/deferring`.

* Optional *dag processor*, which parses DAG files and serializes them into the
*metadata database*. By default, the *dag processor* process is part of the scheduler, but it can
be run as a separate component for scalability and security reasons. If *dag processor* is present
*scheduler* does not need to read the *DAG files* directly. More about
processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`

* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality (similar to installed
packages). Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
Expand Down
3 changes: 2 additions & 1 deletion docs/apache-airflow/redirects.txt
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,6 @@ concepts/xcoms.rst core-concepts/xcoms.rst
concepts/scheduler.rst administration-and-deployment/scheduler.rst
concepts/pools.rst administration-and-deployment/pools.rst
concepts/priority-weight.rst administration-and-deployment/priority-weight.rst
concepts/dagfile-processing.rst authoring-and-scheduling/dagfile-processing.rst
concepts/deferring.rst authoring-and-scheduling/deferring.rst
concepts/datasets.rst authoring-and-scheduling/datasets.rst
concepts/cluster-policies.rst administration-and-deployment/cluster-policies.rst
Expand All @@ -150,6 +149,8 @@ executor/sequential.rst core-concepts/executor/sequential.rst
upgrading-from-1-10/upgrade-check.rst howto/upgrading-from-1-10/upgrade-check.rst
core-concepts/dag-run.rst authoring-and-scheduling/cron.rst
core-concepts/executor/debug.rst core-concepts/debug.rst
concepts/dagfile-processing.rst administration-and-deployment/dagfile-processing.rst
authoring-and-scheduling/dagfile-processing.rst administration-and-deployment/dagfile-processing.rst

# Moving provider executor docs to providers
## The ``../`` indicates that it will move to the root of the docs directory, unlike the rest of
Expand Down