Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-XXX] Group executors in one section #5834

Merged
merged 1 commit into from Aug 18, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.

Scaling Out with Celery
=======================
Celery Executor
===============

``CeleryExecutor`` is one of the ways you can scale out the number of workers. For this
to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
Expand Down
4 changes: 2 additions & 2 deletions docs/howto/executor/use-dask.rst → docs/executor/dask.rst
Expand Up @@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.

Scaling Out with Dask
=====================
Dask Executor
=============

:class:`airflow.executors.dask_executor.DaskExecutor` allows you to run Airflow tasks in a Dask Distributed cluster.

Expand Down
30 changes: 30 additions & 0 deletions docs/executor/index.rst
@@ -0,0 +1,30 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

Executor
========

Executors are the mechanism by which task instances get run.

Airflow has support for various executors. Current used is determined by the ``executor`` option in the ``core``
section of the configuration file.

.. toctree::
:maxdepth: 1
:glob:

*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this purposefully or a typo?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works - show all files from current directory except index.
Other usage of this mechanism:
https://github.com/apache/airflow/blob/master/docs/howto/operator/gcp/index.rst

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

36 changes: 36 additions & 0 deletions docs/executor/kubernetes.rst
@@ -0,0 +1,36 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

Kubernetes Executor
===================

The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance.

Example helm charts are available at ``scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml`` in the source distribution.
The volumes are optional and depend on your configuration. There are two volumes available:

- **Dags**:

- By storing dags onto persistent disk, it will be made available to all workers

- Another option is to use ``git-sync``. Before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod

- **Logs**:

- By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down

- Another option is to use S3/GCS/etc to store logs
2 changes: 0 additions & 2 deletions docs/howto/index.rst
Expand Up @@ -34,8 +34,6 @@ configuring an Airflow environment.
connection/index
secure-connections
write-logs
executor/use-celery
executor/use-dask
run-behind-proxy
run-with-systemd
run-with-upstart
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
Expand Up @@ -81,6 +81,7 @@ Content
ui
concepts
scheduler
executor/index
plugins
security
timezone
Expand Down
23 changes: 4 additions & 19 deletions docs/kubernetes.rst
Expand Up @@ -21,27 +21,14 @@ Kubernetes
Kubernetes Executor
^^^^^^^^^^^^^^^^^^^

The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance.

Example helm charts are available at ``scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml`` in the source distribution.
The volumes are optional and depend on your configuration. There are two volumes available:

- **Dags**:

- By storing dags onto persistent disk, it will be made available to all workers

- Another option is to use ``git-sync``. Before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod

- **Logs**:

- By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down

- Another option is to use S3/GCS/etc to store logs

The :doc:`Kubernetes Executor <executor/kubernetes>` allows you to run tasks on Kubernetes as Pods.

Kubernetes Operator
^^^^^^^^^^^^^^^^^^^

The :doc:`KubernetesPodOperator <executor/kubernetes>` allows you to create Pods on Kubernetes. It works with
any type of executor.

.. code:: python

from airflow.contrib.operators import KubernetesOperator
Expand Down Expand Up @@ -146,8 +133,6 @@ Kubernetes Operator
)


See :class:`airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator`


Pod Mutation Hook
^^^^^^^^^^^^^^^^^
Expand Down