From 484e15510cc9aeb17878b919abea4290fff5ee55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= Date: Sun, 18 Aug 2019 13:44:10 +0200 Subject: [PATCH] [AIRFLOW-XXX] Group executors in one section (#5834) --- .../use-celery.rst => executor/celery.rst} | 4 +-- .../use-dask.rst => executor/dask.rst} | 4 +-- docs/executor/index.rst | 30 ++++++++++++++++ docs/executor/kubernetes.rst | 36 +++++++++++++++++++ docs/howto/index.rst | 2 -- docs/index.rst | 1 + docs/kubernetes.rst | 23 +++--------- 7 files changed, 75 insertions(+), 25 deletions(-) rename docs/{howto/executor/use-celery.rst => executor/celery.rst} (98%) rename docs/{howto/executor/use-dask.rst => executor/dask.rst} (97%) create mode 100644 docs/executor/index.rst create mode 100644 docs/executor/kubernetes.rst diff --git a/docs/howto/executor/use-celery.rst b/docs/executor/celery.rst similarity index 98% rename from docs/howto/executor/use-celery.rst rename to docs/executor/celery.rst index 9e22a8c200de2..acfecafa77683 100644 --- a/docs/howto/executor/use-celery.rst +++ b/docs/executor/celery.rst @@ -15,8 +15,8 @@ specific language governing permissions and limitations under the License. -Scaling Out with Celery -======================= +Celery Executor +=============== ``CeleryExecutor`` is one of the ways you can scale out the number of workers. For this to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and diff --git a/docs/howto/executor/use-dask.rst b/docs/executor/dask.rst similarity index 97% rename from docs/howto/executor/use-dask.rst rename to docs/executor/dask.rst index 08e923af022a8..556752d42d443 100644 --- a/docs/howto/executor/use-dask.rst +++ b/docs/executor/dask.rst @@ -15,8 +15,8 @@ specific language governing permissions and limitations under the License. -Scaling Out with Dask -===================== +Dask Executor +============= :class:`airflow.executors.dask_executor.DaskExecutor` allows you to run Airflow tasks in a Dask Distributed cluster. diff --git a/docs/executor/index.rst b/docs/executor/index.rst new file mode 100644 index 0000000000000..74d927fb0cbcc --- /dev/null +++ b/docs/executor/index.rst @@ -0,0 +1,30 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Executor +======== + +Executors are the mechanism by which task instances get run. + +Airflow has support for various executors. Current used is determined by the ``executor`` option in the ``core`` +section of the configuration file. + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/executor/kubernetes.rst b/docs/executor/kubernetes.rst new file mode 100644 index 0000000000000..418546b10fba7 --- /dev/null +++ b/docs/executor/kubernetes.rst @@ -0,0 +1,36 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Kubernetes Executor +=================== + +The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance. + +Example helm charts are available at ``scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml`` in the source distribution. +The volumes are optional and depend on your configuration. There are two volumes available: + +- **Dags**: + + - By storing dags onto persistent disk, it will be made available to all workers + + - Another option is to use ``git-sync``. Before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod + +- **Logs**: + + - By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down + + - Another option is to use S3/GCS/etc to store logs diff --git a/docs/howto/index.rst b/docs/howto/index.rst index 9360d59ee0f61..98efaeb801247 100644 --- a/docs/howto/index.rst +++ b/docs/howto/index.rst @@ -34,8 +34,6 @@ configuring an Airflow environment. connection/index secure-connections write-logs - executor/use-celery - executor/use-dask run-behind-proxy run-with-systemd run-with-upstart diff --git a/docs/index.rst b/docs/index.rst index 3384f255384c1..2d3358892c877 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -81,6 +81,7 @@ Content ui concepts scheduler + executor/index plugins security timezone diff --git a/docs/kubernetes.rst b/docs/kubernetes.rst index c268d4cbf3a71..bc6564f7d364d 100644 --- a/docs/kubernetes.rst +++ b/docs/kubernetes.rst @@ -21,27 +21,14 @@ Kubernetes Kubernetes Executor ^^^^^^^^^^^^^^^^^^^ -The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance. - -Example helm charts are available at ``scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml`` in the source distribution. -The volumes are optional and depend on your configuration. There are two volumes available: - -- **Dags**: - - - By storing dags onto persistent disk, it will be made available to all workers - - - Another option is to use ``git-sync``. Before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod - -- **Logs**: - - - By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down - - - Another option is to use S3/GCS/etc to store logs - +The :doc:`Kubernetes Executor ` allows you to run tasks on Kubernetes as Pods. Kubernetes Operator ^^^^^^^^^^^^^^^^^^^ +The :doc:`KubernetesPodOperator ` allows you to create Pods on Kubernetes. It works with +any type of executor. + .. code:: python from airflow.contrib.operators import KubernetesOperator @@ -146,8 +133,6 @@ Kubernetes Operator ) -See :class:`airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator` - Pod Mutation Hook ^^^^^^^^^^^^^^^^^