From a6a1bf3f7ea3256386393e25106b5ddb916d51bd Mon Sep 17 00:00:00 2001 From: XD-DENG Date: Tue, 25 Jun 2019 17:08:01 +0800 Subject: [PATCH 1/2] [AIRFLOW-4847] Update config template to reflect supporting different Celery pool implementation The support to different Celery pool implementation has been added in https://github.com/apache/airflow/pull/4308. But it's not reflected in the default_airflow.cfg yet, while it's the main portal of config options to most users. --- UPDATING.md | 11 +++++++++++ airflow/config_templates/default_airflow.cfg | 7 +++++++ 2 files changed, 18 insertions(+) diff --git a/UPDATING.md b/UPDATING.md index 1bb8d96cacb1c..0f1c1baf9676a 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -24,6 +24,17 @@ assists users migrating to a new version. ## Airflow Master +### `pool` config option in Celery section to support different Celery pool implementation + +The new `pool` config option allows users to choose different pool +implementation. Default value is "prefork", while choices include "prefok" (default), +"eventlet", "gevent" or "solo". This may help users achieve better concurrency performance +in different scenarios. + +For more details about Celery pool implementation, please refer to: +- https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency +- https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html + ### Removal of `non_pooled_task_slot_count` and `non_pooled_backfill_task_slot_count` `non_pooled_task_slot_count` and `non_pooled_backfill_task_slot_count` diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index f0fc1c2f72215..17737c0b3f160 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -415,6 +415,13 @@ ssl_key = ssl_cert = ssl_cacert = +# Celery Pool implementation. +# Choices include: prefork (default), eventlet, gevent or solo. +# See: +# https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency +# https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html +pool = prefork + [celery_broker_transport_options] # This section is for specifying options which can be passed to the # underlying celery broker transport. See: From ccde909c68e3a6ca49747c86a4b271e8d1b93ed4 Mon Sep 17 00:00:00 2001 From: XD-DENG Date: Tue, 25 Jun 2019 17:44:47 +0800 Subject: [PATCH 2/2] [AIRFLOW-4847] Fix typo --- UPDATING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/UPDATING.md b/UPDATING.md index 0f1c1baf9676a..ee35150879949 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -27,7 +27,7 @@ assists users migrating to a new version. ### `pool` config option in Celery section to support different Celery pool implementation The new `pool` config option allows users to choose different pool -implementation. Default value is "prefork", while choices include "prefok" (default), +implementation. Default value is "prefork", while choices include "prefork" (default), "eventlet", "gevent" or "solo". This may help users achieve better concurrency performance in different scenarios.