a DistributedExecutor equivalent to CeleryExecutor without Celery #26689
Unanswered
raphaelauv
asked this question in
Ideas
Replies: 1 comment 2 replies
-
It's already possible. You can run multiple Schedulers - each with LocalExecutor and it should work in principle. |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Since the scheduler is capable of doing HA thanks to
SKIP LOCKED
https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html#running-more-than-one-scheduler
a distributed airflow executor without celery ( and it's backend redis/rabbitmq ) would give the possibility to have a simpler design for a airflow distributed mode with workers ( the same way the triggerer(s) do not need celery )
wdyt ?
ps : there is an open discussion on celery about using postgres as a broker backend -> celery/celery#5149
Beta Was this translation helpful? Give feedback.
All reactions