AirflowTaskTimeout - Redis w/ Celery #38609
Unanswered
vandanthaker-ey
asked this question in
Q&A
Replies: 1 comment
-
You have problem with your DNS - it does not have enough resources or it is unstable. Make sure you have a DNS in your deployment that you can rely on.
Thsi is where your DNS is contacted. And obligatory Haiku. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Airflow environment details:
Version - 2.8.2
Executor - Celery
Platform - Azure Kubernetes Service
Celery Broker - Azure Cache for Redis
Metastore - Azure postgresql
Scheduler - 2 pods (Active-Active)
Worker - 2 pods (48 parallelism)
We recently implemented Airflow environment with Celery. For broker we integrated with Azure redis offering. We're seeing below error everyday 2-3 times, where scheduler gracefully restarts.
Workarounds tried -
Happy to share any other information if required. Can you please help understand what potential root cause can be?
Beta Was this translation helpful? Give feedback.
All reactions