Could not get scheduler_job_id

[benyaa]

Apache Airflow version: 2.0.2

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.20.2

Environment:

• Cloud provider or hardware configuration: Kubernetes

What happened: I tried to configure the scheduler, which runs in its own pod, to use the CeleryKubernetesExecutor, but it failed with the error "Could not get scheduler_job_id". I saw that it failed before with the KubernetesExecutor on its own and fixed.
If I set the executor to be KubernetesExecutor it initiates okay, if I set it to be CeleryExceutor it also initiates fine but it only happens when I use the CeleryKubernetesExecutor that it fails.

What you expected to happen: The scheduler should have initiate successfully.

How to reproduce it: I guess it can be reproduced if you configure it to use this executor, I also use a custom pod template and redis as a celery broker so maybe it has something to do with this.

Anything else we need to know:

How often does this problem occur? Once? Every time etc? Every time

Any relevant logs to include? Put them here in side a detail tag:

x.log

WARNING! You should run the image with GID (Group ID) set to 0 even if you use 'airflow' user (UID=50000) You started the image with UID=50000 and GID=50000 This is to make sure you can run the image with an arbitrary UID in the future. See more about it in the Airflow's docker image documentation http://airflow.apache.org/docs/docker-stack/entrypoint ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2021-04-26 10:57:11,733] {scheduler_job.py:1251} INFO - Starting the scheduler

[2021-04-26 10:57:11,734] {scheduler_job.py:1256} INFO - Processing each file at most -1 times

[2021-04-26 10:57:11,736] {kubernetes_executor.py:471} INFO - Start Kubernetes executor

[2021-04-26 10:57:11,736] {scheduler_job.py:1302} ERROR - Exception when executing SchedulerJob._run_scheduler_loop

Traceback (most recent call last):

File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _execute
self.executor.start()

File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_kubernetes_executor.py", line 60, in start
self.kubernetes_executor.start()

File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
raise AirflowException("Could not get scheduler_job_id")

airflow.exceptions.AirflowException: Could not get scheduler_job_id

[2021-04-26 10:57:11,738] {dag_processing.py:453} WARNING - Ending without manager process.

[2021-04-26 10:57:11,738] {scheduler_job.py:1305} INFO - Exited execute loop

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jedcunningham]

I wasn't able to replicate this in master. #13247 fixes this, but looks like it wasn't actually cherry picked into 2.0.1, so I'm going to move it into the 2.0.3 milestone so it lands in the next release.