Make --executor flag work for breeze start-airflow command

[utkarsharma2]

Introduced --executor flag for command breeze start-airflow which accepts CeleryExecutor, and LocalExecutor.
related: #28412

[uranusjr]

Would it be a good idea to make it an option on shell as well?

[utkarsharma2]

@potiuk, CeleryExecutor, LocalExecutor, CeleryKubernetesExecutor, and KubernetesExecutor support many configuration options in airflow config, should we expose all of them to the user? or can we expose some of them?

Also, what would be a desirable format for exposing config options?
Options:

1. We can have one --celery that can accept config string in format worker_concurrency=16, flower_host=0.0.0.0 format.
2. We can expose one parameter for each possible config. Like --celery_worker_concurrency, --celery_flower_host.

[utkarsharma2]

@uranusjr Do you think it should be part of this PR or can we create a separate ticket? either way, I'll be happy to pick up the task :)

[uranusjr]

I’m not sure how involved it would be; if it’s just a handful of lines let’s include it in this PR, otherwise we can split.

[utkarsharma2]

I’m not sure how involved it would be; if it’s just a handful of lines let’s include it in this PR, otherwise we can split.

@uranusjr Sure, I'll check what needs to be done to extend this flag to shell

[potiuk]

I think all of the executors could be gradually added but we should start small.

BTW.. This is just a beginning and draft of the change. This PR needs far more to be even close to completion.

Currently Breeze does NOT support directly running anythong else than airflow with Local (mysql/postgres/mssql) or Sequential (sqlite) executor. In order to that more things nee to happen:

1. When we choose Celery Executor, we should force starting the celery integration (which has celery, redis, possibly we should also - optionally - start flower there)

2. For Kubernetes Executir we need to connect to an external K8S cluster with capabilities of running PODs on them or we should start kind cluster, similarly to what we do in breeze k8s commands and we should be able to talk to the k8s cluster via the API

I think we should start with these two and once we have them working, we could add the "combos" (CeleryKubernetes will require to have both Celery and K8S started)

The idea here is that by running `

[potiuk]

Would it be a good idea to make it an option on shell as well?

Yep. We should

[utkarsharma2]

I think all of the executors could be gradually added but we should start small.

BTW.. This is just a beginning and draft of the change. This PR needs far more to be even close to completion.

Currently Breeze does NOT support directly running anythong else than airflow with Local (mysql/postgres/mssql) or Sequential (sqlite) executor. In order to that more things nee to happen:

1. When we choose Celery Executor, we should force starting the celery integration (which has celery, redis, possibly we should also - optionally - start flower there)

2. For Kubernetes Executir we need to connect to an external K8S cluster with capabilities of running PODs on them or we should start kind cluster, similarly to what we do in `breeze k8s` commands and we should be able to talk to the k8s cluster via the API

I think we should start with these two and once we have them working, we could add the "combos" (CeleryKubernetes will require to have both Celery and K8S started)

The idea here is that by running `

@potiuk Sure, I'll get started with Celery Executor and Kubernetes Executors for now :)

[utkarsharma2]

@potiuk Is it okay to have Celery and a Local executor for now? and I'll raise a separate PR for Kubernetes?

If it's okay, this PR is ready to review.

[potiuk]

Yep. It's perfectly fine. K8S executor is much more complex to work on. I will take a look in the next few days (I am less available over the next few days).

[potiuk]

Looks fantastic thanks!

[ferruzzi]

That's funny, literally just went looking for this feature today to try breeze start-airflow with the celery executor. 👍 Nice timing :)

[utkarsharma2]

@uranusjr I raised a PR for adding the executor option to the shell command here - #31861