Are we thinking about orchestrators the wrong way?

[astrojuanlu]

Let's talk about Airflow. How do people use it in the wild?

exhibit 1:

airflow is for orchestration, never use it to process data. 99% of the people I've talked to whose Airflow cluster is mess are using it like a data processing platform.. troubleshooting performance issues is a total nightmare.

exhibit 2:

Surprised to see so many answer with Airflow. Airflow is not an ETL tool, it is a scheduling tool.

exhibit 3

I would prefer to isolate these two environments so that I don't need to import Kedro in Airflow or Airflow in Kedro. As managing dependencies in Airflow is challenging, it would be better to avoid this problem altogether.
[...]
Thirdly, as the code would be executed by the same processes, it would need to be shared in the form of packages. In this setup Airflow runs in a docker image, so then I'd have to either re-build and re-run this image every time either the Airflow or Kedro project code changes, OR additionally manage lots of virtual python environments somewhere and ship the new versions of the micro-packaged Kedro pipelines there whenever the code changes.

📜

To unpack all this or try to explain it in my own words, if you run the Airflow control plane in the same process/machine as your data processing code, you risk killing Airflow because of a task gone rogue. This is more or less obvious in hindsight, but I've seen products make this mistake and it was extremely painful.

So then people realise that Airflow (or your orchestrator of choice) actually needs to live separately from the tasks, and coordinate those using some sort of queue like Celery, or use something like Kubernetes.

...at which point, why should Airflow take care of all that at all? Just make the tasks call an API, and design the backend of that API in any way you want.

🔮

Long story short: what if we should actually focus¹ on making serving Kedro pipelines as a web application flawless, and make orchestrators (including, but not limited to, Airflow) just call the corresponding REST API?²

(diagram courtesy of @datajoely)

Footnotes

1. "Focus" as in: among all the possible things we could be doing, and given time and resource constraints, what things do we say "yes" to at the cost of saying "no" to others? ↩

2. This might not apply to some platforms that are very idiosyncratic like Databricks ↩

[DimedS]

That's a really interesting idea, @astrojuanlu. I agree that separating orchestration from execution is a very good practice, and moving towards a microservices architecture would be a great step forward.

Does that mean the Kedro Server should be responsible for resource allocation? As I understand it, when Airflow runs on Kubernetes, it creates one Pod per task, and then Kubernetes is responsible for providing resources for all those Pods. In this context, the Kedro Server would need to always be running to be ready to respond, just like an Airflow scheduler. However, additional resources are only needed when executing tasks.

[astrojuanlu]

Thanks @DimedS!

I don't think kedro-server can necessarily be a single thing, because different teams will have different opinions on how to serve their pipelines. But the existence of https://github.com/takikadiri/kedro-boot/ proves that there is some friction when trying to use Kedro in that way. So this would be more about incorporating the learnings of kedro-boot into Kedro (#3540), and providing a blueprint for our recommended way of doing things (think of a "kedro-server" you can install, but assuming that most users will want to extend it)

[datajoely]

Yeah I think focus should be:

• Improve the session so that it works better for microservice applications and means things like kedro-boot don't have to do any creative workarounds.
• (Potentially) ship a very basic kedro-fastapi plug-in that lets people extend however they want

[astrojuanlu]

Would love to know @sbrugman thoughts on this given that you're heavy users of kedro-airflow

[sbrugman]

Separation of scheduling from execution is important, e.g. one can use Airflow to schedule Spark tasks (on k8s).
Note that it's not always possible to install Python dependencies on the Airflow scheduler!

Using Airflow + Kedro + Spark on k8s, we can use the SparkSubmitOperator.
This operator requires a Python script to start, so the kedro executable cannot be directly used.
Instead, we provide a kedro_launcher.py script that does just that + set some environment variables for the config location:

"""Helper script to run kedro via spark-submit"""
import os
import subprocess
import sys

if __name__ == "__main__":
    # config is placed in `src` for syncing
    os.environ["KEDRO_CONF_SOURCE"] = "src/conf/"
    project_source = "/workspace/"
    print(f"Project at {project_source}")
    KEDRO_COMMAND = ["kedro", "run"]
    args = KEDRO_COMMAND + sys.argv[1:]
    result = subprocess.run(args, cwd=project_source, check=True)
    exit(result.returncode)

Kedro only needs to be available in the environment inside the pods, not on the scheduler.
This means that at minimum, the DAG needs to be generated from the pipeline (like we do now, perhaps more generic)