Supporting Cassandra/Yugabyte as airflow backend DB

[tetsushiawano]

Description

Wishing compatability for active-active database as backend DB of airflow.

The current supported DB are not active-active DB by nature in my understanding. (or expensive to enable active-active feature.)

Is there any plan to support database like Cassandra or YugabyteCQL in future?

Use case/motivation

Make multi-region active-active Airflow hosted on our multiple OnPrem Datacenter.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[raphaelauv]

Airflow expect a relational database because it use SqlAlchemy

for Yugabyte , maybe you could make it work , but few postgres "features" are missing -> yugabyte/yugabyte-db#5683

[nathadfield]

@potiuk Would you like to comment on this from the perspective of project direction and priorities?

[potiuk]

Not possible without completely rewriting hiw scheduler works. We heavily rely on lock mechanisms in relation databases. That does not work well with active/active setup

And it is not needed IMHO.

Generally speaking Airflow is not targeting 99.9% available system(8 hours a year). It is far less availability as target. And this is by design. Airflow should not be used as a backbone of system that requires it. It is a batch scheduling system. Having minutes availability for Airflow is NON-goal for us.

Every single 9 there order of magnitude complexity and cost of development and requires usually much more complex maintrnance. If someone wills to invest 10x the time that has been used to develop current scheduler - maybe they should. But we never saw it as a goal. That's why we rejected things like zookeeper, non-relational databases and supporting active-active setup in the past

Supporting such setup without real need is at most following a 'trend' without really considering if it is worth the cost. You have to remember that every single decision of architecture bears a cost. And in this case - the cost is huge.

If someone would like to spend time and write Airflow Improvement Proposal and convince others and bear the cost of implementing it - yes sure, anyone is free to start it. But considering cost of implementation, deployment and deeply considering what it means to run Airflow in such a way, reasoning why it is necessary will be a crucial part of it. And convincing community members that it is worth it is the first step.

For now I personally am not convinced. But luckily it's not me to convince. Bring it to devlist, state your reasoning, fend off people who will not like it, reach consensus, get the AIP to pass the vote and implement it. This is the way how one should approach it.