Build Airflow/Superset from source

[xeniape]

This needs refinement to find out what exactly can be build from source.

Description & Motivation

We want to be able to build our python products from source that we can patch code where needed. We currently have a spike that has been tested with Airflow 3.0.1: extending this to 2.x will be cumbersome and involve conditional logic in multiple places as the code structure is different between those versions. so maybe one option is to only do that for 3.x when it becomes the LTS line.

[adwk67]

I had a look at doing this with airflow. Setting up the patchable stuff for creating the patches (i.e. before we build the image) for airflow works as expected and as per other products.

But applying those patches during the build will be a little different for python products, as e.g. airflow has already been installed with pip install, and that takes care of a lot of complexity for us (e.g. providers, constraints etc.). Which we would mean that we'd either

• need an extra command - something like patchable apply --strip <levels to strip> --directory <packages dir> - to patch things in-place or
• doing a patchable checkout on the whole code (like we do elsewhere), but then we would need to deal with everything that the current command does:
  • uv pip install --no-cache-dir apache-airflow[${AIRFLOW_EXTRAS}]==${PRODUCT_VERSION} --constraint /tmp/constraints.txt --build-constraints /tmp/build-constraints.txt

[dervoeti]

Here's a very basic approach that contains a simple modification to the startup message of Airflow 3.0.1 using the current state of Patchable: https://github.com/stackabletech/docker-images/tree/spike/airflow-from-source

It's just a PoC so can certainly be improved, but it works for me.

[adwk67]

It does not (yet) seem to possible to build the UI assets cleanly from the github source: https://www.github.com/apache/airflow/issues/54453

[adwk67]

Added a commit that builds the UI assets. Tested with Airflow 3.0.1. Not compatible with 2.x yet.

[lfrancke]

• After discussion we decided to skip Airflow 2 for this one if it causes any issues (timeboxed)
• We want to split this ticket into one for Airflow and another one for Superset
• One goal is to enable Patchable with this
• Suggestion is to maybe use this opportunity to run pytests at the "build" time so we can test our opa authoriser
  • To be refined!

[lfrancke]

Here's what I'd like you to do in this order:

• Airflow 3.x (5 days maximum)
  • Get it to a state where we can use Patchable - this is my main goal for this work and "building from source" is one way of achieving that
• Airflow 2.x (4h maximum)
  • If it turns out we can't support it then that's fine with me. We want to get rid of it anyway in the future.
• Superset
  • Let's talk again before this is started so we can see the outcome of the Airflow process first and decide whether it's worth it for Superset right now.

If this is taking longer than a week (Airflow 3) let me know and we can decide whether we want to continue or not.