Architecture for Catalogue Metadata Ingestion

[julialawrence]

The proposal is as follows:

Preferred architectural open metadata ingestion approach:

1. MWAA in preference to our own instance of Airflow
2. MWAA + Kubernetes in preference to MWAA + Fargate in preference to our own managed instance of airflow + Kubernetes in preference with MWAA with any other scheduling mechanism.

Leave in discussion until Monday and promote to ADR if agreed.

[jacobwoffenden]

From testing with MWAA 2.6.3, we cannot install Open Metadata's ingestion framework package due to

The conflict is caused by:
openmetadata-ingestion 1.1.0.0 depends on idna<3 and >=2.5
The user requested (constraint) idna==3.4

https://github.com/apache/airflow/blob/e514f908c8f943e5bc922b0f1674879fbbd2eeb3/constraints-3.10.txt#L355

[jacobwoffenden]

Further investigation shows that Open Metadata requires basic authentication to be enabled in Airflow, however MWAA doesn't support this.

In addition, from Open Metadata's Slack

Integrating MWAA with OpenMetadata follows Push Model. Meaning that you will be in charge of writing the DAGs using the Python SDK and deploying it to MWAA. When the DAGs Run in MWAA, it will extract metadata from your sources and push it to OpenMetadata Application.
Due to MWAA being a managed environment, it does not allow you to run custom plugins that is required by OpenMetadata Application to create ingestions on the fly from UI. Hence you see the above error page.
You can follow the docs here to setup DAGs with PythonVenvOperator and run in MWAA.
Here are the docs to get started with Python SDK.

---

The suggestion is to continue using Open Metadata's built-in ingestion (Collate's modified Airflow container https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/Dockerfile) as we've proven this works in our firebreak PoC

[julialawrence]

Decision reached. Closing