M2: Production-ready Airflow integration

[aronchick]

ETA: 2023Q1

Parent milestone: #1183

TODO from M1 parent milestone:

• [DAG] Add python sdk tests in CI #1564
• [Won't Do] Go and Python cli must generate exactly the same payload #1555
• Rewrite the logic used to verify signed requests #1581
• Publish Python client and SDK to pypi #1483

M2 items:

• [DAG] Write a plan for pipelines #1155
• [DAG] Write new (stable) Airflow Operator #1484
  • Airflow operator #1924 (first iteration)
  • more iterations to come...
• Pipelines view in the UI #1177
• 5 pipeline examples #1178

Lineage:

• [DAG] Draft OpenLineage design doc #1479 (thanks to Polyphene.io [grant programme])
• OpenLineage PoC OpenLineage extractor enricorotundo/bacalhau-airflow-provider#1 (thanks to Polyphene.io [grant programme])
• [DAG] Integrate OpenLineage #1503 (thanks to Polyphene.io [grant programme to PoC this out])
• Support for DAGs: Ability to view lineage #1235

Host airflow:

• Go through networking doc and related PR
• [DAG] Hosting for Airflow and OpenLineage #1480
• Self-hosted Airflow (i.e. orchestrator runs inside Bacalhau) #1246

[aronchick]

Enrico - please fill out the best you can