Using openmetada to configure and deploy dagster for ingestion

[slamer59]

Is your feature request related to a problem? Please describe.
For now, only Airflow can be setup through the UI.

Support alternative orchestrators for metadata ingestion #4986

Describe the solution you'd like
I want to be able to deploy dagster and select it in OpenMetadata UI.

• Helm chart should be able to deploy it
• Openmetadata should be able to run Ingestion with dagster

We might want to integrate [as a service extending the interface. ](OpenMetadata/AirflowRESTClient.java at main · open-metadata/OpenMetadata)

Mapping API
This is the dagster API

deployPipeline : I don't think there is something similar in dagster. What is the purpose of it ? Usually we create a pipeline and create a Docker image. A daemon is connected through gRPC. We add this pipeline in a configfile ConfigMap.

deletePipeline : we remove it from the ConfigMap

runPipeline : correspond to LaunchRunMutation

toggleIngestion : What is the purpose of this ? We can activate schedulers with Graphql API

getPipelineStatus : One way RunsQuery you have all runs but we can filter for OpenMetadata secific run. FilteredRunsQuery

getServiceStatus and testConnection : health check of daemon and pipeline is possible.

killIngestion : TerminateRun mutation

requestGetHostIP : What is it for ?

getLastIngestionLogs : What is it for ? We can requests metadata asociated to a pipeline.

Describe alternatives you've considered
Today one way is to do something similar to this
Not installing Airflow will through error in the UI.

Additional context
This follows this discussion
https://openmetadata.slack.com/archives/C02AZGN0WKY/p1664297118682629

[geoHeil]

Will this be part of 0.13 @pmbrull ?

[pmbrull]

hi @geoHeil we are not going to work on this in the near future. We prepared some docs around the architecture https://docs.open-metadata.org/connectors/ingestion/deployment so that the community has more vision on preparing these changes. No ETA for now. We can give support if anyone wants to pick it up

[geoHeil]

I might pick something up here in the coming days/weeks. Once it is completed I will show/share it with you and discuss how this could potentially be integrated.

[pmbrull]

hi @geoHeil we are currently working on getting this done for Argo here: #8595

This implementation should be something similar