API/metrics for pipeline health check

[whynowy]

Summary

In terms of the health status of a pipeline, there's nothing to indicate it directly. It would be good to have a health check page or API giving the health status, to make monitoring and alerting easier.

The API/metrics should give a response including:

• Pipeline overall status
• Which vertex or component has what an issue if not healthy.

This API should still be working even if the pipeline is unhealthy, daemon service might be a good place to host this.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[dpadhiar]

Assign to me.