Setup alerts for broken data pipelines

[ravenac95]

Describe the feature you'd like to request

We currently don't have any alerts for any of our pipelines failing other than manually checking right now. We should have automated checks for ourselves.

Describe the solution you'd like

Some kind of automated check messages us either on discord or some other method.

Describe alternatives you've considered

N/A

[ryscheng]

Dagster+ solution
https://docs.dagster.io/dagster-plus/managing-deployments/alerts#dagster-alerts

Or the cheap version is to periodically scan Dagster API for failures, then throw that into a Discord comment

[ravenac95]

We can also think about doing things like mentioned here: dagster-io/dagster#19944

My thought would be something like this but it would send a message to our discord.

In order to have that setup we'd need to connect the secret to our dagster deployment. I would then pull that secret directly from the python code (perhaps we have an environment variable to reference it's location). Otherwise we can use the already provided secret resovler: https://github.com/doitintl/kube-secrets-init that is a side car to our applications. However, I think it would be better to have our dagster hand written components use the secret manager directly as opposed to relying on the side car. It just exposes less secrets in the environment directly. So you can't accidentally run env and log secrets by accident.