Send Slack alerts for DAG failures

[pankajkoti]

The PR adds on_failure_callback for all the Ask Astro DAGs
to notify DAG failures to the desired alerts Slack channel by leveraging
the Slack Notifier from the Apache Airflow Slack provider.
For the alerts to work, it will need a connection of type slack
created in the deployment. If the connection ID for this connection
is different than the default slack_api_default, then the
connection ID needs to be set in the deployment as an environment
variable named ASK_ASTRO_ALERT_SLACK_CONN_ID.

closes: #231

[josh-fell]

Just curious, to cut down the code duplication throughout, would it make sense to have a utility function/class and import it into these DAGs instead?

[pankajkoti]

@josh-fell I thought about it, but wondered what would be a good place to keep such a util in the repo. Then thought to keep it in the DAG to make the DAG self-sufficient 🙂

Would creating a new dir utils under airflow/include make sense and keeping it in there?

cc: @sunank200 @Lee-W @pankajastro

[sunank200]

You might want to rebase this PR.

@josh-fell I thought about it, but wondered what would be a good place to keep such a util in the repo. Then thought to keep it in the DAG to make the DAG self-sufficient 🙂

Would creating a new dir utils under airflow/include make sense and keeping it in there?

cc: @sunank200 @Lee-W @pankajastro

@pankajkoti @josh-fell we could have a monitoring folder inside airflow/include. We should use taskflow API which calls this common method.

[pankajkoti]

We should use taskflow API which calls this common method.

@sunank200 We're using on_failure_callback at the DAG level to leverage Slack Notifier which is already a parameter for the dag decorator. I don't think I got what you mean by Taskflow API. Please explain.

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	0ce3d12
Status:	✅  Deploy successful!
Preview URL:	https://4c7f2537.ask-astro.pages.dev
Branch Preview URL:	https://231-send-dag-failure-alerts.ask-astro.pages.dev

View logs

[pankajkoti]

@josh-fell Thanks for the suggestion. I created the required utility in include/utils/slack module.

@sunank200 I have resolved the conflicts but I did not understand the second point in your previous comment.

Requesting re-reviews please.

Sample alert

[pankajastro]

LGTM 👍

[sunank200]

I think this approach looks good to me. I thought you would be using some other custom method for this implementation which would look for any error in the DAG run.

Have you tested this implementation? If yes i think its good to be merged.

[utkarsharma2]

LGTM

[pankajkoti]

I thought you would be using some other custom method for this implementation which would look for any error in the DAG run.

The on_failure_callback at the DAG level would send an alert when any of the tasks in the DAG fail. This approach looked easier and better than adding a custom method.

Have you tested this implementation? If yes i think its good to be merged.

Yes, and I did paste a sample alert here #249 (comment).
I follow the convention to test changes before marking a PR ready for review 🙂. However, as mentioned in the PR description, for alerts to work, we would need to create the Slack API connection in the needed deployments from where we would like to start sending these alerts.

@josh-fell I am merging the PR, appreciate your review and please add more comments if you would have, I will address them in a subsequent PR.

[josh-fell]

@pankajkoti This is exactly what I was envisioning. Looks great!

[pankajkoti]

Thanks @josh-fell , appreciate you review 🙏🏽