notification improvements

[zsaltys]

There are various actions in data.all that happen asynchronously such as a share being processed, dataset and environment stacks being updates, health of shares being verified etc. All of these actions can fail for whatever reason. Currently when things fail users are not able to find out until they check the logs or inspect data.all UI.

I'd like to propose that all asynchronous actions in data.all that can fail would send notifications and also be integrated with the recent reminder service that was implemented for shares to remind users that approvals for shares are pending.

A list of actions that should send notifications and to whom they should be sent:

a) An approved share fails (or revoked) to become successfully processed - both dataset owner and requester team should be notified and reminded. Optionally DA admins should also be able to subscribe to these notifications.

b) Dataset or Environment stack fails to update environment admin team should be notified as well as the dataset owner team. Reminders should be sent. DA admins should be able to optionally subscribe as well.

c) Any background task such as catalog reindexing, share health verified - if it fails (throws an exception) then DA admins should be notified ONCE.

d) If a share becomes unhealthy both the requester and dataset owner should be notified and reminded. DA admins should be able to optionally subscribe also.

e) If attempting to re-apply a share fails both the requester and dataset owner should be notified ONCE. DA admins should be able to optionally subscribe also.

There are already a few tickets filed for this effort that we should combine into this major story:

#1251
#1299

There might be more that I was not able to find.

[noah-paige]

Hi @zsaltys - also want to link these comments from an earlier discussion on persistent reminders feature that begin to start thinking through the above issue described

#1248 (comment)
#1248 (comment)

Ultimately I envision we would have a list of different "NotificationEvents" (i.e. StackFailedNotification, ShareUnHealthyNotification, CatalogIndexerFailedNotification, etc.) and so on that each have their own generic notification groups and reminder schedules depending on the event specified

Also this list of events should be easily extendable if tomorrow we wanted to add a notification for NewTableMetricsNotification OR NewTeamInvitedNotification for example.

I think it may be best to take some time and iron our a good generic and flexible design on how this notification system would look like in data.all. I think once we define it once it will be seamless to extend to more and more type of events or more and more type of delivery channels to send outbound notifications (on top of already in the UI and (optionally) via email)

Please let me know if you are thinking the same and let's use this issue here to discuss further on what that design could entail

[TejasRGitHub]

Another small issue - Currently if the share is with a consumption role then the email notification body text contains something like - share request for dataset <DATASET_NAME> for principal ggief66q.

The principal name is not readable and it is the internal id used by data.all for that principal. Instead of id this should be replaced with a user friendly readable consumption role name