Introduce repeatable alert

[samanhappy]

Currently the notification in EaseProbe is using edge triggered mode which means one notification will be sent only once when the status changes, but people may miss notifications in some situations, below are some:

• one was busy at something else when receiving a important fail notification, and when he came back the alert was just forgotten, as you know, people always forget
• in some times of emergency, one will receive many notifications in a very short period, unfortunately there are two different problems in these notifications, one is resloved, but other one was just neglected

In other notification systems using level trigger mode as I know, aliyun is using a silence duration to control repeatable alert.

So is it necessary for us to introduce a similar mechanism? What is your opinion? @haoel @proditis

[haoel]

This is a good issue.

I thought this before I wrote the first line code of EaseProbe, edge trigger or level trigger really put me in a dilemma. And it never disappears from my mind. Thanks to bring this here.

for the level trigger, I have the following concerns

• Most of the time, an accident could cause many services unavailable, and they all keep sending alerts, your phone or other notifiers would be flooded totally.

• The silence duration would be magic work, nobody can set it right.

Maybe, we can use something like "exponential backoff". For example, only sending 5 alerts, the time between them can be deltaTime = 2*deltaTime (deltaTime can be initialized as probe interval)

[samanhappy]

I like the backoff strategy, it's commonly used for handling failed operations in Kubernetes, all we need is to choose a reasonable amount of time and a maximum number of retries.