Add metric for permanently failed notifications

[mneverov]

Fixes: #2361

Signed-off-by: Max Neverov neverov.max@gmail.com

[mneverov]

@SuperQ could you please take a look?

[simonpasquier]

the metric needs to be incremented before L701 too (when the context is done).

[mneverov]

you're right. Fixed, thanks.

[simonpasquier]

For readability it might be better to extract the existing code into a private method and wrap the new instrumentation around:

func (r RetryStage) Exec(ctx context.Context, l log.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
    r.metrics.numNotifications.Inc()
    ctx, alerts, err := r.exec(ctx, l, alerts...)
    if err != nil {
        r.metrics.numFailedNotifications.Inc()
    }
    return ctx, alerts, err
}

func (r RetryStage) exec(ctx context.Context, l log.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
...
}

[simonpasquier]

For readability it might be better to extract the existing code into a private method and wrap the new instrumentation around:

func (r RetryStage) Exec(ctx context.Context, l log.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
    r.metrics.numNotifications.Inc()
    ctx, alerts, err := r.exec(ctx, l, alerts...)
    if err != nil {
        r.metrics.numFailedNotifications.Inc()
    }
    return ctx, alerts, err
}

func (r RetryStage) exec(ctx context.Context, l log.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
...
}

[simonpasquier]

Not totally convinced by the metric name. We should also count the number of tries.
I'm wondering if we shouldn't use the existing alertmanager_notifications_failed_total to count notifications that couldn't be delivered permanently and introduce alertmanager_notification_requests_total/alertmanager_notification_requests_failed_total metrics to count notification request attempts and failures.

[mneverov]

@simonpasquier thanks for the update. I tried to be non-intrusive and keep the existing meaning for the notifications_failed_total. I agree that your suggested metrics have much clearer intent given these names.
Should I introduce them and delete the old metric or just add the new ones?

[simonpasquier]

I think that existing metrics have their own merit and shouldn't be dropped. I'd advise to rename the current alertmanager_notifications_total and alertmanager_notifications_failed_total metrics as alertmanager_notification_requests_total and alertmanager_notification_requests_failed_total and use alertmanager_notifications_total and alertmanager_notifications_failed_total for counting actual notification results.

[mneverov]

@simonpasquier could you please have another look?

[simonpasquier]

👍 thanks!