Rule row discussion

[mdefazio]

Background

We have recently implemented several new features on the rule row as well as have several more coming. These include the following:

• Updated rule statuses (Link to RFC)
• P50/90/95 metric
• Success ratio metric
• Interval below minimum config setting warning (Link to PR)
• Snoozing (Link to PR: Relative snoozing with scheduled snoozing coming after)

With all of these recent features, this aims to take a step back and look at the whole rather than individual pieces in their respective PRs. This isn't meant to block current PRs (particularly snoozing and interval warning), but more of a follow-up.

Goals:

Provide a holistic view of a rule row, focusing on Stack management while collaborating closely with Observability, which allows a more scalable UI and more focused hierarchy.

• This is meant to be more of a monitoring workflow rather than alert triage. Currently a user must go through a rule to get to the subsequent alerts, but this will change in the coming few minors with an Alerts view.
• Provide a quick glance overview of the health of my rules and any performance hits that may be happening.
• Make space for better warning, insights per rule
• Make space for snoozing capability

Another round of rule table updates???

Some of the elements shown here do not all need to land at once and I have tried to build upon the recent rule mockups, rather than change completely. That said, this is purely for sake of discussion and not meant to be a design review for handoff. There simply has been a lot of moving parts and changes across teams.

Current table

This is the baseline I am working from. This has Enabled/Disabled/Snoozing functionality, and the percentile and success metrics. However, the last responses will be updated, new warnings included (interval min), and more emphasis on snoozing end times, recurring times, etc.

Concept for discussion

Some questions

1. The API will have [Succeeded, Failed, Warning, (No data)]. How do we want to present these to the user?
2. When a rule has an execution warning, is it (presented to the user as) 'Succeeded + with warnings' or would I rather simply see 'Warning'
3. What other types of warnings do we need to account for?
4. What is the order of columns?
5. How can we better visualize the metric data we are now showing as opposed to only showing numbers?

cc/ @ryankeairns @katrin-freihofner (Will share wider after some initial review to make sure this is worth discussing at the moment or table for later)

[katrin-freihofner]

Thank you @mdefazio I think this is a great concept!
What's unclear to me (and maybe I'm misinterpreting the mockup) do we need a time selector here? Your updates show information based on the last 200 executions (or last 10 alert count total)which might not be the same timeframe as shown in the timeselector. Additionally, I'm not sure if it's even needed to see a specific timeframe here.

The second thing that I'm wondering about is if you considered having the warning indicator closer to the reason for the warning (for example next to the last execution if that triggered the warning).

Lastly, I really like the idea of the sparkline chart for the alert count - depending on the setup the rule might only create 1 or 0 alerts, right?

[mdefazio]

What's unclear to me (and maybe I'm misinterpreting the mockup) do we need a time selector here?

You're probably right @katrin-freihofner, I dimmed this part out as I was less confident in those choices.

having the warning indicator closer to the reason for the warning (for example next to the last execution if that triggered the warning).

I agree.

depending on the setup the rule might only create 1 or 0 alerts, right?

Correct. This is the biggest change to this row based on our new rule status updates.

If a rule runs and generates 5 alerts, and then the runs again and those values have not changed (or changed but remain above the threshold), those 5 alerts are still active, but no new alerts have been generated. So my thinking was this would continue to show 5 (represented in the bar chart and the badge count). If the rule then runs again, and those alerts are resolved, the bar chart and badge would then show 0.

Wondering if the badge should only show currently active alerts, not total historical alerts (from the past X executions). I think so, since this would essentially represent our previous status of 'Active' for the rule. So perhaps this should instead be our notification badge and use the accent / pink style for when it's above 0, subdued otherwise.

Here is an updated representation of these changes:

[katrin-freihofner]

Thank you for your answers @mdefazio. About

If a rule runs and generates 5 alerts,...

I did not know, that a rule can generate 5 alerts - I thought it was a query that is true (generate alert) or false (don't generate alert). Thanks for clarifying.

[mdefazio]

It depends on the rule type, but an alert instance could be generated for each host, node, entity, etc. Some rule types would in fact be a single result like you're thinking though.

But again, with an alert table coming, perhaps this view does not need to show that level of granularity.