Show blacklisting in the Grafana Dashboard

[Zelldon]

Is your feature request related to a problem? Please describe.
Came up in a recent incident review that it would be nice to see whether instances are blacklisted or not, maybe also the rate of blacklisting.

I realized that we added a while ago the metrics for that, but never added this to the dashboard, see #6715.

Describe the solution you'd like

Either show a graph of blacklisted instance count or just some indication whether instances are blacklisted.

I had yesterday a hard time to find a good visualization, since the count is only exported if there was a new blacklisted instance. If the pod is restarted the metric is not exported. It might make sense to refill the metric on restart OR we need to work with it, but this would mean we can only show limited data, see below.

Example A - Show an indication that instances are blacklisted

In general I like this, since it shows directly whether there is something wrong.
The problem here is if the time frame is smaller (were no instances are blacklisted) than this indication is green. 👎

Example B - Graph

Showing a graph is not that fruitful, since as described above the metric is not always exported.

Showing zero for null values

Example C - Rate

The rate of such metric is also not really useful, since the change is too rare.

Example D - Table

Other alternative would be to show in a table the recent count, but this is also very limited (to the time frame) and other

Describe alternatives you've considered.

Ideally we would export the metric always, this would simplify things enormous. Then we can choose better one or more of the possible visualization from above.

Additional context

Blacklisting always shows that something problematic happened in the process execution (processing). An exception was thrown during the execution, which mostly an indication of a bug.

I would like to find a good way how we can visualize it and hope someone has some comments, opinion or ideas.

\cc @pihme

[pihme]

What about exposing the metric as a rate and visualizing it with a heat map?

[npepinpe]

Prioritized as planned for now under the assumption we won't work on a better long term alternative to blacklisting in the next quarter, and this will already be an improvement in terms of visibility. Before opening a PR for this, please discuss and decide on the visualization/metric that we want.

[Zelldon]

The question for me is really what we want to achieve with the metric.

Do we want to know really how many (exact) are blacklisted? Then Prometheus might be not the best fit, but if we just want to have an indicator that SOMETHING is blacklisted then this could work:

Here we could change the colors and shown values to something like Blacklisted (if x > 0) and nothing (if x <= 0).I think this is similar what I have shown above with example A. This would potentially already help.

[pihme]

What about exposing the metric as a rate and visualizing it with a heat map?

[Zelldon]

This wouldn't help since the metric is not exported all the time. Plus what should tell me the heatmap would be the question?

[pihme]

I would expect to see a change in the blacklisting rate.

Kinda like here:

If it is not exported for some time, I will just see a black column, but I can always make the time scale wider and then I should see data pretty much for all the time, because each export would be aggregated in the rate (I hope).

Then I could look at a big enough time scale and if my blacklisting suddenly jumps from 2 per day to 200 per day, that would be my signal. And if the jump is correlated to e.g. the point there was an update or a pod restart, that would also be interesting.

This is all speculation though, one would have to see what it actually looks like.

[Zelldon]

Thanks @pihme

yeah so it would look like this

15 mins

90 days

Problem is that we not store data long enough on our prometheus server. So there can be data deleted which contained some blacklisting, which is why I ask what we want to see. The real count or just an indication all in all I think our current metric doesn't work well. Ideally we would report always or on bootstrap with the current value of blacklisted instances.

[pihme]

Thanks @Zelldon. Yeah I regularly forget we forget data. I think 90 days would be good enough, and yes, the visualization looks like what I had in mind. But if the history is 30 days or less, it becomes less useful. So I see your point.

[Zelldon]

I think the best would be as described as alternative above:

Describe alternatives you've considered.
Ideally we would export the metric always, this would simplify things enormous. Then we can choose better one or more of the possible visualization from above.

This means at least on restart we export the metric at least once.

[Zelldon]

@remcowesterhoud since you mentioned to me that you had to look at a cluster with zdb to find out whether something was blacklisted might be useful for you to just have this in the metrics :)

[menski]

Right now we don't see value to implement this, we have other ways to identify blacklisted instances, which also allow us to see the corresponding process instance keys, i.e. logs or zdb.

Hopefully at one point we can get rid of blacklisting

[korthout]

ZPA triage:

• we want to gain more insights into the frequency of blacklisting before choosing to replace the concept

[remcowesterhoud]

Attaching this to the "Blacklisting replacement" as having insights in these metrics may be useful for this epic.

[Zelldon]

Reopening since we still need panels on the dashboard.

[koevskinikola]

ZPA triage:

• @Zelldon you mentioned HERE that you'll add the Grafana dashboards. Is this still your plan, or ZPA should do something from our side?
  • ZPA also has this issue marked as upcoming so we would do it soon either way.