Remove broken histograms from metrics

[metalmatze]

We want to remove the histograms for the operations as they were utterly broken.
Example histogram:

machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.005"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.01"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.025"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.05"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.1"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.25"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="1"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="2.5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="10"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="+Inf"} 64
machine_controller_controller_operation_duration_seconds_sum{operation="validate-machine"} 0.00042467500000000006
machine_controller_controller_operation_duration_seconds_count{operation="validate-machine"} 64

In this example, all buckets were hit 64 making them useless. For all the other machine-controller histograms I looked at, I saw the same results.

[alvaroaleman]

This issue is about re-adding them because they will get removed within #270 not about removing them, right @metalmatze?

[metalmatze]

No. You asked me to create an issue to track the removal progress, didn't you?
I don't think there's anything similar we can add right now.

[alvaroaleman]

Oh, sorry I wasn't very clear there, what I wanted is an issue about re-adding them because I think they are pretty useful if they work.

I don't think there's anything similar we can add right now.

Can you elaborate?

[metalmatze]

The problem with the histograms are firstly the buckets. On the one hand buckets the validation operations were way to high, it took less than 50ms, and on the other hand the buckets were way to small for e.g. the join operation which took 2.5min on DigitalOcean.
The only solution I might see is adding a counter of how many operations were observed and then one counter for the sum of all durations, so that you could still calculate the average.
In the end I'm not sure what this helps with. We can't really alert on that and a dashboard doesn't add much value IMHO.

[alvaroaleman]

Alright, that makes sense.