Use Prometheus collector pattern for stateless metrics

[metalmatze]

What this PR does / why we need it:
If a machine can't be deleted we want to alert on it.
With the old approach we had state in the metrics that we couldn't delete. Hence a machine that once existed still had metrics which made us unable to alert on the metric, even when the machine was long gone.

By using the Prometheus collector pattern we simply query the informers to give us a list of machines and nodes. We iterate over those lists and forget about the state afterwards.
This improvement allows metrics of deleted machines to not show up anymore.
We can now tell, that if a machine_controller_machine_deleted metric is there for more than 10min the machine is not able to be deleted.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Special notes for your reviewer:
I've also removed the histograms for the operations as they were utterly broken.
Example histogram:

machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.005"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.01"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.025"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.05"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.1"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.25"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="0.5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="1"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="2.5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="5"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="10"} 64
machine_controller_controller_operation_duration_seconds_bucket{operation="validate-machine",le="+Inf"} 64
machine_controller_controller_operation_duration_seconds_sum{operation="validate-machine"} 0.00042467500000000006
machine_controller_controller_operation_duration_seconds_count{operation="validate-machine"} 64

In this example, all buckets were hit 64 making them useless. For all the other machine-controller histograms I looked at I saw the same results.

[alvaroaleman]

Thanks for the PR! Can you create a tracking issue for the defunct metrics you removed?

[metalmatze]

Done in #271

[alvaroaleman]

This should be NodeCollector not NodeController right?

[metalmatze]

Yes, that's a typo. 😇

[alvaroaleman]

I am actually not sure if this metric makes sense, because it will include all nodes, not only the ones that were created by a machine-controller, we could get the same info from kube state metrics

[metalmatze]

Yup, had the same thought yesterday. Then I also looked into kube-state-metrics again and found kube_node_created.
The reason I added them back were

1. We already had node metrics.
2. You still get node metrics, even though you have not deployed kube-state-metrics.

I'm fine with removing them.

[alvaroaleman]

This should read something like nodeCreationTimestamp or am I misunderstanding something?

[metalmatze]

I am not 100% sure on that one either.
I searched for created and deleted in our Prometheus metrics and found that kube-state-metrics simply uses e.g. kube_node_created and others.
The Prometheus docs don't say anything specific about naming for timestamps either.
https://prometheus.io/docs/instrumenting/writing_exporters/#naming