Alert on node and machine IP mismatch

[embik]

Description of the feature you would like to add / User story

As a KKP admin
I would like to be alerted of primary IP mismatches between Node objects and the actual machine/VM
in order to be aware that a node changed IP addresses.

Solution details

• Raise a Prometheus alert if Node and Machine status disagree over the primary IP of the node.
• Use Prometheus metrics exposed by kube-state-metrics and machine-controller for that, consider implementing metrics upstream if they are not available yet.

Alternative approaches

Use cases

When underlying infrastructure updates a VM's ip address (e.g. by returning a differente DHCP), node to node communication in Kubernetes breaks down. We also need to document that as per kubermatic/docs#1444. But if it still happens, we should raise an alert so admins are aware and can investigate.

Additional information

[csengerszabo]

/milestone clear