clusterloader2 - Empty scheduler latency metrics on cluster with multiple masters

[dpovolotskiy]

What happened:
At the end of measuring scheduler metrics in a multi-master cluster, I got empty results for scheduler latency.

What you expected to happen:
I expect to get some meaningful metrics from this measurement.

How to reproduce it (as minimally and precisely as possible):
To reproduce you need to restart active scheduler (in my case I change manifest of scheduler) and need that scheduler from other node became active scheduler. In this case clusterloader2 will take wrong node to scrape metric from and you will get empty values of scheduler latency.

Anything else we need to know?:
I investigated this issue and found that this bug is related to choosing masterName. When clusterloader chooses masterName it takes first ControlPlane node from node list and returns its name, but active scheduler may locate on other ControlPlane node.
My suggestion is to use Leases resources as a way to find proper node on which active scheduler is placed.
If it's possible I'm ready to implement solution for this bug.

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten