Exclude master from LoadBalancer / NodePort

[justinsb]

The servicecontroller documents that the master is excluded from the
LoadBalancer / NodePort, but this is broken for clusters where we are
using taints for the master (as introduced in 1.6), instead of marking
the master as unschedulable.

This restores the desired documented behaviour, by excluding nodes that
are labeled as masters with the new 1.6 labels, even if they use the new
1.6 taints.

Fix #33884

Exclude nodes labeled as master from LoadBalancer / NodePort; restores documented behaviour.

[k8s-reviewable]

This change is 

[MrHohn]

/lgtm

[eghobo]

@justinsb: are you planning back port it to 1.6 release?

[justinsb]

Surprisingly difficult to see the failure in that test run - I think it is a statefulset, so going to rerun

@k8s-bot cvm gce e2e test this

[justinsb]

@thockin do you think you could have a look and make sure you agree that this is a regression, and if so /approve?

[thockin]

/approve

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MrHohn, justinsb, thockin

Needs approval from an approver in each of these OWNERS Files:

• pkg/controller/OWNERS [thockin]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[justinsb]

@k8s-bot unit test this

[k8s-github-robot]

Automatic merge from submit-queue

[justinsb]

Going to propose for cherry-pick (cc @eghobo) - it's a nuisance on AWS, but it's actually a serious problem on GCE, where we only open the firewall rule to the nodes and we don't have a health check.... but the master is added. Not sure how this got through e2e.

[k8s-cherrypick-bot]

Commit found in the "release-1.6" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

[Sprinkle]

Tried this in 1.6.3 and master node is still added to ELB on AWS.

@justinsb Could it be in servicecontroller.go, createLoadBalancer is calling includeNodeFromNodeList which is still only doing return !node.Spec.Unschedulable?

[justinsb]

@Sprinkle you're absolutely right. Going to open another issue to track this - I have no idea why we have two functions (which is why I want to give it its own issue, in case there is a reason!).

[justinsb]

Issue #45772

[tomfotherby]

Tried this in 1.7.0 and master nodes are no longer added to ELB on AWS. (Kubernetes installer: https://github.com/kz8s/tack) 👍