fix: exclude non-ready nodes from azure load balancer

[ricky-rav]

What type of PR is this?

/kind bug
/kind regression

What this PR does / why we need it:

fix: exclude non-ready nodes and deleted nodes from azure load balancers

Make sure that nodes that are not in the ready state and are not newly created (i.e. not having the "node.cloudprovider.kubernetes.io/uninitialized" taint) get removed from load balancers.
Also remove nodes that are being deleted from the cluster.

Ref: kubernetes-sigs/cloud-provider-azure#1195

Which issue(s) this PR fixes:

Fixes #108283

/priority critical-urgent
/sig cloud-provider
/area provider/azure
cc @feiskyer
cc @nilo19

NONE

[k8s-ci-robot]

Hi @ricky-rav. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vpickard]

/cc @deads2k

[vpickard]

/ok-to-test

[deads2k]

/priority important-soon

[andyzhangx]

/lgtm
Need approval from @andrewsykim @cheftako

[andrewsykim]

In general I'm not a fan of using a separate node cache to filter nodes that are out of band from the node filtering service controller is already doing. It will also become harder to reason as the behaviors deviate over time like this PR is doing.

Approving since this is a bug fix and the separate node cache/filter already exists. I have not reviewed the azure specific changes in a lot of depth though, just approving that this is addressing a bug.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, ricky-rav

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/legacy-cloud-providers/OWNERS [andrewsykim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andrewsykim]

(I'm not sure this needs to go into v1.24 though, a cherry-pick afterwards might be better but will defer to azure subproject leads on that)

[nilo19]

@feiskyer do we need this on 1.24?

[ricky-rav]

@nilo19 we most definitely need it in 1.24 and then I'll have to backport it to previous releases.

[deads2k]

1.24.0 is about fully baked. Adding to 1.24.z, I'm in favor of