feat: support load balancer choosing logic for multi-slb

[nilo19]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Support load balancer choosing logic for multi-slb feature. Major changes:

1. Add a new configuration option multipleStandardLoadBalancerConfigurations and a new service annotation service.beta.kubernetes.io/azure-load-balancer-configurations: lb1,lb2.
2. Change the logic of ListManagedLoadBalancers. The current LB filtering logic is only useful for basic LB clusters. Add a new logic for multi-slb. For single-slb, managed LBs only be <clustername> and .
3. Implement the multi-lb selection algorithm. It reads service annotation and lb configs to select the most eligible LB for the service. If the LB does not exist, it will be created. If the last service on the LB is removed or moved to another LB, the LB will be deleted.

Which issue(s) this PR fixes:

Fixes #
Related #4013

Special notes for your reviewer:

Does this PR introduce a user-facing change?

feat: support load balancer choosing logic for multi-slb

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nilo19

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nilo19]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for kubernetes-sigs-cloud-provide-azure canceled.

Name	Link
🔨 Latest commit	9bee28d
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/64880961c1f2f60008ce0678

[lzhecheng]

Add a UT?

[nilo19]

It has been covered in other tests.

[lzhecheng]

I see, the TestGetEligibleLoadBalancers. However, a UT for this function is more straightforward and raises coverage.
Anyway, it's up to you and I don't have strong opinion on this.

[nilo19]

Thanks for the suggestions. All lines have been covered.

[lzhecheng]

How about return error if slice of strings.Split(ruleName, "-") is empty?

[nilo19]

It is not possible to be empty for managed rules. For unmanaged rules, I don't think we need to block this if the user adds a rule to the LB.

[lzhecheng]

Defensive programming.
For managed rules, I do believe these should not be empty but it is possible there's something wrong when calling API.
For unmanaged rules, CCM may panic and lose logs. So I think a check here will save our effort if cx needs us debug.

[nilo19]

Added a warning here. We can't break the reconciliation just because there are unmanaged rules.

[lzhecheng]

How about:

errMsg := fmt.Errorf("reconcileLoadBalancer: invalid multiple standard load balancers configurations: %s", err.Error())
klog.Error(errMsg)
return nil, errMsg

[nilo19]

May I know the reason why this is better? To me, this makes the error message longer without adding critical information. However, I think the error log message needs to be changed because there are multiple error possibilities not just invalid configs.

[lzhecheng]

I put the comment because I saw the following change in this PR and I think they should be of the same code style, right?

	svcs, err := az.KubeClient.CoreV1().Services("").List(context.Background(), metav1.ListOptions{})
	if err != nil {
		klog.Errorf("reconcileMultipleStandardLoadBalancerConfigurations: failed to list all load balancer services: %w", err)
		return fmt.Errorf("failed to list all load balancer services: %w", err)
	}

[nilo19]

The original error here is returned by List and the message is not under my controller so I enrich the error message.

[MartinForReal]

/retest

[lzhecheng]

/lgtm