New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support of Azure standard load balancer and public IP #61884

Merged
merged 2 commits into from Mar 30, 2018

Conversation

@feiskyer
Member

feiskyer commented Mar 29, 2018

What this PR does / why we need it:

This PR adds a new configure loadBalancerSku and excludeMasterFromStandardLB to support Azure standard load balancer and public IP.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
First step of #60485

Special notes for your reviewer:

This is the first step of #60485, and only VMAS is supported yet. VMSS support will be added in a following PR.

Release note:

Azure cloud provider now supports standard SKU load balancer and public IP. To use it, set cloud provider config with
{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, it will be default to true, which means master nodes are excluded to the backend of standard LB.

Also note standard load balancer doesn't work with annotation `service.beta.kubernetes.io/azure-load-balancer-mode`. This is because all nodes (except master) are added as the LB backends.

@feiskyer feiskyer requested review from karataliu and brendandburns and removed request for karataliu Mar 29, 2018

@feiskyer

This comment has been minimized.

Member

feiskyer commented Mar 29, 2018

/sig azure

@feiskyer

This comment has been minimized.

Member

feiskyer commented Mar 30, 2018

/retest

matches := backendPoolIDRE.FindStringSubmatch(backendPool)
if len(matches) == 2 {
lbName := matches[1]
if strings.HasSuffix(lbName, InternalLoadBalancerNameSuffix) == isInternal {

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

It is possible to pull master nodes into the pool, if it only has public/internal LB.

This comment has been minimized.

@feiskyer

feiskyer Mar 30, 2018

Member

Added a check before calling ensureHostInPool

@feiskyer

This comment has been minimized.

Member

feiskyer commented Mar 30, 2018

@karataliu As discussed offline, added two configs now. PTAL

{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, master nodes are also added to the backend of standard LB. This is useful for small clusters, e.g. 3 nodes with HA, in which case master nodes are also running workload Pods.

LoadBalancerSku string `json:"loadBalancerSku" yaml:"loadBalancerSku"`
// ExcludeMasterFromStandardLB excludes master nodes from standard load balancer.
ExcludeMasterFromStandardLB bool `json:"excludeMasterFromStandardLB" yaml:"excludeMasterFromStandardLB"`

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

Should assign default values to those 2 somewhere. ExcludeMasterFromStandardLB should default to true?

This comment has been minimized.

@feiskyer

feiskyer Mar 30, 2018

Member

ExcludeMasterFromStandardLB is bool, so it is default to false. This works for all deployments

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

Consider using pointers thus it'll be easy to change default value

ref: Azure/acs-engine#1584 (review)

@@ -242,3 +243,11 @@ func (az *Cloud) newRouteTableCache() (*timedCache, error) {
return newTimedcache(rtCacheTTL, getter)
}
func (az *Cloud) useStandardLoadBalancer() bool {
if strings.ToLower(az.LoadBalancerSku) == loadBalancerSkuStandard {

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

can return strings.EqualFold(az.LoadBalancerSku,loadBalancerSkuStandard) directly

@@ -123,7 +130,7 @@ func (az *Cloud) mapLoadBalancerNameToVMSet(lbName string, clusterName string) (
// This would be the name for Azure LoadBalancer resource.
func (az *Cloud) getLoadBalancerName(clusterName string, vmSetName string, isInternal bool) string {
lbNamePrefix := vmSetName
if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) {
if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) || az.useStandardLoadBalancer() {

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

Anywhere to add hint that selectLoadBalancer does not work for StandardLoadBalancer?

This comment has been minimized.

@feiskyer

feiskyer Mar 30, 2018

Member

Good catch, will add this check in getLoadBalancer()

},
{
description: "non-default standard external LB should map to empty string",
lbName: "azuretest-internal",

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

external LB ?

}
for _, c := range cases {
if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

should set LoadBalancerSku back for !useStandardLB. This works now since the Standard SKU cases are in the end.

for _, c := range cases {
if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

Also, switch back for !useStandardLB

if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}
vmset := az.getLoadBalancerName(c.clusterName, c.vmSet, c.isInternal)

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

better 'loadbalancerName := '

@feiskyer

This comment has been minimized.

Member

feiskyer commented Mar 30, 2018

@karataliu addressed comments. PTAL

@feiskyer

This comment has been minimized.

Member

feiskyer commented Mar 30, 2018

/retest

@@ -273,6 +284,11 @@ func (az *Cloud) selectLoadBalancer(clusterName string, service *v1.Service, exi
Location: &az.Location,
LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{},
}
if az.useStandardLoadBalancer() {

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

This will not be true now due to ' wantLb && !az.useStandardLoadBalancer()'

This comment has been minimized.

@feiskyer
@@ -623,6 +630,22 @@ func (as *availabilitySet) ensureHostInPool(serviceName string, nodeName types.N
}
}
if !foundPool {
if as.useStandardLoadBalancer() && len(newBackendPools) > 0 {
// Although standard load balancer supports backends from multiple availability
// sets, the same interface couldn't reference more than one load balancer of

This comment has been minimized.

@karataliu

karataliu Mar 30, 2018

Contributor

better to use full name 'interface' -> 'network interface', i got a bit confused when looking at the comment.

This comment has been minimized.

@feiskyer

feiskyer Mar 30, 2018

Member

so are you referring go interface? that make sense

@karataliu

This comment has been minimized.

Contributor

karataliu commented Mar 30, 2018

/lgtm

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Mar 30, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, karataliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Mar 30, 2018

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Mar 30, 2018

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit 3cf80b1 into kubernetes:master Mar 30, 2018

13 of 14 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-e2e-gce
Details
cla/linuxfoundation feiskyer authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details

@feiskyer feiskyer deleted the feiskyer:standard branch Mar 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment