Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support setting azure LB idle timeout #66045

Merged
merged 1 commit into from
Jul 13, 2018

Conversation

cpuguy83
Copy link
Contributor

What this PR does / why we need it:

Adds a new annotation to allow users to configure the idle timeout of
the Azure LB.

Release note:

Support configuring the Azure load balancer idle connection timeout for services

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 10, 2018
@cpuguy83
Copy link
Contributor Author

/assign @khenidak

@@ -1280,7 +1304,8 @@ func equalLoadBalancingRulePropertiesFormat(s, t *network.LoadBalancingRulePrope
reflect.DeepEqual(s.LoadDistribution, t.LoadDistribution) &&
reflect.DeepEqual(s.FrontendPort, t.FrontendPort) &&
reflect.DeepEqual(s.BackendPort, t.BackendPort) &&
reflect.DeepEqual(s.EnableFloatingIP, t.EnableFloatingIP)
reflect.DeepEqual(s.EnableFloatingIP, t.EnableFloatingIP) &&
s.IdleTimeoutInMinutes == t.IdleTimeoutInMinutes
Copy link
Contributor Author

@cpuguy83 cpuguy83 Jul 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course this doesn't work as expected because it's a pointer.

@feiskyer
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 11, 2018
@@ -74,6 +74,10 @@ const (
// ServiceAnnotationAllowedServiceTag is the annotation used on the service
// to specify a list of allowed service tags separated by comma
ServiceAnnotationAllowedServiceTag = "service.beta.kubernetes.io/azure-allowed-service-tags"

// ServiceAnnotationLoadBalancerIdleTimeout is the annotation used on the service
// to specify the idle timeout for connections on the load balancer.
Copy link
Member

@feiskyer feiskyer Jul 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timeout is in minutes?

nit: s/the idle timeout/the idle timeout in minutes/

@feiskyer
Copy link
Member

LGTM in general. @cpuguy83 Could you document the timeout unit clearly?

@@ -487,6 +505,11 @@ func (az *Cloud) reconcileLoadBalancer(clusterName string, service *v1.Service,
lbBackendPoolName := getBackendPoolName(clusterName)
lbBackendPoolID := az.getBackendPoolID(lbName, lbBackendPoolName)

lbIdleTimeout, err := getIdleTimeout(service)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what if there is no service.beta.kubernetes.io/azure-load-balancer-idle-timeout setting, it will return nil, nil, are we going to handle that? And what if it's a negative?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want nil, nil in this case as this will set the LB timeout to the default value. I will add a check here to see if the key even exists and only return nil when it does not, and a separate error case for when it does exist but the value is empty.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For validation of the value beyond if it converts to an int... I didn't want to do too much validation here as I expect the backend to deal with this, but I do see value in validating here since they are fairly hard values that are well documented (or at least were easy for me to find), which saves us a failed request.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all updated

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The range is [4,30]

@cpuguy83
Copy link
Contributor Author

/test pull-kubernetes-integration

@@ -74,6 +74,10 @@ const (
// ServiceAnnotationAllowedServiceTag is the annotation used on the service
// to specify a list of allowed service tags separated by comma
ServiceAnnotationAllowedServiceTag = "service.beta.kubernetes.io/azure-allowed-service-tags"

// ServiceAnnotationLoadBalancerIdleTimeout is the annotation used on the service
// to specify the idle timeout for connections on the load balancer in miutes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo miutes

FrontendPort: to.Int32Ptr(port.Port),
BackendPort: to.Int32Ptr(port.Port),
EnableFloatingIP: to.BoolPtr(true),
IdleTimeoutInMinutes: lbIdleTimeout,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IdleTimeoutInMinutes only applies to TCP protocol. Could you also add a check and only set it when transportProto is TCP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, also updated annotation name to clarify that.

Copy link
Member

@andyzhangx andyzhangx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cpuguy83
could you also write a unit test for getIdleTimeout func?

min = 4
max = 30
)
errInvalidTimeout := fmt.Errorf("idle timeout value must be a whole number representing minutes between %d and %d", min, max)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to define a var here, put it under
return nil, fmt.Errorf("idle timeout value must be a whole number representing minutes between %d and %d", min, max)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using this multiple times.

// Return a nil here as this will set the value to the azure default
return nil, nil
}
if val == "" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it necessary to check here since below code would still parse val:

to, err := strconv.Atoi(val)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary, removing this explicit check.

@cpuguy83 cpuguy83 force-pushed the az_lb_timeout branch 2 times, most recently from 8cbf2ca to 8000342 Compare July 12, 2018 17:00
@cpuguy83
Copy link
Contributor Author

/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-kops-aws

Adds a new annotation to allow users to configure the idle timeout of
the Azure LB.
@cpuguy83
Copy link
Contributor Author

/test pull-kubernetes-integration

Copy link
Member

@andyzhangx andyzhangx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
would you take a another look @feiskyer ?

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 13, 2018
@feiskyer
Copy link
Member

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, cpuguy83, feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 66121, 66140, 66045). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 16c5ba4 into kubernetes:master Jul 13, 2018
Copy link
Contributor

@rite2nikhil rite2nikhil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm

@cpuguy83 cpuguy83 deleted the az_lb_timeout branch July 13, 2018 17:22
@feiskyer
Copy link
Member

@cpuguy83 PR got merged. Could you also file a PR to cloud-provider-azure and add docs for the new annotation?

@cpuguy83
Copy link
Contributor Author

@marpaia
Copy link
Contributor

marpaia commented Jul 17, 2018

/sig azure

@djsly
Copy link
Contributor

djsly commented Sep 24, 2018

@cpuguy83 / @feiskyer anyway we could get this in 1.11 ? WE are currently blocked with a client requesting > 4min idle timeout and unfortunately, 1.12 would be too risky to push to production once it comes out.

Thanks!

k8s-ci-robot added a commit that referenced this pull request Oct 4, 2018
…45-upstream-release-1.11

Automated cherry pick of #66045: Support setting azure LB idle timeout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants