Bug 1989246: use new default leader election values to handle apiserver rollout on SNO #1175

creydr · 2021-08-04T10:28:45Z

In case of CNO gets deployed on SNO, it can run into leader election timeouts during an api server rollout (as only one api server is available and it doesn't restart fast enough yet).
Therefor the default values for leader election have been updated (openshift/library-go#1104) to the recommended values.

This PR updates the version of the openshift/library-go library to use these new values and being able to handle a 60 seconds downtime of the api server on SNO.

How to reproduce / validate:

Start a SNO cluster with this patch (e.g. via clusterbot launch 4.9.0-0.nightly,openshift/cluster-network-operator#1175 aws,single-node)
Trigger a apiserver rollout (e.g. via k patch kubeapiserver/cluster --type merge -p '{"spec":{"forceRedeploymentReason":"something"}}')
Check the logs from CNO controller pod during the update
Validate the pod did not get restarted cause it lost the lease (e.g. check no previous logs are available / doesn't contain leaderelection lost

Special notes for reviewers:
The important file is vendor/github.com/openshift/library-go/pkg/config/leaderelection/leaderelection.go, which uses the new recommended values:

cluster-network-operator/vendor/github.com/openshift/library-go/pkg/config/leaderelection/leaderelection.go

Lines 96 to 109 in 048145f

    
           if ret.LeaseDuration.Duration == 0 { 
        
           	ret.LeaseDuration.Duration = 137 * time.Second 
        
           } 
        
           if ret.RenewDeadline.Duration == 0 { 
        
           	// this gives 107/26=4 retries and allows for 137-107=30 seconds of clock skew 
        
           	// if the kube-apiserver is unavailable for 60s starting just before t=26 (the first renew), 
        
           	// then we will retry on 26s intervals until t=104 (kube-apiserver came back up at 86), and there will 
        
           	// be 33 seconds of extra time before the lease is lost. 
        
           	ret.RenewDeadline.Duration = 107 * time.Second 
        
           } 
        
           if ret.RetryPeriod.Duration == 0 { 
        
           	ret.RetryPeriod.Duration = 26 * time.Second 
        
           }

openshift-ci · 2021-08-04T10:28:51Z

@creydr: This pull request references Bugzilla bug 1989246, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

Bug 1989246: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-04T10:28:54Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2021-08-04T10:29:58Z

@creydr: This pull request references Bugzilla bug 1989246, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

Bug 1989246: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-04T10:30:40Z

/test

openshift-ci · 2021-08-04T10:30:48Z

@creydr: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test e2e-agnostic-upgrade
/test e2e-aws-ovn-windows
/test e2e-aws-sdn-multi
/test e2e-gcp
/test e2e-gcp-ovn
/test e2e-metal-ipi-ovn-ipv6
/test images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test e2e-azure-ovn
/test e2e-azure-ovn-dualstack
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6-ipsec
/test e2e-openstack
/test e2e-openstack-kuryr
/test e2e-openstack-ovn
/test e2e-ovn-hybrid-step-registry
/test e2e-ovn-ipsec-step-registry
/test e2e-ovn-step-registry
/test e2e-vsphere-ovn
/test e2e-vsphere-windows

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-network-operator-master-e2e-agnostic-upgrade
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-windows
pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-multi
pull-ci-openshift-cluster-network-operator-master-e2e-azure-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-gcp
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn-upgrade
pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6-ipsec
pull-ci-openshift-cluster-network-operator-master-e2e-openstack-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-hybrid-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-ipsec-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-windows
pull-ci-openshift-cluster-network-operator-master-images
pull-ci-openshift-cluster-network-operator-master-unit
pull-ci-openshift-cluster-network-operator-master-verify

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-04T10:31:05Z

/test all

creydr · 2021-08-05T09:16:40Z

/retest

creydr · 2021-08-05T13:06:45Z

/retest-required

creydr · 2021-08-06T07:20:18Z

@zhaozhanqi, @alexanderConstantinescu, @trozet could you please review this PR?
Thanks

openshift-ci · 2021-08-06T07:21:40Z

@creydr: This pull request references Bugzilla bug 1989246, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

Bug 1989246: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-06T10:03:30Z

/retest

creydr · 2021-08-06T13:14:24Z

/retest-required

creydr · 2021-08-08T14:24:50Z

/retest

openshift-ci · 2021-08-09T06:55:34Z

@creydr: An error was encountered searching for bug 1989246 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


could not unmarshal response body: invalid character '<' looking for beginning of value

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

Bug 1989246: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-09T06:57:50Z

/bugzilla refresh

openshift-ci · 2021-08-09T06:57:53Z

@creydr: An error was encountered searching for bug 1989246 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


could not unmarshal response body: invalid character '<' looking for beginning of value

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

petr-muller · 2021-08-09T09:09:23Z

/bugzilla refresh

openshift-ci · 2021-08-09T09:09:25Z

@petr-muller: An error was encountered searching for bug 1989246 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


could not unmarshal response body: invalid character '<' looking for beginning of value

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-09T12:15:32Z

/retest-required

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

creydr · 2021-08-12T15:58:02Z

/retest

creydr · 2021-08-12T17:54:09Z

/retest

creydr · 2021-08-12T20:04:58Z

/retest

creydr · 2021-08-13T03:01:10Z

/retest

creydr · 2021-08-13T17:49:23Z

/retest-required

creydr · 2021-08-14T17:18:20Z

/retest

creydr · 2021-08-15T07:04:46Z

/retest

abhat · 2021-08-16T21:52:57Z

/lgtm
/approve

openshift-ci · 2021-08-16T21:55:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, alexanderConstantinescu, creydr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

creydr · 2021-08-17T06:24:31Z

/retest

creydr · 2021-08-17T07:54:26Z

/retest-required

openshift-ci · 2021-08-17T09:27:12Z

@creydr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-openstack-ovn	`4bf24b6`	link	`/test e2e-openstack-ovn`
ci/prow/e2e-gcp-ovn-upgrade	`4bf24b6`	link	`/test e2e-gcp-ovn-upgrade`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

creydr · 2021-08-17T09:31:03Z

/retest-required

openshift-bot · 2021-08-17T10:01:58Z

/retest-required