Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement backoff manager #87829

Merged
merged 1 commit into from Feb 7, 2020
Merged

Conversation

@zhan849
Copy link
Contributor

zhan849 commented Feb 4, 2020

What type of PR is this?
/kind feature

What this PR does / why we need it:
per #87795 and #87794, it seems that a component manages backoff could be generally useful. This PR adds backup manager interface in "k8s.io/apimachinery/pkg/util/wait":

interface:

// BackoffManager manages backoff with a particular scheme based on its underlying implementation. It provides
// an interface to return a timer for backoff, and caller shall backoff until Timer.C returns. If the second Backoff()
// is called before the timer from the first Backoff() call finishes, the first timer will NOT be drained.
// The BackoffManager is supposed to be called in a single-threaded environment.
type BackoffManager interface {
	Backoff() clock.Timer
}

exponential backoff manager:

  1. does jittered exponential backoff with cap
  2. reset backoff after not backoff-ed for "a while"
// NewExponentialBackoffManager returns a manager for managing exponential backoff. Each backoff is jittered and
// backoff will not exceed the given max. If the backoff is not called within resetDuration, the backoff is reset.
// This backoff manager is used to reduce load during upstream unhealthiness.
func NewExponentialBackoffManager(initBackoff, maxBackoff, resetDuration time.Duration, backoffFactor, jitter float64, c clock.Clock) BackoffManager

jittered backoff manager

  1. backoffs with given duration and jitter
// NewJitteredBackoffManager returns a BackoffManager that backoffs with given duration plus given jitter. If the jitter
// is negative, backoff will not be jittered.
func NewJitteredBackoffManager(duration time.Duration, jitter float64, c clock.Clock) BackoffManager

This would be useful to reduce load for components where upstream.

Also, refactored JitterUntil so we can re-use the logic for exponential backoff

/assign @lavalamp
/cc @wojtek-t @liggitt @smarterclayton

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
Open questions: currently I'm defining the interface as, if the 2nd Backoff() gets called before the 1st Backoff() finishes, timer will NOT be drained, but reset directly. Thoughts about we drain timer before issuing the next backoff?

Does this PR introduce a user-facing change?:

Introduced BackoffManager interface for backoff management
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 4, 2020

Hi @zhan849. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@lavalamp

This comment has been minimized.

Copy link
Member

lavalamp commented Feb 4, 2020

/ok-to-test

@zhan849

This comment has been minimized.

Copy link
Contributor Author

zhan849 commented Feb 4, 2020

/retest

@zhan849 zhan849 force-pushed the zhan849:harry/backoff-manager branch from 3d89e02 to fc8a39d Feb 4, 2020
@zhan849

This comment has been minimized.

Copy link
Contributor Author

zhan849 commented Feb 5, 2020

seems all failed tests are infra related (OutofPods, etc). could be glitches

/retest

@zhan849

This comment has been minimized.

Copy link
Contributor Author

zhan849 commented Feb 5, 2020

failure of pull-kubernetes-e2e-kind-ipv6 not likely to be related, but
/retest

@zhan849

This comment has been minimized.

Copy link
Contributor Author

zhan849 commented Feb 5, 2020

/retest

@wojtek-t

This comment has been minimized.

Copy link
Member

wojtek-t commented Feb 5, 2020

That looks cool - thanks

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 5, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wojtek-t, zhan849

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit f0bd622 into kubernetes:master Feb 7, 2020
16 checks passed
16 checks passed
cla/linuxfoundation zhan849 authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-e2e-kind-ipv6 Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.