Optimistically update leader lock

[linxiulei]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Client-go: Optimized leaders renewing leases by updating leader lock optimistically without getting the record from the apiserver first. Also added a new metric `leader_election_slowpath_total` that allow users to monitor how many leader elections are updated non-optimistically.

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.29 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.29.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Mon Nov 27 09:58:46 UTC 2023.

[linxiulei]

Partly fix #121806

[linxiulei]

/assign @wojtek-t @aojea

[wojtek-t]

This has in theory unbounded cardinality - we shouldn't make a name to be a label.

Why do we need a name? If we really need to check the name, we should use logs - for metrics we should focus oncount.

OTOH - I see other metrics above to be labeled with name - isn't that an issue?

@dgrisonnet @logicalhan ^^

[logicalhan]

What name is this? If it's a per-cluster (or per control-plane node) name it should be practically bounded and we've traditionally been okay with this.

[wojtek-t]

It's "per component taking part in leader election".

So in general yes - it should be bounded [modulo bugs etc.]. So I'm assuming this is fine for you...

[wojtek-t]

Building a bit on @aojea comment from another PR - changing public interfaces in our official client library is always problematic.
Can we leave the SwitchMetric interface as it was and simply add a new LeaderMetric interface as:

type LeaderMetric interface {
  SwitchMetric
  SlowPathExercised(name string)
}

I'm also thinking about better naming for the method - but don't have a better suggestion.

[wojtek-t]

OTOH - we're changing below MetricsProvider interface so maybe that's an overkill.
I wouldn't expect people to generally overwrite provider factory and we're fixing all implementations.
So maybe that's actually fine as is.

[linxiulei]

FTR, I keep it as it is.

[howardjohn]

Coming back to this: it is problematic and causes another tight coupling on controller-runtime and client-go which provides its own implementation.

If the original name was kept, even adding the new method would have been more compatible, since it would allow a single consumer to support 1.29 and 1.30. By changing the name, it strictly forces 1.30 usage.

[stefanprodan]

This change breaks the upgrade to 1.30 for every controller built on top of controller-runtime e.g. Flux fluxcd/pkg#763. Hopefully the controller-runtime maintainers will do a release soon that deals with this breaking change.

[linxiulei]

Working on #124372 to address this. Will cherry pick to 1.30 after merged.

[jiahuif]

/triage accepted

[linxiulei]

/cc @pohly

[wojtek-t]

The core code LGTM - I added few more comments about the test.

[linxiulei]

Addressed all suggestions and updated release-note

[wojtek-t]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: c9d315df71af36c8b687a357047a80f42ade1ead

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: linxiulei, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/client-go/tools/leaderelection/OWNERS [wojtek-t]
• staging/src/k8s.io/component-base/metrics/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment