LeaderElection logic stops after a leader failed to update a lock #1155
Comments
|
I've hit the same error. My situation is that I want to use leader election as kind of a cluster lock, where the election duration is quite short. After serveral turns of lock/unlock actions, the same error " the object has been modified; please apply your changes to the latest version and try again" was found. I believe it can be caused by the ctx in 'tryAcquireOrRenew' is canceled by caller before the |
|
I update the example to reproduce these issue. package main
import (
"context"
"flag"
"log"
"os"
"os/signal"
"syscall"
"time"
"github.com/google/uuid"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/tools/leaderelection"
"k8s.io/client-go/tools/leaderelection/resourcelock"
"k8s.io/klog/v2"
)
func buildConfig(kubeconfig string) (*rest.Config, error) {
if kubeconfig != "" {
cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
return nil, err
}
return cfg, nil
}
cfg, err := rest.InClusterConfig()
if err != nil {
return nil, err
}
return cfg, nil
}
func main() {
klog.InitFlags(nil)
var kubeconfig string
var leaseLockName string
var leaseLockNamespace string
var id string
flag.StringVar(&kubeconfig, "kubeconfig", "", "absolute path to the kubeconfig file")
flag.StringVar(&id, "id", uuid.New().String(), "the holder identity name")
flag.StringVar(&leaseLockName, "lease-lock-name", "", "the lease lock resource name")
flag.StringVar(&leaseLockNamespace, "lease-lock-namespace", "", "the lease lock resource namespace")
flag.Parse()
if leaseLockName == "" {
klog.Fatal("unable to get lease lock resource name (missing lease-lock-name flag).")
}
if leaseLockNamespace == "" {
klog.Fatal("unable to get lease lock resource namespace (missing lease-lock-namespace flag).")
}
// leader election uses the Kubernetes API by writing to a
// lock object, which can be a LeaseLock object (preferred),
// a ConfigMap, or an Endpoints (deprecated) object.
// Conflicting writes are detected and each client handles those actions
// independently.
config, err := buildConfig(kubeconfig)
if err != nil {
klog.Fatal(err)
}
client := clientset.NewForConfigOrDie(config)
run := func(ctx context.Context) {
// complete your controller loop here
klog.Info("Controller loop...")
select {}
}
// use a Go context so we can tell the leaderelection code when we
// want to step down
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// listen for interrupts or the Linux SIGTERM signal and cancel
// our context, which the leader election code will observe and
// step down
ch := make(chan os.Signal, 1)
signal.Notify(ch, os.Interrupt, syscall.SIGTERM)
go func() {
<-ch
klog.Info("Received termination, signaling shutdown")
cancel()
}()
// we use the Lease lock type since edits to Leases are less common
// and fewer objects in the cluster watch "all Leases".
lock := &resourcelock.LeaseLock{
LeaseMeta: metav1.ObjectMeta{
Name: leaseLockName,
Namespace: leaseLockNamespace,
},
Client: client.CoordinationV1(),
LockConfig: resourcelock.ResourceLockConfig{
Identity: id,
},
}
// start the leader election code loop
wait := make(chan struct{}, 1)
go func() {
leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
Lock: lock,
// IMPORTANT: you MUST ensure that any code you have that
// is protected by the lease must terminate **before**
// you call cancel. Otherwise, you could have a background
// loop still running and another process could
// get elected before your background loop finished, violating
// the stated goal of the lease.
ReleaseOnCancel: true,
LeaseDuration: 60 * time.Second,
RenewDeadline: 15 * time.Second,
RetryPeriod: 5 * time.Second,
Callbacks: leaderelection.LeaderCallbacks{
OnStartedLeading: func(ctx context.Context) {
// we're notified when we start - this is where you would
// usually put your code
run(ctx)
},
OnStoppedLeading: func() {
// we can do cleanup here
klog.Infof("leader lost: %s", id)
},
OnNewLeader: func(identity string) {
// we're notified when new leader elected
if identity == id {
// I just got the lock
log.Print("Got the lock")
wait <- struct{}{}
return
}
klog.Infof("new leader elected: %s", identity)
},
},
})
}()
<-wait
// Sleep a while depends on the network, 1 node cluster needn't this.
time.Sleep(3 * time.Millisecond)
cancel()
// sleep to give `release` a chance to run.
time.Sleep(300 * time.Millisecond)
}With below command in a k8s node, 1 out of 5 can fail. |
|
Let me see if I understand your use case, you want to get the Leader as a Lock and then release it quickly, so you use the context to do that, but when you use the context it also cancels the subsequent operations , is that right? |
Yeah, pretty so. The real action protected by the lock is done in the leader callback with another context. I just cancel the Maybe it's not a normal case where the leaderelection package used. If so, it's O.K. not to treat it as a problem, since all the kubernetes components which are using the package haven't reported any related issue yet. |
It doesn't look like the normal use case, basically it seems that what you want is a new method I think that what you want can be done better using the Lease API https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/, but maybe having a method to stop leading without killing yourself in the process by cancelling the context is something we need 🤔 Let me ask @deads2k @wojtek-t and @liggitt , they have more historical context and it will be interesting to hear their thoughts |
|
I am happy that this issue is finally progressed. It's not a fundamental solution, but I share my temporary method for someone who needs to fix this situation urgently or cannot change package version. You can re-leader-election using defer func() like below. Make function that covers leaderelection.RunOrDie(runLeaderElection in above example) with defer func(). In defer func(), run the function itself again. This can help someone who needs temporary solution. |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
|
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I imported client-go v0.22.2 and used leader election logic, but I only deployed one pod.
When the pod failed to update the lock for whatever reason, the pod does not run OnStartedLeading() callback forever even though that the pod still has 'lease' resource.
My leader election code is below.
Logs printed like below.
You can definitely see that OnStoppedLeading() callback is called.
However, there is no new leader election after above logs.
Interesting point is that the pod still has lease!
I doubt Update() function in leaderelection.go.
If the leader fails to Update() in release() function, I think it just return false and got stuck.
What is the problem? How can I fix it?
The text was updated successfully, but these errors were encountered: