Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ETCD lock example #10096

Closed
sanamsarath opened this issue Sep 15, 2018 · 13 comments
Closed

ETCD lock example #10096

sanamsarath opened this issue Sep 15, 2018 · 13 comments

Comments

@sanamsarath
Copy link

@sanamsarath sanamsarath commented Sep 15, 2018

Can someone provide the exact use case of lock feature in ETCD.

How can I use the lock id generated after lock request, should I have to include that id in the atomic operations like put, get or delete?? also want to know if lock applies to the entire key value store or if a single key can be locked and updates on other keys can be performed in parallel??

Thanks.

@wenjiaswe

This comment has been minimized.

Copy link
Contributor

@wenjiaswe wenjiaswe commented Sep 17, 2018

@jingyih

This comment has been minimized.

Copy link
Member

@jingyih jingyih commented Sep 17, 2018

Etcd lock service API is documented here.

And there is an example of using etcdctl lock command.

To answer your question, I do not think a lock id should be associated with a specific etcd key, or an etcd command. My understanding is that etcd lock is a service that can be used by users(or other systems) for protecting access to whatever resource they wanted to protect (not necessarily any resource in etcd database), something like:
1. acquire an etcd lock
2. do something (again, not necessarily related to etcd database)
3. unlock the same etcd lock

After a closer look at our current etcdctl lock command, I did not find a way to create a demo flow I mentioned above. The current behavior of etcdctl lock is that it tries to acquire the lock, blocks itself if the lock is already locked by some other user. If lock is acquired, upon hitting the Ctrl-C, it unlocks the acquired lock. I did not find a way to add operations between locking and unlocking.

@gyuho Do you have any comment on this topic?

/cc @wenjiaswe

@gyuho

This comment has been minimized.

Copy link
Member

@gyuho gyuho commented Sep 17, 2018

@sanamsarath @jingyih @wenjiaswe You can create a session with Go client and simulate concurrency as follows:

ss1, err := concurrency.NewSession(cli, concurrency.WithContext(ctx))
if err != nil {
    log.Fatal(err)
}
defer ss1.Close()
mu1 := concurrency.NewMutex(ss1, "/my-lock/")
fmt.Println("Lock 1")
fmt.Println(mu1.Lock(context.Background()))
fmt.Println("Lock 2")

doCancel := func() {
    fmt.Println("cancel 1")
    cancel()
    fmt.Println("cancel 2")
    mu1.Unlock(context.Background())
}

go func() {
    time.Sleep(5 * time.Second)
    doCancel()
}()

ss2, err := concurrency.NewSession(cli, concurrency.WithContext(ctx))
if err != nil {
    log.Fatal(err)
}
defer ss2.Close()
mu2 := concurrency.NewMutex(ss2, "/my-lock/")
fmt.Println("Lock 3")
fmt.Println(mu2.Lock(context.Background()))
fmt.Println("Lock 4")
Lock 1
<nil>
Lock 2
Lock 3
cancel 1
cancel 2
<nil>
Lock 4
@sanamsarath

This comment has been minimized.

Copy link
Author

@sanamsarath sanamsarath commented Sep 20, 2018

@jingyih @gyuho Thanks for the explanation and example.

Just want to be clear about the following, as I am trying to implement the c++ client library for etcd

mu1 := concurrency.NewMutex(ss1, "/my-lock/") - > here you are acquiring a lock.

mu1.Lock(context.Background()) -> does this method use the already acquired lock id to do some
operations on etcd server?? what does background function does??

mu1.Unlock(context.Background()) -> And this method sends the unlock request with lock id??

@sanamsarath

This comment has been minimized.

Copy link
Author

@sanamsarath sanamsarath commented Sep 20, 2018

@gyuho

lease generated for a lock is never expiring,

lock acquired using command line:

[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lock mylock --ttl=20
mylock/52dd65ee79d88077

lease timetolive is not terminating:

[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(19s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(16s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(14s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(19s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(18s)ive 52dd65ee79d88077
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]#
[root@db-etcd-5f6775c7c8-r2gts bin]# ./etcdctl lease timetolive 52dd65ee79d88077
lease 52dd65ee79d88077 granted with TTL(20s), remaining(16s)ive 52dd65ee79d88077

@hexfusion

This comment has been minimized.

Copy link
Member

@hexfusion hexfusion commented Sep 20, 2018

lease generated for a lock is never expiring,

@sakshamsharma etcd --version etcdctl version ?

@hexfusion

This comment has been minimized.

Copy link
Member

@hexfusion hexfusion commented Sep 20, 2018

I guess it is possible if new leader is elected during this test TTL could be reset. Can you reproduce this?

@jingyih

This comment has been minimized.

Copy link
Member

@jingyih jingyih commented Sep 20, 2018

maybe related to this
kubernetes/kubernetes#65497

@sanamsarath

This comment has been minimized.

Copy link
Author

@sanamsarath sanamsarath commented Sep 21, 2018

@hexfusion version 3.3 and I have only 1 etcd node.

@hexfusion

This comment has been minimized.

Copy link
Member

@hexfusion hexfusion commented Sep 21, 2018

@sanamsarath 3.3.x? please output etcd --version so you're saying it is reproducible?

@sanamsarath

This comment has been minimized.

Copy link
Author

@sanamsarath sanamsarath commented Sep 21, 2018

@hexfusion sorry about that, i was lazy.

etcd Version: 3.3.2+git
Git SHA: 227c3e5
Go Version: go1.10.3
Go OS/Arch: linux/amd64

yes it is reproducile.

@jingyih

This comment has been minimized.

Copy link
Member

@jingyih jingyih commented Sep 21, 2018

@sanamsarath
I think this is the intended behavior. Since you are still holding the lock, the lease gets auto renewed. If you kill the process that is holding the lock, or if there is network partition between this machine you are on and the etcd server, the lease will expire. The API doc mentioned this behavior:

On success, it will return a unique key that exists so long as the lock is held by the caller.

The lock is held until Unlock is called on the key or the lease associate with the owner expires.

We might need to do better documentation.

@sanamsarath

This comment has been minimized.

Copy link
Author

@sanamsarath sanamsarath commented Sep 22, 2018

@jingyih Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.