lease not found on one member of cluster after restart

[AALOON]

Hi.
We have a problem. After unexpected restart service/reboot one member of the cluster, this member returns to the cluster and missing a lot of leases and as a result a lot of keys (lease not found on add/update and not found on get keys). On other hand other two members work properly and have such leases => keys.
Our configuration is 3 members cluster, etcd version 3.4.13
From clent side is Include="dotnet-etcd" Version="4.1.1"

Usage flow:

1. creating leases
2. creating and updating some keys with such lease
3. reding some keys

Logs from members of cluster.

logs.zip

[ptabor]

@tangcong Might be the same issue as the one addressed in #12531 ?

[jiapeish]

Could you try this PR #12531 and see if this issue still occurs?

[AALOON]

@jiapeish It didn't work.

[tangcong]

@AALOON can you provide more detail info to reproduce it ? it does not seem to be the same issue as #12531 .

[AALOON]

@tangcong Hi.
I couldn't reproduce it by synthetic test, but we had such flow:

1. We create two leases for each session with it is timeout (from few minutes to few days). id = 2, lease1 = 2, lease2 = -2 (we do not keep alive leases, leases revoked (remove) in the end of session or by expiration)
2. During session We Create keys with specified leases depending on business logic.
3. During session Using this keys , search, lookup, update, delete
4. but after restart one of instance cluster, it returns, but cluster inconsistent
5. two instances have keys and leases. the one which was restarted does not have some of leases
6. but interesting thing some of keys (not all) is presented in this instance if search them without specify lease and I think problem is here.

For now we just removed usage of leases from our application. And it works fine. But we have to manage expiration logic and some other logic by application.
Seems to me leases not synced by raft or revoked on start of broken instance.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.