Skip to content

Fixed ResourceLockTest.revalidateLockOnDifferentSession()#11885

Merged
lhotari merged 3 commits intoapache:masterfrom
merlimat:fix-resource-lock-test
Sep 1, 2021
Merged

Fixed ResourceLockTest.revalidateLockOnDifferentSession()#11885
lhotari merged 3 commits intoapache:masterfrom
merlimat:fix-resource-lock-test

Conversation

@merlimat
Copy link
Contributor

@merlimat merlimat commented Sep 1, 2021

Motivation

Fixes #11690

The main issue in the test is that there are 2 ResourceLock instances that are competing.

  • RL_1 acquires the lock
  • RL_2 tries to “steal” the lock
  • When RL_2 deletes the existing lock, RL_1 might get the notification fast and delete the lock after RL_2 did acquire it.

That eventually lead in both locks getting expired:

10:22:34.784 INFO  [main] o.a.p.m.c.i.ResourceLockImpl@68 - CREATED RESOURCE LOCK [RL 1 ]  - 
10:22:34.807 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@184 - [RL 1 ]  - Acquired resource lock on /key-193314812191875
----------------------- START --------------------
10:22:34.807 INFO  [main] o.a.p.m.c.i.ResourceLockImpl@68 - CREATED RESOURCE LOCK [RL 2 ]  - 
10:22:34.831 WARN  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@149 - [RL 2 ] FAILED TO ACQUIRE LOCK WITH NO REVALIDATION: 
10:22:34.833 WARN  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@151 - [RL 2 ] REVALIDATING LOCK 
10:22:34.837 INFO  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@253 - [RL 2 ]  - Deleting stale lock at /key-193314812191875
10:22:34.862 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@205 -  [RL 1 ]  - Lock on resource /key-193314812191875 was invalidated
10:22:34.884 INFO  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@184 - [RL 2 ]  - Acquired resource lock on /key-193314812191875
10:22:34.884 INFO  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@258 - [RL 2 ]  - Successfully re-acquired stale lock at /key-193314812191875
----------------------- DONE --------------------
10:22:34.885 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@253 -  [RL 1 ]  - Deleting stale lock at /key-193314812191875
10:22:34.912 INFO  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@205 - [RL 2 ]  - Lock on resource /key-193314812191875 was invalidated
10:22:34.912 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@205 -  [RL 1 ]  - Lock on resource /key-193314812191875 was invalidated
10:22:34.942 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@184 -  [RL 1 ]  - Acquired resource lock on /key-193314812191875
10:22:34.943 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@258 -  [RL 1 ]  - Successfully re-acquired stale lock at /key-193314812191875
10:22:34.943 INFO  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@207 -  [RL 1 ]  - Successfully revalidated the lock on /key-193314812191875
10:22:34.963 WARN  [metadata-store-9-1] o.a.p.m.c.i.ResourceLockImpl@210 -  [RL 1 ]  - Failed to revalidate the lock at /key-193314812191875. Marked as expired
10:22:34.963 WARN  [metadata-store-11-1] o.a.p.m.c.i.ResourceLockImpl@210 - [RL 2 ]  - Failed to revalidate the lock at /key-193314812191875. Marked as expired

Modification

  • Avoid the competing ResourceLock instances. Instead simulate the case of existing lock left by previous broker instance.

@merlimat merlimat added type/flaky-tests doc-not-needed Your PR changes do not impact docs labels Sep 1, 2021
@merlimat merlimat added this to the 2.9.0 milestone Sep 1, 2021
@merlimat merlimat self-assigned this Sep 1, 2021
@lhotari lhotari merged commit f42dd55 into apache:master Sep 1, 2021
@merlimat merlimat deleted the fix-resource-lock-test branch September 1, 2021 23:06
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
* Fixed ResourceLockTest.revalidateLockOnDifferentSession()

* Fixed merge conflict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs release/2.9.0 type/flaky-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky-test: LockManagerTest.revalidateLockOnDifferentSession

3 participants