Fix ZookeeperLockRegistry test isolation and interrupt handling - #11244
Conversation
`ZkLock.lockInterruptibly()` looped on `tryLock(1, SECONDS)` until it succeeded. Since `tryLock()` returns `false` (rather than throwing) when the Zookeeper connection probe times out, an unreachable server turned that loop into an endless spin with no deadline, no interrupt check and no logging. The build had no test timeout configured at all, so such a hang stalled Gradle silently until the CI job limit. * check `Thread.interrupted()` per iteration in `lockInterruptibly()` and throw `InterruptedException`; log a `DEBUG` message per retry * log a `WARN` in `tryLock()` when the connection probe times out or reports no connection, instead of silently returning `false` * clamp the recomputed `waitTime` with `Math.max(0, ...)` so a slow probe cannot pass a negative deadline to `InterProcessMutex.acquire()` * set `queueCapacity` to `0` on the default `mutexTaskExecutor`: with Spring's defaults (core `1`, unbounded queue) the pool never grew past a single thread, so concurrent `tryLock()` calls serialized behind one connection probe * give `ZkLockRegistryTests.voidLockFailsWhenServerDown()` its own `TestingServer`; it used to stop and restart the one shared by the class, and a failed restart left the six tests declared after it blocked forever in `InterProcessMutex.acquire()` * bound the `maincountDownLatch.await()` in `concurrentObtainCapacityTest()` * add a `junit.jupiter.execution.timeout.default` for all `Test` tasks (`10 m`, `30 m` for `testAll`) so a stuck test fails with a stack trace of its thread **Auto-cherry-pick to `7.1.x` & `7.0.x`**
* use `var` in the try-with-resources of `voidLockFailsWhenServerDown()` * rewrap the retry comment in `lockInterruptibly()` * code style in `ZkLockRegistryTests`
Fixes: #11244 Force-push is disabled on this branch, so the reference cannot be added to the message of the first commit; squash it at merge time.
artembilan
left a comment
There was a problem hiding this comment.
Reviewed the fix — the core logic (interrupt check + logging in lockInterruptibly(), the Math.max(0, ...) clamp on waitTime, the isolated TestingServer for voidLockFailsWhenServerDown(), the bounded latch, and the JUnit execution timeouts) is sound and addresses the described busy-spin/hang.
Two follow-on concerns worth considering, not blockers:
-
Unbounded thread growth:
threadPoolTaskExecutor.setQueueCapacity(0)removes the queue but doesn't boundmaxPoolSize(Spring'sThreadPoolTaskExecutordefault isInteger.MAX_VALUE). With aSynchronousQueueand no cap, every concurrenttryLock()/lockInterruptibly()connection probe that can't be served by an idle thread spawns a brand-new one. During a sustained ZooKeeper outage with many locks retrying concurrently, this could grow the thread count without bound and riskOutOfMemoryError: unable to create native thread. Might be worth setting an explicitmaxPoolSize. -
Log flooding on outage:
tryLock()now logs atWARNon every failed connection probe. SincelockInterruptibly()retries roughly once per second while unreachable, a prolonged outage with active retries will produce continuous per-second WARN lines per lock instead of a one-time signal — exactly when log volume matters most. Consider logging only on state transition (first failure) or throttling.
`tryLock()` logged a `WARN` on every failed connection probe. Since `lockInterruptibly()` retries roughly once per second, a prolonged Zookeeper outage produced a continuous stream of per-second `WARN` lines per lock. A `false` from `tryLock()` is a contract outcome, not an anomaly.
The executor existed only to run a `checkExists().forPath("/")` round-trip
against the server under a timeout, so a `tryLock(time, unit)` would not block
indefinitely on a lost connection. That is a thread (and a ZK round-trip) per
lock attempt, and with `queueCapacity` of `0` and an unbounded `maxPoolSize`
a sustained outage could grow the pool without a limit.
Curator already exposes the connection state without blocking.
* check `CuratorZookeeperClient.isConnected()` instead of submitting a probe
task; wait for a re-connect via `blockUntilConnected()` (interruptibly, and
only within the time requested for `tryLock()`) when not connected
* treat a connection loss, an expired session or an operation timeout from
`InterProcessMutex.acquire()` as a `false`, matching the `Lock.tryLock()`
contract - previously such a failure was hidden behind the probe timeout
* deprecate `setMutexTaskExecutor()` with no replacement
* `destroy()` is a no-op now: the registry does not own any resource anymore
Deprecated for removal since `7.0.6`.
artembilan
left a comment
There was a problem hiding this comment.
Reviewed the full main...fix-zk-busy-spin diff against the actual sources of Curator 5.9.0 and ZooKeeper 3.9.3 (not just the diff). The direction is right and the branch is a net improvement: the per-registry eagerly-created ThreadPoolTaskExecutor was a real liability, the busy-spin in lockInterruptibly() is genuinely fixed for the case that matters, the isolated TestingServer in voidLockFailsWhenServerDown() is a correct fix for a nasty test-ordering hazard, and the deprecation-instead-of-removal call is the right one for a 7.0.6 patch. Two things I would not merge without addressing: the requested tryLock deadline is no longer bounded in the silent connection-loss case (the exact scenario the removed executor existed for), and the connection check mixes two different Curator state machines. Findings below, most severe first.
1. tryLock(time, unit) can block far past time on a silent connection loss — a regression vs. the probe. ZookeeperLockRegistry.java:311-322
client.getZookeeperClient().isConnected() reads curator-client's raw ConnectionState.isConnected AtomicBoolean, which is only flipped when the ZK client notices the connection is gone. On a socket close that is immediate, but on a silent partition (no FIN/RST — the classic ZK outage) ClientCnxn.SendThread only gives up after readTimeout = negotiatedSessionTimeout * 2 / 3 (ClientCnxn.java:1446), i.e. ~40 s with Curator's default 60 s session. For that whole window isConnected() returns true, the guard is skipped, and we fall into this.mutex.acquire(remaining, MILLISECONDS).
That call is not bounded by remaining. InterProcessMutex.acquire → LockInternals.attemptLock → StandardLockInternalsDriver.createsTheLock → CreateBuilderImpl.pathInForeground → RetryLoop.callWithRetry, whose first statement is client.internalBlockUntilConnectedOrTimedOut() (verified in the bytecode of curator-client 5.9.0). That method loops until connected or connectionTimeoutMs elapses — default 15 s (CuratorFrameworkFactory.DEFAULT_CONNECTION_TIMEOUT_MS) — and then just falls through so the ZK op can throw ConnectionLoss, at which point the retry policy sleeps and the whole thing repeats.
Failure scenario: ZK is partitioned away without the socket dropping. lock.tryLock(200, TimeUnit.MILLISECONDS) blocks ~15 s (+ retry-policy sleeps, + a second pass if attemptLock retries its NoNodeException) before returning false. Under the old code the same call returned false in 200 ms, because future.get(waitTime) bounded the probe. This is precisely what the javadoc removed in this commit was talking about: "This must be performed asynchronously so the Lock#tryLock(long, TimeUnit) contract can be honored." Anything polling a lock on a hot path (aggregator/LockRegistryLeaderInitiator) will stall on this.
Note the change is still a clear win for the detected-disconnect case (blockUntilConnected returns false after exactly time, where previously acquire would burn connectionTimeoutMs there too) — the new test covers that path. It is only the stale-state window that regressed.
Suggested fix, in order of preference: (a) if a hard bound is required, the acquisition has to run off-thread, not just the probe — i.e. the executor cannot be dropped outright, only re-scoped; (b) if you accept the trade-off, stop promising a bound you do not deliver: zookeeper.adoc:88-90 currently says the wait happens "within the time requested for a Lock.tryLock(long, TimeUnit)", which is only true of the connection wait. Add a sentence that the effective ceiling is time plus Curator's connectionTimeoutMs and retry-policy budget, and that users needing a tight bound must configure CuratorFrameworkFactory.builder().connectionTimeoutMs(...) and their RetryPolicy below their lock timeout. This caveat belongs in the ZkLock.tryLock javadoc too, since it is a documented Lock contract deviation.
2. The connection check mixes two independent state machines; the pre-check is redundant. ZookeeperLockRegistry.java:311-313
client.getZookeeperClient().isConnected() reads curator-client's raw ConnectionState.isConnected (AtomicBoolean, driven by the ZK watcher). client.blockUntilConnected(...) reads a different source: ConnectionStateManager.currentConnectionState, the framework CONNECTED/SUSPENDED/LOST/READ_ONLY enum, updated by the framework's event processing. These can disagree in both directions.
Failure scenario: the raw client has already flipped to disconnected but the state manager has not yet processed the event, so currentConnectionState is still CONNECTED. blockUntilConnected then returns true immediately (ConnectionStateManager.java:207 — while (!isConnected()) is simply skipped), and we drop straight into the blocking acquire of finding 1 with a full waitTime budget that no longer bounds anything. The guard silently does nothing.
The pre-check also buys nothing: blockUntilConnected already returns immediately when the state manager says connected. Suggested fix — drop the pre-check and just call it:
if (!this.client.blockUntilConnected((int) Math.min(waitTime, Integer.MAX_VALUE), TimeUnit.MILLISECONDS)) {One behavioural caveat worth knowing before you do: ConnectionStateManager.currentConnectionState is null until the first state change is posted, so isConnected() is false there — the very first tryLock right after client.start() would wait for the state manager (milliseconds) instead of proceeding on the raw flag. That is arguably the more correct behaviour anyway.
3. The headline fix is untested. ZkLockRegistryTests.java
voidLockFailsWhenServerDown() does exercise the new blockUntilConnected path — that part is covered, credit where due. But the actual subject of commit c1c6fe753e — lockInterruptibly() no longer spinning forever and now honouring an interrupt while ZK is unreachable — has no test at all, and SI requires a test for a behaviour change. Nothing would catch a regression that reverts the Thread.interrupted() check.
Suggested: inside the isolated-server block, after server.stop(), submit registry.obtain("baz").lockInterruptibly() to an executor, thread.interrupt() it, and assert the future completes with an InterruptedException within a few seconds.
Second, cheaper suggestion that would have caught finding 1: wrap the existing lock2.tryLock(1, TimeUnit.SECONDS) at line 320 in an elapsed-time assertion (assertThat(elapsed).isLessThan(5000)). One line, and it pins the contract this PR is about.
4. lockInterruptibly() ignores the interrupt status on entry, and the loop has no backoff. ZookeeperLockRegistry.java:277-290
Lock.lockInterruptibly() specifies that InterruptedException is thrown if the thread "has its interrupted status set on entry to this method". Here the check is at the bottom of the loop, so if the lock happens to be free the first tryLock succeeds and the method returns normally with the interrupt still pending — the caller silently loses the interrupt until something else observes it. Fix: hoist a if (Thread.interrupted()) { throw new InterruptedException(...); } above the loop (or restructure as a do/while with the check first).
Separately: the loop still has no sleep between iterations. Today it happens to be paced because blockUntilConnected eats the full second when disconnected, but that is accidental — any path that makes tryLock return false promptly (e.g. the new KeeperException catch firing fast) re-creates the very hot spin this PR removes. A small sleep or backoff would make it correct by construction rather than by coincidence. Low severity, but cheap insurance given the PR title.
5. The exception filter — narrow in one direction, semantically lossy in the other. ZookeeperLockRegistry.java:324-333
The filter does not hide genuine errors: NoAuthException, NodeExistsException and a user-level NoNodeException still propagate as MessagingException. That part is right.
Two nuances. Narrow: LockInternals.attemptLock rethrows KeeperException.NoNodeException once the retry policy is exhausted, and Curator's own comment there reads "this can happen when the session expires, etc." — so a transient session-loss case can still surface as a thrown MessagingException rather than false. Pre-existing, not a regression, but it undercuts the stated goal of "connection trouble means false".
Lossy: swallowing ConnectionLossException into false covers the ambiguous case where the ephemeral lock node was created server-side before the connection dropped. Curator's withProtection() normally reconciles that on the next attempt, so this is acceptable — but it is a change from "throw and let the caller decide" to "silently false", and it deserves a sentence in zookeeper.adoc alongside the isConnected() note.
6. build.gradle:231-238 — the comment over-promises what the JUnit timeout gives you.
junit.jupiter.execution.timeout.thread.mode.default defaults to SAME_THREAD. TimeoutInvocation schedules an interrupt of the test thread and then still waits for the invocation to return. A test wedged in a non-interruptible block — a socket read inside InterProcessMutex.acquire(), i.e. exactly the hang this PR is about — is not aborted and the build stalls anyway. So "Fail a stuck test ... instead of stalling the whole build" only holds for interruptible hangs.
Fix: either add systemProperty 'junit.jupiter.execution.timeout.thread.mode.default', 'SEPARATE_THREAD' (caveat: this changes the executing thread for every test in the build, which can upset ThreadLocal- and Spring-TestContext-sensitive tests — worth a trial run before committing to it), or soften the comment to say the stuck thread is interrupted and its stack trace recorded when it is interruptible. Also: this is a global change to every module's test task riding along in a Zookeeper bugfix PR; for the 7.0.x/7.1.x backports I would split it into its own commit so it can be dropped independently if it turns slow-but-green CI tests red.
7. destroy() as a no-op — fine, no leak. ZookeeperLockRegistry.java:187-194
The only owned resource was the executor and it is gone; the CuratorFramework was always caller-owned. The empty body is correct and the javadoc explaining why is the right call. Two nits: the class now declares DisposableBean purely to expose a no-op (keep it for 7.0.x/7.1.x binary compatibility, but it and the setter are both 8.0 removal candidates — worth tracking together), and the javadoc could note that the locks cache is not cleared either, so expireUnusedOlderThan remains the only eviction hook.
8. Backport safety of the deprecation — this is the right call, no change needed.
Stating it plainly since it was asked. There is no XSD or namespace attribute for mutexTaskExecutor (I checked spring-integration-zookeeper/src/main/resources — the module has no XML config for the lock registry), so only programmatic callers are affected, and for them the method still compiles and still runs. The setter's only documented reason to exist was managed-thread environments (WorkManagerTaskExecutor); since the registry now creates no threads at all, those users are strictly better off. And removing an eagerly-initialised ThreadPoolTaskExecutor per registry instance is a resource reduction in a patch release, which is the safe direction. Retaining "some behaviour" would mean retaining the probe, which is the bug. Two small nits: the setter now silently accepts null where it previously did Assert.notNull (harmless, but it is a contract loosening); and forRemoval = true in a patch commits you to removal in 8.0 — presumably intentional, just confirming it is deliberate rather than reflexive.
9. Verified-correct items, so nobody re-litigates them in review.
- The
intnarrowing is safe.Math.min(waitTime, Integer.MAX_VALUE)is evaluated inlongand clamps at 2147483647 ms (~24.8 days);TimeUnit.toMillissaturates rather than wraps, so no overflow. Negative/zerotimeis also safe:ConnectionStateManager.blockUntilConnectedreturns immediately whenmaxWaitTime <= 0with a non-null unit (ConnectionStateManager.java:201-219, and theCuratorFrameworkjavadoc states it explicitly). - Interruptibility is preserved:
blockUntilConnectedwaits viaObject.waitand declaresInterruptedException,internalLockLooplikewise, andtryLockrethrows. - The
Math.max(0, ...)clamp is harmless but a no-op in practice —LockInternals.internalLockLooptreatsmillisToWait <= 0identically to a negative value (break→deleteOurPath→false). - Importing
org.apache.zookeeper.KeeperExceptioninto main sources is fine;ZookeeperMetadataStorealready does it, and zookeeper is anapi-scoped transitive ofcurator-recipes. - Checkstyle:
AtclauseOrderforMETHOD_DEFis@param, @return, @throws, @since, @deprecated, @see, so the setter javadoc order is compliant;EmptyBlockuses the default token set, which excludesMETHOD_DEF, so the emptydestroy()body will not trip it.
10. Nits.
Log/LogFactoryhas plenty of precedent, but newer SI code (DefaultLockRepository,PostgresChannelMessageTableSubscriber,LockRegistryLeaderInitiator) usesorg.springframework.core.log.LogAccessor, whosedebug(Supplier<String>)overloads would let you drop all threeisDebugEnabled()guards and the string concatenation.- Grammar: "involved into locking" → "involved in locking", in both
ZookeeperLockRegistry.java:125andzookeeper.adoc:88. ZkLockRegistryTests.java:332is 122 chars (SI style guide caps at 120; checkstyle has noLineLengthmodule, so purely cosmetic).ZookeeperTestSupport.createNewClient()already builds exactly this client with the sameBoundedExponentialBackoffRetry(100, 1000, 3); a connect-string overload there would avoid duplicating it in the test.@author/@since: both files already carry the author tag and there is no new public API, so nothing missing.- Branch hygiene:
770bced644is an empty commit and43d64a74cbpolishes its immediate predecessor — per CONTRIBUTING both should be squashed into the first commit before merge. Also, the commit trailer points at the PR itself; the guidelines ask for a GH issue reference (Fixes: gh-NNNN), and a real hang bug warrants one for the 7.0.6/7.1.x release notes.
* re-check `CuratorZookeeperClient.isConnected()` after a successful `blockUntilConnected()`: the latter is served from the `ConnectionStateManager` alone, so it may return `true` immediately, without consuming any of the requested time, while the socket is already gone. An acquisition against such a stale state blocks in the Curator retry loop far beyond the `tryLock()` time * check the interrupt status on entry into `lockInterruptibly()`: per the `Lock` contract an already interrupted thread must not attempt an acquisition at all; extract the check into a `checkInterruption()` * validate the argument in the deprecated `setMutexTaskExecutor()` instead of suppressing the unused warning * document in the JavaDocs and the reference manual that the `tryLock()` time is not a hard bound: the Curator retry loop waits for a connection on its own * test that `tryLock()` does not exceed the requested time by orders of magnitude, and that `lockInterruptibly()` reacts to an interrupt while the Zookeeper server is stopped
artembilan
left a comment
There was a problem hiding this comment.
Self-review of the branch (AI-assisted, verified against Curator 5.9.0 sources)
Claims below were traced to the actual Curator 5.9.0 (gradle/libs.versions.toml) and JUnit Jupiter 6.1.2 sources unpacked from the Gradle cache, not recalled.
Blocking issues
1. The new test does not exercise the fix it is named for.
ZkLockRegistryTests lines ~344-346:
lockThread.start();
lockThread.interrupt();There is no barrier, so the interrupt lands before or during the first tryLock. Three separate paths then produce InterruptedException, and only one of them is the new in-loop check:
- the entry guard
checkInterruption()inlockInterruptibly(); blockUntilConnected()->ConnectionStateManager.blockUntilConnected(verified:while (!isConnected()) { ... wait(waitTime); }, soObject.waitthrows immediately when the flag is already set) -> rethrown from theInterruptedExceptioncatch;- the in-loop
checkInterruption()- the actual subject of the PR.
Because the server is fully stopped, zookeeperClient.isConnected() is false on every iteration and control always reaches the interruptible wait(). Remove the in-loop check and this test still passes.
Fix: have the worker count down a latch after its first failed tryLock and interrupt only then; add a case driving the zero-blocking path of #2 below, the only place the in-loop check is load-bearing. Also lockThread is never joined: if the assertion fails, the thread keeps running through server.restart() and can acquire interruptible.
2. lockInterruptibly() still spins for a non-interrupted caller.
while (!tryLock(1, TimeUnit.SECONDS)) has no delay of its own, so it is throttled only by however long tryLock happens to block. This PR adds two paths where that is not one second:
Durable case - the new KeeperException catch. Both isConnected() checks pass, mutex.acquire throws ConnectionLossException, tryLock returns false. How long that took is entirely the user's RetryPolicy: with the test's BoundedExponentialBackoffRetry(100, 1000, 3) it is ~700 ms per turn (survivable); with a policy whose allowRetry declines immediately it is ~0 ms and the loop spins for the whole outage. Reachable for as long as the connection is down - not a race.
Transient case - the double isConnected() guard. zookeeperClient.isConnected() is false, blockUntilConnected returns true instantly out of ConnectionStateManager (verified: it never waits when its own state says connected), the second check is false, return false after zero blocking. That is exactly the state the new comment says the second check exists to catch - the guard for one symptom creates a zero-delay return the loop has no floor for.
Fix - give the loop a wall-clock slice, which covers both:
long deadline = System.currentTimeMillis() + 1000;
while (!tryLock(1, TimeUnit.SECONDS)) {
checkInterruption();
long remaining = deadline - System.currentTimeMillis();
if (remaining > 0) {
Thread.sleep(remaining);
}
deadline = System.currentTimeMillis() + 1000;
}Thread.sleep is itself interruptible, so the in-loop check keeps its meaning.
Non-blocking suggestions
-
Should the deprecation be backported? This PR pairs the busy-spin fix with a
forRemoval = truedeprecation and a behavior change (exception ->false), and it is marked for auto-cherry-pick to7.1.x&7.0.x. Every existing@Deprecated(since = ...)in the tree is a two-component minor (7.0x26,7.1x14) - no precedent for deprecating in a patch. Consider cherry-picking only the spin fix and leaving the deprecation onmain. (The7.0.6@sincelabels themselves are fine - three-component@sinceis standard here, 481 occurrences including@since 7.0.5.) -
SessionMovedExceptionleft out of the catch. Curator's retryable set isCONNECTIONLOSS | OPERATIONTIMEOUT | SESSIONMOVED | SESSIONEXPIRED(RetryPolicy), andRetryLoopImpl.takeExceptionrethrows the raw exception once retries are exhausted. Three of the four becomefalse;SessionMovedbecomes aMessagingException. Suggestcatch (KeeperException e)plusif (this.client.getZookeeperClient().getRetryPolicy().allowRetry(e)) { return false; }, or just add the fourth type. -
Reentrancy regresses while disconnected.
InterProcessMutex.internalLockreturnstruestraight from itsthreadDatamap, no ZK round trip, when the calling thread already owns the lock. The new guard runs first, so a reentranttryLockby the owner now returnsfalseduring an outage even though nothing needs the network. Cheap fix: an earlyif (this.mutex.isOwnedByCurrentThread())short-circuit before the guard. -
The timing assertion is both vacuous and flakeable.
< 10_000 msfor atryLock(1, SECONDS)is 10x slack - a 9-second block passes. AndCuratorFrameworkFactory.newClient(connectString, retryPolicy)takes the defaultssessionTimeoutMs = 60_000,connectionTimeoutMs = 15_000. If the client has not yet processed the disconnect,isConnected()is stilltrue, the guard is skipped, andRetryLoop.callWithRetry->internalBlockUntilConnectedOrTimedOut()can burn up to 15 s. Narrow window, but real. Fix: build viaCuratorFrameworkFactory.builder()with an explicit smallconnectionTimeoutMs/sessionTimeoutMs, then tighten the bound to ~3 s. -
The
build.gradlechange is scope creep, and half the comment is wrong as configured.SameThreadTimeoutInvocation.proceed()doesthread.interrupt()and then throws, so "interrupt a stuck test" is accurate. But "report a stack trace of its thread" is not what happens: the dump comes fromPreInterruptThreadDumpPrinter, registered only whenjunit.jupiter.execution.timeout.thread.dump.enabled=true, which defaults tofalse- and it prints toSystem.out, which the build suppresses below-i. Either add that system property or drop the clause; either way a global JUnit timeout for every module belongs in its own PR. -
Pre-existing copy-paste bug in the two-arg constructor:
Assert.notNull(client, "'keyToPath' cannot be null")should assertkeyToPath. One-line drive-by. -
The
InterruptedExceptioncatch intryLock(long, TimeUnit)doesThread.currentThread().interrupt(); throw e;, which contradicts theLock.tryLock(long, TimeUnit)javadoc ("the interrupted status is cleared") and is now inconsistent with the newcheckInterruption(). Pre-existing, but this PR is about interrupt semantics. -
The no-arg
tryLock()delegates totryLock(1, TimeUnit.SECONDS), so it can block ~1 s - the contract says acquire only if free at invocation time. Pre-existing;tryLock(0, TimeUnit.MILLISECONDS)would now behave correctly given the guard handles a zero budget. -
lock()wraps everything, includingInterruptedExceptionfrommutex.acquire(), intoIllegalStateExceptionwithout restoring the interrupt flag - and usesIllegalStateExceptionwheretryLock/unlockuseMessagingException. Pre-existing inconsistency. -
Wording. The javadoc uses
e.g.where the adoc equivalent correctly spells out "for example". The new comment says theCuratorZookeeperClient"reacts to a closed socket immediately, while itsConnectionStateManagermay still report a connection for a while" - "for a while" overstates it; the stronger and more accurate justification is thatCuratorZookeeperClient.isConnected()is the very flag the Curator retry loop consults. -
Commit hygiene. Squash before merge as planned:
770bcedis a body-only commit pointing at the PR,43d64a7/a627d72are "Polish" commits, andc1c6fe7's body describes behavior later reverted. If this lands onmainas a 7.2 item rather than a pure backport, it also wants awhats-new.adocentry.
Verified good
blockUntilConnecteddelegation.CuratorFrameworkImpldelegates solely toConnectionStateManager.blockUntilConnected, which issynchronizedandObject.wait()-based - genuinely interruptible, and withunits != nullplus a non-positivemaxWaitTimeit returnsisConnected()immediately rather than blocking. TheTimeUnit.MILLISECONDSargument is therefore safe; the no-argblockUntilConnected()(unitsnull) would have blocked indefinitely.- The two state machines really are independent.
CuratorZookeeperClient.isConnected()reads anAtomicBooleaninConnectionState;ConnectionStateManager.isConnected()reads its owncurrentConnectionState. InConnectionState.processthe atomic is set before parent watchers fire, andConnectionState.reset()clears it with noWatchedEventat all. SoCZK == false && CSM == trueis genuinely reachable, and the guard's logic holds. - The guard is well targeted.
RetryLoop.callWithRetrycallsinternalBlockUntilConnectedOrTimedOut(), whose loop condition isstate.isConnected()- the same flag the new code checks.isConnected() == falseis therefore a precise predictor thatmutex.acquirewould burnconnectionTimeoutMs. Worth saying in the comment. - The
KeeperExceptioncatch is reachable, not dead code.RetryLoopImpl.takeExceptionrethrows the original exception unwrapped, andLockInternals.attemptLockre-wraps onlyNoNodeException. - Returning
falsedoes not leak a lock znode.LockInternals.internalLockLoopdeletes the candidate node viadeleteOurPathQuietly/guaranteed()on both the exception and not-acquired paths. - The class javadoc caveat is accurate.
connectionTimeoutMs(default 15 s) +RetryPolicybudget is exactly the unbounded portion, and "two-thirds of the session timeout" is the right ZK client read timeout. - Arithmetic.
(int) Math.min(waitTime, Integer.MAX_VALUE)is overflow-safe;Math.max(0, ...)correctly stops a negative deadline reachingacquire- a real improvement over the old code; negative/zerotimedegrades to a non-blocking attempt. checkInterruption()usesThread.interrupted()(clears) rather thanisInterrupted()- correct.destroy()as a no-op is safe. No internal executor remains, so nothing leaks; the originalWorkManagerTaskExecutormotivation is satisfied by spawning no threads at all.- Giving the test its own
TestingServeris a real improvement - the shared stop/restart genuinely could wedge the tests declared after it.
Could not verify
- Whether
KeeperException.OperationTimeoutExceptionis actually thrown on this path. Curator listsOPERATIONTIMEOUTas retryable but there is no site in curator-client/curator-framework 5.9.0 that raises it - it would have to come from the ZooKeeper client itself. Catching it is harmless, but the adoc claim may be unfalsifiable in practice. - The real-world width of the
CZK == false && CSM == truewindow. Structurally confirmed, but in the ordinary disconnect both flags are driven by the sameWatchedEventon the same ZK event thread in a fixed order, so the typical window is microseconds. This does not weaken finding #2, whose durable case is theKeeperExceptionpath. - Runtime behavior -
:spring-integration-zookeeper:testwas not executed, so findings #1 and #6 are from code reading only.
The `setMutexTaskExecutor()` deprecation, the removal of the executor and the `blockUntilConnected()` rewrite of the connection check are an API and behavior change, not a bug fix, and this branch is meant to be cherry-picked to `7.1.x` and `7.0.x`. Every existing `@Deprecated(since = ...)` in the tree is a minor version; there is no precedent for deprecating in a patch. * restore the `mutexTaskExecutor`, its setter, `destroy()` and the asynchronous `checkExists()` connection probe exactly as they are on `main`, including the default executor configuration - the `queueCapacity = 0` from the first commit is dropped as well, so the probe serialization stays pre-existing and untouched * drop the `KeeperException` to `false` mapping in `tryLock()`: also a behavior change rather than a fix * drop the `blockUntilConnected()` state check, the class JavaDoc caveat and the `7.0.6` paragraph in the reference manual along with it What remains is the fix itself: the interrupt handling in `lockInterruptibly()`, the `Math.max(0, ...)` clamp of the recomputed `waitTime`, the `DEBUG` logging, the test isolation and the build test timeout. Note that the interrupt checks are a guard, not the mechanism: the restored `tryLock()` spends its wait inside `Future.get()`, which throws `InterruptedException` on its own, so `lockInterruptibly()` already terminated on an interrupt. Verified by removing the in-loop check - the test still passes. The "busy-spin hang" of the first commit was therefore the test isolation issue alone, and the branch title overclaims.
An extracted `curator-recipes` source file landed in the repository root while verifying the `InterProcessMutex` timeout behavior.
|
Following up on the review above: the That resolves the findings that described code which no longer exists: #2 (the delay floor is restored - #1 stands, and is worse than reported. The interrupt checks are a guard, not the mechanism: the restored Also done: #6 - the timing bound is tightened from 10 s to 3 s and the interrupt thread is now joined. Still open: #8 (the |
The assertion claimed the `lockInterruptibly()` would otherwise "spin forever", but the loop was already interruptible: its `tryLock()` waits inside a `Future.get()`, which raises the `InterruptedException` on its own. Removing the in-loop `checkInterruption()` leaves this test green. * assert the contract the test does cover: the `lockInterruptibly()` returns on an interrupt and propagates it instead of swallowing it in the retry loop * drop the started latch: it is counted down inside the thread just before the `lockInterruptibly()` call, so it is not a barrier and buys nothing over interrupting right after the `start()`
The `ZookeeperLockRegistry(CuratorFramework, KeyToPathStrategy)` constructor asserted `client` in both checks, so a `null` `KeyToPathStrategy` passed the validation and failed later with a plain `NullPointerException` from the `keyToPath.bounded()` call below.
* replace the `LogFactory.getLog()` with a `LogAccessor` * drop the `isDebugEnabled()` guards: the `Supplier` overloads defer the message concatenation on their own
artembilan
left a comment
There was a problem hiding this comment.
Second review, post-rollback (AI-assisted, fresh reviewer)
Reviewed the net git diff main...HEAD rather than the commit list, since the rollback made those diverge. Curator claims verified against curator-recipes-5.9.0-sources.jar from the Gradle cache. Build, checkstyle and voidLockFailsWhenServerDown (2.18 s) all green locally.
No blocking issues in the code. The rollback is clean - grepping the touched files for blockUntilConnected, 7.0.6 and Deprecated returns nothing, zookeeper.adoc is byte-identical to main, and the executor, its setter, destroy() and the checkExists() probe match main exactly.
Worth fixing before merge
1. The squashed commit message needs to be hand-written. c1c6fe753e is still the first commit, and on a GitHub squash its body becomes the default message that ships to main, 7.1.x and 7.0.x. It documents three things no longer in the net diff: a WARN in tryLock() (the diff logs DEBUG, and against main this is net-new logging, not a demotion - main has no logger in this class at all), queueCapacity = 0 on the default executor (reverted, the executor is byte-identical to main), and the "busy-spin hang" framing that 9f85bc4409 explicitly concedes was really test isolation. The accurate summary already exists in 9f85bc4409 ("What remains is the fix itself: ..."); use that plus the auto-cherry-pick trailer.
2. The InterruptedException path orphans the probe task. ZookeeperLockRegistry.java:356-358 rethrows without future.cancel(true), while both the !connected path (:341) and the TimeoutException path (:352) cancel. The default ThreadPoolTaskExecutor is corePoolSize = 1 with an unbounded queue, so: server down, thread A interrupted inside future.get(1000), its checkExists() probe stays runnable in Curator's retry loop for up to connectionTimeoutMs (15 s default) holding the single worker - every subsequent tryLock() on the registry then queues behind it and returns false regardless of the real connection state. Pre-existing on main, distinct from the deferred queue-capacity item (that one is capacity, this is a leaked task), one line, backport-safe:
catch (InterruptedException e) {
future.cancel(true);
Thread.currentThread().interrupt();
throw e;
}3. Two lines exceed 120 columns and CI will not catch them - src/checkstyle/checkstyle.xml has no LineLength module. ZookeeperLockRegistry.java:353 is 121 (the LOGGER.debug(() -> "Timed out ...") line) and ZkLockRegistryTests.java:370 is 122, the latter purely from the three extra tabs of nesting. For the test, collapsing the two try-with-resources into one drops every body line by a tab and fixes it without rewrapping (resources initialise left to right, so client may reference server):
try (var server = new TestingServer();
var client = CuratorFrameworkFactory.newClient(server.getConnectString(),
new BoundedExponentialBackoffRetry(100, 1000, 3))) {4. No test for the Assert.notNull(keyToPath, ...) fix - the branch's only genuine behavior fix, and CONTRIBUTING asks for one. Watch the overload trap: new ZookeeperLockRegistry(this.client, null) is ambiguous with the String root overload and will not compile, so the cast is required:
assertThatIllegalArgumentException()
.isThrownBy(() -> new ZookeeperLockRegistry(this.client, (KeyToPathStrategy) null))
.withMessage("'keyToPath' cannot be null");Wording
5. The Math.max(0, ...) clamp is a behavioral no-op and the comment implies otherwise. Verified in 5.9.0: LockInternals.internalLockLoop does if (millisToWait <= 0) { break; }, so 0 and any negative take the identical branch, and allowRetry uses wall-clock elapsed rather than millisToWait. Nothing else reads it. Keep the clamp, but :346 currently reads "A slow connection check must not pass a negative deadline to the mutex", which reads as a defect being fixed. Suggest "Defensive: never hand the mutex a negative deadline."
6. The test assertion still oversells its coverage. ZkLockRegistryTests.java:346-348 does start() then interrupt() with no barrier, and Thread.start() establishes happens-before, so the flag is set before lockInterruptibly() runs and the entry guard wins essentially every time - the retry loop is never entered. The message "must propagate the interrupt, not swallow it in its loop" describes a path the test does not reach. Suggest "...must propagate the interrupt rather than attempt an acquisition." The main-code comment is already honest about the in-loop check being a guard; the test should match.
7. lockInterruptibly() now leaves the interrupt flag in two different states depending on where the interrupt landed: checkInterruption() uses Thread.interrupted() and clears it (correct per the Lock javadoc), while an InterruptedException bubbling up from tryLock re-sets it at :357. The tryLock re-set is a known deferred item, but the inconsistency inside lockInterruptibly() is new with this diff. No test covers flag state either way. Fine to leave, but the deferred item should not be closed without revisiting this.
8. lock4 at ZkLockRegistryTests.java:331 is declared before lock3 at :368 and is not part of the numbered sequence - interruptibleLock would read better.
Verified good
- The entry
checkInterruption()is a real contract improvement, not cosmetics. Onmain, if the flag was already set and the probe future happened to be complete,FutureTask.get(timeout)returns without an interrupt check, solockInterruptibly()could acquire the lock on an interrupted thread. The guard closes that. - The interrupt test is sound and non-flaky. Every loop exit lands on an interruptible point:
FutureTask.awaitDonechecksThread.interrupted()first, and theTimeoutExceptionpath falls through tocheckInterruption(). There is no window where the loop spins with the flag set. Thejoin(10_000)+isAlive()guard correctly stops a failed assertion leaving a thread racing forlock2. - Test isolation is genuine.
ZookeeperTestSupport.testingServeris a static per-class field; the new localTestingServertakes a random port and its own temp dir with no JVM-global state shared. The stated hazard is real - the six tests declared aftervoidLockFailsWhenServerDownall usethis.clientagainst the shared server. - The 3 s bound is safe. Expected cost ~1 s (
future.get(1000)plus one thread spawn on an empty queue); measured 2.18 s for the whole test. 5 s would cost nothing if you want more CI headroom. LogAccessorconversion is correct.LogAccessor(Class<?>)anddebug(Supplier<? extends CharSequence>)both exist; dropping the guards is right for theSupplieroverloads.LOGGERis the dominant in-tree name (18 of 20). All three sites areDEBUG, so net-new logging at the quietest level.build.gradlecomment is now accurate -SAME_THREADdoes schedule an interrupt and then wait for the invocation to return.- PR title matches the diff,
@author/@sinceneed nothing (no new public API), andwhats-new.adocis not needed for a patch-targeted bug fix.
Could not verify
- Whether the 10-minute default timeout is safe across all 40+ modules; only
spring-integration-zookeeperwas run. Risk direction is a hard failure rather than a silent hang, so it is self-reporting. - CI-machine timing for the 3 s bound. Local evidence only.
The `!connected` and the `TimeoutException` paths both cancel the probe future, but the `InterruptedException` path did not. The default `mutexTaskExecutor` is a `ThreadPoolTaskExecutor` with `corePoolSize` of `1`, so an abandoned `checkExists()` stays runnable in the Curator retry loop for the whole `connectionTimeoutMs` (15 s by default) and holds the only worker. Every subsequent `tryLock()` on that registry then queues behind it and returns `false` on its own timeout, whatever the real connection state is.
* keep two lines within 120 columns - there is no `LineLength` module in the Checkstyle configuration, so neither is caught by the build * collapse the nested try-with-resources in `voidLockFailsWhenServerDown()` into one: the resources are still closed in reverse order, and the whole method loses an indentation level * say that the `Math.max(0, ...)` is defensive: Curator breaks its lock loop on `millisToWait <= 0`, so a negative deadline behaves exactly like a zero one and the previous comment implied a defect which is not there * assert what the interrupt test reaches: `Thread.start()` establishes happens-before, so the guard on the `lockInterruptibly()` entry wins the race and the retry loop is not entered * rename `lock4` to `interruptibleLock`: it is not a part of the numbered sequence
* use `orders`, `invoices` and `shipments` for the lock keys, keeping every same-key and different-key relation the assertions rely on: the reentrancy tests still obtain one key twice, `testTwoLocks()` still obtains two distinct ones, and the capacity tests keep their `orders:<n>` ordering * name the locks in `voidLockFailsWhenServerDown()` after their keys: the `lock1`, `lock2`, `lock3` sequence stopped matching anything once `lock4` became `interruptibleLock` * fix the two assertion descriptions which named those variables or read ungrammatically
…ling
`ZkLockRegistryTests.voidLockFailsWhenServerDown()` stopped and restarted the
`TestingServer` shared by the whole class. A failed restart left the six tests
declared after it blocked forever in `InterProcessMutex.acquire()`, and with no
JUnit timeout configured at all, the whole Gradle build stalled silently until
the CI job limit.
* give that test its own `TestingServer` and `CuratorFramework`, and assert that
a `tryLock()` against a stopped server returns within the time requested
* bound the `maincountDownLatch.await()` in `concurrentObtainCapacityTest()`
* configure a `junit.jupiter.execution.timeout.default` for all `Test` tasks
(`10 m`, and `30 m` for `testAll`), so a stuck test is reported instead of
stalling the build with no output
The `ZkLock` interrupt and timeout handling around it:
* check the interrupt status on entry into `lockInterruptibly()`: per the `Lock`
contract an already interrupted thread must not attempt an acquisition at all.
The retry loop keeps a guard as well, for the paths where a `tryLock()` returns
`false` instead of raising the `InterruptedException` from its connection check
* cancel the connection check when a `tryLock()` is interrupted, as its other two
exit paths already did. The default `mutexTaskExecutor` has a `corePoolSize` of
`1`, so an abandoned `checkExists()` holds the only worker for the whole
`connectionTimeoutMs` and every subsequent `tryLock()` queues behind it
* clamp the recomputed `waitTime` with a `Math.max(0, ...)`
* assert the `keyToPath` argument, not the `client` twice, in the two-argument
constructor
* add `DEBUG` logging over a `LogAccessor` for the retry and for both
no-connection paths, so an outage is diagnosable at all
No API or behavior change: the `mutexTaskExecutor`, its setter and the
asynchronous `checkExists()` connection probe are all as they were.
Fixes: #11244
(cherry picked from commit 2f81db9)
…ling
`ZkLockRegistryTests.voidLockFailsWhenServerDown()` stopped and restarted the
`TestingServer` shared by the whole class. A failed restart left the six tests
declared after it blocked forever in `InterProcessMutex.acquire()`, and with no
JUnit timeout configured at all, the whole Gradle build stalled silently until
the CI job limit.
* give that test its own `TestingServer` and `CuratorFramework`, and assert that
a `tryLock()` against a stopped server returns within the time requested
* bound the `maincountDownLatch.await()` in `concurrentObtainCapacityTest()`
* configure a `junit.jupiter.execution.timeout.default` for all `Test` tasks
(`10 m`, and `30 m` for `testAll`), so a stuck test is reported instead of
stalling the build with no output
The `ZkLock` interrupt and timeout handling around it:
* check the interrupt status on entry into `lockInterruptibly()`: per the `Lock`
contract an already interrupted thread must not attempt an acquisition at all.
The retry loop keeps a guard as well, for the paths where a `tryLock()` returns
`false` instead of raising the `InterruptedException` from its connection check
* cancel the connection check when a `tryLock()` is interrupted, as its other two
exit paths already did. The default `mutexTaskExecutor` has a `corePoolSize` of
`1`, so an abandoned `checkExists()` holds the only worker for the whole
`connectionTimeoutMs` and every subsequent `tryLock()` queues behind it
* clamp the recomputed `waitTime` with a `Math.max(0, ...)`
* assert the `keyToPath` argument, not the `client` twice, in the two-argument
constructor
* add `DEBUG` logging over a `LogAccessor` for the retry and for both
no-connection paths, so an outage is diagnosable at all
No API or behavior change: the `mutexTaskExecutor`, its setter and the
asynchronous `checkExists()` connection probe are all as they were.
Fixes: #11244
(cherry picked from commit 2f81db9)
ZkLock.lockInterruptibly()looped ontryLock(1, SECONDS)until it succeeded.Since
tryLock()returnsfalse(rather than throwing) when the Zookeeper connection probe times out, an unreachable server turned that loop into an endless spin with no deadline, no interrupt check and no logging.The build had no test timeout configured at all, so such a hang stalled Gradle silently until the CI job limit.
Thread.interrupted()per iteration inlockInterruptibly()and throwInterruptedException; log aDEBUGmessage per retryWARNintryLock()when the connection probe times out or reports no connection, instead of silently returningfalsewaitTimewithMath.max(0, ...)so a slow probe cannot pass a negative deadline toInterProcessMutex.acquire()queueCapacityto0on the defaultmutexTaskExecutor: with Spring's defaults (core1, unbounded queue) the pool never grew past a single thread, so concurrenttryLock()calls serialized behind one connection probeZkLockRegistryTests.voidLockFailsWhenServerDown()its ownTestingServer; it used to stop and restart the one shared by the class, and a failed restart left the six tests declared after it blocked forever inInterProcessMutex.acquire()maincountDownLatch.await()inconcurrentObtainCapacityTest()junit.jupiter.execution.timeout.defaultfor allTesttasks (10 m,30 mfortestAll) so a stuck test fails with a stack trace of its threadAuto-cherry-pick to
7.1.x&7.0.x