Fix deadlock when afterDisconnect is called during launch() #304
Conversation
Deadlock happens between: (a) https://github.com/jenkinsci/ssh-slaves-plugin/blob/main/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L953 (b) https://github.com/jenkinsci/ssh-slaves-plugin/blob/main/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L413 Launch() is stuck here: https://github.com/jenkinsci/ssh-slaves-plugin/blob/main/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L486 , and `callable` is stuck waiting here: https://github.com/jenkinsci/trilead-ssh2/blob/8e7118e74deacd7ef044df9076fcb9b95280c163/src/com/trilead/ssh2/channel/FifoBuffer.java#L212 After some time `afterDisconnect` is called which initiates tear down process. Teardown process calls launcherExecutorService.shutdown() (here: https://github.com/jenkinsci/ssh-slaves-plugin/blob/main/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L941) and continues to (a). But because there is no writer writing to FifoBuffer, launch() never finishes and is stuck in (b), and because `shutdown()` only prevents new tasks being submitted (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ExecutorService.html) there is no one to interrupt lock.wait() in FifoBuffer, thus deadlock. The solutions is to use recommended way of shutting down ExecutorService from docs.
|
Also, the mentioned deadlock happened here too: https://github.com/jenkinsci/ssh-slaves-plugin/pull/283/checks?check_run_id=5979901734 (see "Standard error" logs there) |
In this PR the Docker agent is not provisioned in time, the issue is in the Docker agent not in the plugin, and the test failed by timeout. For some reason, the Docker provision is flaky sometimes in jenkins.io |
|
I just see the last comment an overnight test |
Deadlock happens between:
(a) synchronized void tearDownConnection
(b) synchronized (this) {
Launch() is stuck here: results = srv.invokeAll(callables); ,
and
callableis stuck waiting here: lock.wait(); // wait until the writer gives us somethingAfter some time
afterDisconnectis called which initiates tear down process. Teardown process calls launcherExecutorService.shutdown() and continues to (a).But because there is no writer writing to FifoBuffer, launch() never finishes and is stuck in (b), and because
shutdown()only prevents new tasks being submitted (see docs) there is no one to interrupt lock.wait() in FifoBuffer, thus deadlock. The solutions is to use recommended way of shutting down ExecutorService from docs.See JENKINS-68656.
Submitter checklist