Jicofo deadlock detected

[damencho]

We detected deadlock on beta. We were unable to create conferences till jicofo was restarted.

Java stack information for the threads listed above:

"pool-4-thread-17":
at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:236)
- waiting to lock <0x0000000775bc70d0> (a org.jitsi.jicofo.FocusManager)
at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:400)
at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:246)
at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-4":
at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1222)
- waiting to lock <0x0000000775fae078> (a org.jitsi.jicofo.JitsiMeetConference)
at org.jitsi.jicofo.FocusManager.createConference(FocusManager.java:302)
at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:244)
- locked <0x0000000775bc70d0> (a org.jitsi.jicofo.FocusManager)
at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:400)
at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:246)
at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Smack Listener Processor (0)":
at org.jitsi.jicofo.FocusManager.conferenceEnded(FocusManager.java:333)
- waiting to lock <0x0000000775bc70d0> (a org.jitsi.jicofo.FocusManager)
at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1242)
- locked <0x0000000775fae078> (a org.jitsi.jicofo.JitsiMeetConference)
at org.jitsi.jicofo.ChatRoomRoleAndPresence.verifyFocusRole(ChatRoomRoleAndPresence.java:296)
at org.jitsi.jicofo.ChatRoomRoleAndPresence.localUserRoleChanged(ChatRoomRoleAndPresence.java:313)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.fireLocalUserRoleEvent(ChatRoomImpl.java:401)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.setLocalUserRole(ChatRoomImpl.java:412)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processOwnPresence(ChatRoomImpl.java:1159)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processPacket(ChatRoomImpl.java:1116)
at org.jivesoftware.smack.Connection$ListenerWrapper.notifyListener(Connection.java:819)
at org.jivesoftware.smack.PacketReader$ListenerNotification.run(PacketReader.java:457)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

[damencho]

Ping me for full thread dump and heap dump if needed.

[paweldomas]

Reopening with new stack:

Java stack information for the threads listed above:

"pool-4-thread-17":
at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:231)
- waiting to lock <0x0000000775bca230> (a org.jitsi.jicofo.FocusManager)
at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:396)
at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:242)
at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-12":
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.leave(ChatRoomImpl.java:274)
- waiting to lock <0x000000077634a790> (a java.util.HashMap)
at org.jitsi.jicofo.JitsiMeetConference.leaveTheRoom(JitsiMeetConference.java:422)
at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:310)
- locked <0x000000077634a810> (a org.jitsi.jicofo.JitsiMeetConference)
at org.jitsi.jicofo.JitsiMeetConference.start(JitsiMeetConference.java:289)
- locked <0x000000077634a810> (a org.jitsi.jicofo.JitsiMeetConference)
at org.jitsi.jicofo.FocusManager.createConference(FocusManager.java:291)
at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:239)
- locked <0x0000000775bca230> (a org.jitsi.jicofo.FocusManager)
at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:396)
at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:242)
at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Smack Listener Processor (0)":
at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:301)
- waiting to lock <0x000000077634a810> (a org.jitsi.jicofo.JitsiMeetConference)
at org.jitsi.jicofo.ChatRoomRoleAndPresence.verifyFocusRole(ChatRoomRoleAndPresence.java:296)
at org.jitsi.jicofo.ChatRoomRoleAndPresence.electNewOwner(ChatRoomRoleAndPresence.java:224)
at org.jitsi.jicofo.ChatRoomRoleAndPresence.memberPresenceChanged(ChatRoomRoleAndPresence.java:167)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.notifyParticipantJoined(ChatRoomImpl.java:756)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.access$500(ChatRoomImpl.java:41)
at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$MemberListener.joined(ChatRoomImpl.java:893)
- locked <0x000000077634a790> (a java.util.HashMap)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jivesoftware.smackx.muc.MultiUserChat.fireParticipantStatusListeners(MultiUserChat.java:2104)
at org.jivesoftware.smackx.muc.MultiUserChat.access$900(MultiUserChat.java:79)
at org.jivesoftware.smackx.muc.MultiUserChat$5.processPacket(MultiUserChat.java:2179)
at org.jivesoftware.smackx.muc.PacketMultiplexListener.processPacket(PacketMultiplexListener.java:82)
at org.jivesoftware.smackx.muc.RoomListenerMultiplexor$RoomMultiplexListener.processPacket(RoomListenerMultiplexor.java:223)
at org.jivesoftware.smack.Connection$ListenerWrapper.notifyListener(Connection.java:819)
at org.jivesoftware.smack.PacketReader$ListenerNotification.run(PacketReader.java:457)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

[paweldomas]

Fixed in d2e0bf5