Gerrit Trigger SSH often gets stuck in CLOSED WAITING

[thadguidry]

We have seen this issue occur multiple times (once a day) on Ubuntu 14 and 16 and multiple servers.
Gerrit Trigger threads set to 3 on Jenkins. Using latest Gerrit Trigger 2.24.0

We see the SSH connections in CLOSED_WAITING we decided to take a heap dump to assist the team in helping us debug the issue. Can open and browse in Java VisualVM
https://drive.google.com/open?id=0B533WzlrxWraVHZEU01MSnpXcW8

Related to #66 #68 #69 ??

[rinrinne]

What is "CLOSED_WAITING"? I don't know such status name. Googling but not found.

[rinrinne]

Maybe not CLOSED_WAITING but TIMED_WAITING, it is one of thread status that is defined by Thread.State.

I checked heapdump with jvisualvm (used below "contains" filter in classes tab)

gerrit.trigger
gerritevents
com.jcraft.jsh

• Observed a lot of RefReplicated event objects.
• ReplicationCache has a lot of RefReplicatedId instance.
• Cannot get actual thread status from heapdump because it is managed by class field in sum.misc.VM.

Maybe your system has gerrit slave then do replication. I guess your issue is caused by replication status.

If you are facing issue related to #66, #68, #69, event object is no longer generated. To identify the root cause, you need to get more heapdump.

[thadguidry]

@rinrinne Will try to get more heap dumps. The WAITING was looking at the actual SSH status with netstat. I'll provide the exact commands my team used for that tomorrow as an update on this comment.

UPDATE:

Commands we use

netstat -an
GERRIT_SSH_PORT=29418 

netstat -an | grep ${GERRIT_SSH_PORT} 

netstat -an | grep ${GERRIT_SSH_PORT} | grep ESTABLISHED

[thadguidry]

We are no longer seeing this specific issue of the port not being ESTABLISHED, but in a CLOSED_blah state.