ManagedSelector can livelock under high load

[sbordet]

There is evidence that under high load a thread may livelock running selector actions.

[sbordet]

@gregw I have reviewed and updated your code, see 333c22d for a description of the changes.

[gregw]

@sbordet your changes LGTM!

[lgangloff]

Hello,
We're facing the same issue under high load test with the lastest snapshot 9.4.8-20171107.091315.
We're using jetty-runner with this xml config (maxThreads & requestHeaderSize were uppered)

<Get name="ThreadPool">
      <Set name="minThreads" type="int"><SystemProperty name="threads.min" default="50"/></Set>
      <Set name="maxThreads" type="int"><SystemProperty name="threads.max" default="350"/></Set>
      <Set name="idleTimeout" type="int"><SystemProperty name="threads.timeout" default="60000"/></Set>
      <Set name="detailedDump">false</Set>
    </Get>

	<New id="httpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
		<Set name="secureScheme">https</Set>
		<Set name="outputBufferSize">32768</Set>
		<Set name="requestHeaderSize">16384</Set>
		<Set name="responseHeaderSize">8192</Set>

		<Call name="addCustomizer">
			<Arg>
				<New class="org.eclipse.jetty.server.ForwardedRequestCustomizer" />
			</Arg>
		</Call>
	</New>

  <Call name="addConnector">
    <Arg>
      <New id="httpConnector" class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server"><Ref refid="Server" /></Arg>
        <Arg name="acceptors" type="int"><Property name="jetty.http.acceptors" default="-1"/></Arg>
        <Arg name="selectors" type="int"><Property name="jetty.http.selectors" default="-1"/></Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.HttpConnectionFactory">
                <Arg name="config"><Ref refid="httpConfig" /></Arg>
              </New>
            </Item>
          </Array>
        </Arg>
        <Set name="host"><Property name="jetty.http.host" /></Set>
        <Set name="port"><SystemProperty name="jetty.http.port" default="8080" /></Set>
        <Set name="idleTimeout"><Property name="jetty.http.idleTimeout" default="30000"/></Set>
        <Set name="soLingerTime"><Property name="jetty.http.soLingerTime" default="-1"/></Set>
        <Set name="acceptorPriorityDelta"><Property name="jetty.http.acceptorPriorityDelta" default="0"/></Set>
        <Set name="acceptQueueSize"><Property name="jetty.http.acceptQueueSize" default="0"/></Set>
      </New>
    </Arg>
  </Call>

We have 2 virtual server each hosting 3 Jetty instance (on port 8080/8081/8082) having 1 webapp per instance.
Requests are load balanced with haproxy and the sticky session strategy. We store the session data under PostgreSQL with a JDBCSessionDataStore.

Here are the thread dump that shows:

• 1 deadlock
• 2 threads locking on monitors

Our high load test reaches the maxThreads and Jetty doesn't respond anymore.
Note that if we disable the session persistence, everything works fine.

Thanks for your reply.

thread-dump-out.zip

[sbordet]

@lgangloff your issue is different from what described in this issue, but it is indeed a bug.
Can you please open a new issue ? Thanks !

[gregw]

@sbordet is it a bug? I did an analysis and no deadlock detected. I do see many threads waiting on the lock for the same session, so there is some contention on session.... anyway if it is a bug it is definitely a different issue

[lgangloff]

Sorry, the fact that the threads are locking monitor at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:362) made me feel that was the same issue. Anyway, see #1947
Thanks !

[sbordet]

@gregw yes it is a bug as we call user code while holding locks.
Not sure about the tool you used, but the thread dump from @lgangloff shows clearly a deadlock in the session code. I commented on #1947.

[gregw]

There is some evidence that we are favouring selection too much and that we should always consume the action list as it exists initially before selecting.