Broker can lock up due to defunct http connections

[gianm]

HTTP connections from the broker to realtime or historical nodes can stop working but not get reset (perhaps they will be reset once something in the network stack times them out, but this could take hours). Any broker jetty threads that happen to be trying to talk to a defunct server will become stuck until the connections reset. It's apparently easy for this to happen to all jetty threads when there are a lot of concurrent queries, which locks up the broker.

We could set a default timeout on broker -> compute requests. Or for slick extra credit, abort outgoing connections when the broker's ServerView realizes that a compute has gone missing.

[fjy]

We've gone with setting a default timeout of 5 mins on broker --> compute requests for the time being. We'll keep this issue open and eventually push better cleanup of outgoing connections.

[sixtus]

I ran into this immediately after deploying 0.6.42 in production :/

[sixtus]

This is getting really really bad now, it's like happening within minutes event though I got 4 brokers (instead of two when using druid-0.4)

[gianm]

This issue only refers to behavior when there is an overabundance of broken connections, like connections to compute nodes that simply don't work or tcp connections stuck half-open due to compute crashes. I don't think any of that should be really common (hopefully not common enough to be "really bad" unless servers are constantly crashing) and am wondering if you are really seeing this issue or if you are seeing something similar but different.

What sort of symptoms are you seeing?

[sixtus]

I see brokers simply not answering anymore.

My current working theory is the relationship of druid.broker.http.numConnections and druid.server.http.numThreads. Might that be it?

[fjy]

This is also a day 1 druid issue. The default timeout that was introduced
can be turned off by setting druid.broker.http.readTimeout to null or a
large value.
On Jan 14, 2014 7:49 AM, "Gian Merlino" notifications@github.com wrote:

This issue only refers to behavior when there is an overabundance of
broken connections, like connections to compute nodes that simply don't
work or tcp connections stuck half-open due to compute crashes. I don't
think any of that should be really common (hopefully not common enough to
be "really bad" unless servers are constantly crashing) and am wondering if
you are really seeing this issue or if you are seeing something similar but
different.

What sort of symptoms are you seeing?

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/325#issuecomment-32276060
.

[gianm]

The query path goes like broker http thread -> broker compute connection pool -> compute http threads -> compute computation pools. Can you tell which resource is the one blocking things? Thread dumps and netstats during an outage are good ways of figuring that out.

If one of the broker resources is blocking things, then those properties you mentioned are a good place to start looking for a solution. If the compute resources look more problematic, there are also properties for number of compute http threads (druid.server.http.numThreads) and the size of the computation pool (druid.processing.numThreads).

[sixtus]

There is an undocumented config requirement:

[num of brokers] * broker:druid.processing.numThreads > historical:druid.server.http.numThreads

Otherwise you get unresponsive brokers very quickly under query load.

If you step back a sec, that seems very obvious. It's just not documented explicitly and I ran into it.

[xvrl]

Do you mean greater than the total number of http threads across all
historical nodes, or greater than then number of http threads on each
historical node?

This requirement may also be very specific to your work-load. I believe it
would greatly depend on your cache hit-rate and how many queries actually
get sent down to historical nodes vs. how many results get pulled from
cache.

On Mon, Jan 20, 2014 at 1:22 AM, Hagen Rother notifications@github.comwrote:

There is an undocumented config requirement:

[num of brokers] * broker:druid.processing.numThreads >
historical:druid.server.http.numThreads

Otherwise you get unresponsive brokers very quickly under query load.

If you step back a sec, that seems very obvious. It's just not documented
explicitly and I ran into it.

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/325#issuecomment-32744317
.

[fjy]

Hey guys, is there still a problem or can I resolve it?

[sixtus]

It seems it can be avoided by configuration. sum of brokers' http connections vs. historical's threads. broker locks up if larger.

[drcrallen]

There was a problem at one time where requests were processing in the http thread pool instead of the processing thread pool. I thought there was a PR that fixed it but can't find it at this time. Is this still an issue?