Backpressure for broker-to-historical communication #4933
Labels
Comments
1 task
|
Also related: #6014, an attempt to solve this. |
|
#6014 was closed but still has relevant discussion: #6014 (comment) |
This was referenced Aug 1, 2018
gianm
added a commit
to gianm/druid
that referenced
this issue
Sep 7, 2018
Adds a new property "druid.broker.http.maxQueuedBytes" and a new context parameter "maxQueuedBytes". Both represent a maximum number of bytes queued per query before exerting backpressure on the channel to the data server. Fixes apache#4933.
Merged
gianm
added a commit
that referenced
this issue
Sep 10, 2018
* Broker backpressure. Adds a new property "druid.broker.http.maxQueuedBytes" and a new context parameter "maxQueuedBytes". Both represent a maximum number of bytes queued per query before exerting backpressure on the channel to the data server. Fixes #4933. * Fix query context doc.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If the result set size is very large and the client making the query is not pulling the results out fast enough, brokers can go OOM. This is because the broker lacks backpressure for large result sets, and they can "pile up" in memory. I believe that the specific part lacking backpressure is the collection and merging of results from individual historical (and realtime) nodes.
This is mostly an issue with scan queries and groupBy queries, which can generate large result sets.
When implementing backpressure it would be important to consider the effect this may have on the historical nodes. Backpressure would presumably extend all the way down to them, and in that case, would block threads their http server pools.
Related: #4229, which introduced
maxScatterGatherBytesas a kind of workaround to prevent broker crashes (although it will fail queries).Also related: #4865, a symptom.
The text was updated successfully, but these errors were encountered: