-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
groupBy query: limit push down to segment scan is poor performance #9689
Comments
Thanks for the report @xiangqiao123 , do you have any additional details on the query you issued, such as what the limit was, any extractionFn or lookups, and any details about segment size, column type, if string approximate cardinality, or anything else you think might be useful in helping us determine why the performance degraded? |
@xiangqiao123 |
It looks like the The perf regression is probably because #8426 was added in 0.17.0, and before then, we wouldn't push down the limit all the way to the segment. I bet the pushdown to the merge buffer level isn't as bad because there's only one merge buffer per query, so the cost of init is only paid once, instead of per segment. I think to improve the performance we'd need AlternatingByteBufferHashTable to implement growability. |
|
@xiangqiao123 that is simplest of queries in terms of aggregation with a largish limit, even with efforts to reduce overheads there would be a non-zero cost in pushing the limit to segment scan phase, for your use case I guess it would be advisable to disable limit pushdown to segments (in reality, you might need to measure before blindly believing in my claim though :) @gianm I agree with your observations. it sucks that we need to implement grow-ability to save cost of zeroing out where we already have allocated all the memory or is there any other advantage?. another consideration, I don't know/remember if we tried, a different layout for marking used/unused buckets. |
The problem is the memory is not initialized when we allocated it, so it has garbage in it, and therefore we need to initialize the parts we're going to use. And initialization can take a while if your buffer size is big (like a gigabyte, as some people do).
I bet it would be faster, especially if we use the Memory.clear API. I did some work to migrate some of the groupBy code to use Memory instead of ByteBuffer (#9308, #9314; see also MemoryBenchmark) partially motivated by performance. If we moved this part too it should help with initialization speed. But growability would probably help even more. It should be relatively straightforward for AlternatingByteBufferHashTable: each time we grow we swap into the other half of the buffer. |
thanks for pointing those out . from a quick look, it looks like we should just delete agree with everything, all of them could/should be independently done. |
I don't see a reason to keep ByteBufferHashTable around. The only reason I didn't replace all usages of it was time constraints. Although for the LimitedBufferHashGrouper. specifically, I think in order to migrate that to MemoryOpenHashTable, we'd need to also add a MemoryMinMaxHeap (replacing ByteBufferMinMaxOffsetHeap). I would suggest refactoring the code of LimitedBufferHashGrouper a bit so rather than subclassing the hashtable and overriding stuff, it instead has a (not-subclasses) MemoryOpenHashTable and a MemoryMinMaxHeap and uses them both together to achieve its ends. |
Affected Version
0.17.1
Description
When druid-0.17.1 is used, the performance of group by query is worse than druid-0.16.1. When
applylimitpushdowntosegment is set to false in query context, the performance returns to normal.
From this figure, we can see that the CPU consumption is in LimitedBufferHashGrouper.
![image](https://user-images.githubusercontent.com/5836707/79065418-bdfaba00-7ce2-11ea-8c86-6b4ce88b0f94.png)
query performance is according to query/time from historical log:
{"query/time":13372,"query/bytes":6118553,"success":true,"identity":"allowAll"}
@himanshug
The text was updated successfully, but these errors were encountered: