You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, hash aggregates are performing well when having a small number of output groups, but the results on db-benchmark h2oai/db-benchmark#182 test on data with a high number of output groups. #9234 improved the situation a bit, but DataFusion is still much slower than even the slowest result when comparing to the published results.
This seems mostly having to do with the way we use individual key/groups.
For each new key, we take the indices of the group, resulting in lots of small allocations and cache unfriendliness and other overhead if we have many keys with only a small (just 1-2) number of rows per group in a batch. Also the indices are converted from a Vec to an Array, making the situation worse (accounts for ~22% of the instructions on the master branch!), other profiling results seem to be from related allocations too.
To make it efficient for tiny groups, we should probably change the hash aggregate algorithm to take based on all indices from the batch in one go, and "slice" into the resulting array for the individual accumulators.
Here is some profiling info of the db-benchmark questions 1-5 against master:
Currently, hash aggregates are performing well when having a small number of output groups, but the results on db-benchmark h2oai/db-benchmark#182 test on data with a high number of output groups.
#9234 improved the situation a bit, but DataFusion is still much slower than even the slowest result when comparing to the published results.
This seems mostly having to do with the way we use individual key/groups.
For each new key, we take the indices of the group, resulting in lots of small allocations and cache unfriendliness and other overhead if we have many keys with only a small (just 1-2) number of rows per group in a batch. Also the indices are converted from a Vec to an Array, making the situation worse (accounts for ~22% of the instructions on the master branch!), other profiling results seem to be from related allocations too.
To make it efficient for tiny groups, we should probably change the hash aggregate algorithm to take based on all indices from the batch in one go, and "slice" into the resulting array for the individual accumulators.
Here is some profiling info of the db-benchmark questions 1-5 against master:
Reporter: Daniël Heres / @Dandandan
Assignee: Daniël Heres / @Dandandan
Original Issue Attachments:
PRs and other links:
Note: This issue was originally created as ARROW-11300. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: