You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We already have a mechanism to reliably bound memory consumption during partial aggregation phase (when we consume table rows and accumulating per-thread aggregation states). This mechanism is called "external aggregation" and the mentioned memory bound is provided by the user through the max_bytes_before_external_group_by setting. Currently maintained aggregation states are flushed to disk once this limit is reached.
During merging we don't have such guarantees: the amount of data residing in memory is proportional to total_aggregation_state_size * (aggregation_memory_efficient_merge_threads / MAX_BUCKETS), MAX_BUCKETS value is constant and equals 256.
This fact complicates running CH in memory constrained environment.
Use case
Aggregation with large state that doesn't fit in memory limit during merging phase.
Describe the solution you'd like
Implement merging algorithm similar to the one we use in aggregation in order.
We already have a mechanism to reliably bound memory consumption during partial aggregation phase (when we consume table rows and accumulating per-thread aggregation states). This mechanism is called "external aggregation" and the mentioned memory bound is provided by the user through the
max_bytes_before_external_group_by
setting. Currently maintained aggregation states are flushed to disk once this limit is reached.During merging we don't have such guarantees: the amount of data residing in memory is proportional to
total_aggregation_state_size * (aggregation_memory_efficient_merge_threads / MAX_BUCKETS)
,MAX_BUCKETS
value is constant and equals 256.This fact complicates running CH in memory constrained environment.
Use case
Aggregation with large state that doesn't fit in memory limit during merging phase.
Describe the solution you'd like
Implement merging algorithm similar to the one we use in aggregation in order.
The text was updated successfully, but these errors were encountered: