Skip to content

Make BUFFER_CAPACITY_AFTER_SORT_ELIMINATION configurable #21417

@adriangb

Description

@adriangb

#21182 introduces elimination of SortExec through sort pushdown. This resulted in slowdowns for some queries because SortExec (which loads all data into memory for sorting) acted as a buffer between SortPreservingMergeExec and the I/O bound scans. This caused I/O fragmentation and slowdowns. We introduced BufferExec to restore this buffering behavior. This new setup uses strictly less memory than SortExec because it doesn't need to buffer the entire dataset in memory, just enough to keep SortPreservingMergeExec constantly fed. BufferExec does support a memory limit for buffering (and will also respect the global memory limits / spill where needed). We hard coded this memory limit to 64MB since that is enough for our benchmark data, however we think it should be made configurable and defaulted to something more like 1Gi or 2Gi (large enough to hold at least 1 maybe 2 row groups for wide data sets). There is little negative to making this number large since if memory limits are reached BufferExec will spill to disk, and no matter how big we make the number memory use will still be lower than it was with SortExec. It may cause higher peak memory use and not make the query any faster, so a sensible default does make sense. 1Gi seems like a reasonable number.

Metadata

Metadata

Assignees

Labels

No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions