Skip to content

Memory accounting inaccuracies in multi level merge #23760

Description

@ariel-miculas

Describe the bug

Not enough memory is reserved for the SortPreservingMerge

  • track the record batch buffering (even with buffer_size=1 there's one record batch being prefetched by spawn_buffer)
  • take into account that for the minimum requirement for two stream merge, we need to reserve 3 record batches because there could be a record batch crossing happening inside one of the streams
  • for multi-column sort, take into account the additional overhead of ReusableRows

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions