Skip to content

Implement arrow_buffer::MemoryPool for DataFusion's MemoryPool #18926

@notfilippo

Description

@notfilippo

Is your feature request related to a problem or challenge?

Related to #16841. The ability to correctly account for memory usage of arrow buffers in execution nodes is crucial to maximise resource usage while preventing OOMs.

Describe the solution you'd like

Implement the arrow_buffer::MemoryPool trait for DataFusion's MemoryPool to make Arrow arrays claimable.

Describe alternatives you've considered

See the discussion at #16841

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions