Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement memory management in shuffle writer #320

Open
andygrove opened this issue Oct 5, 2022 · 1 comment
Open

Implement memory management in shuffle writer #320

andygrove opened this issue Oct 5, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@andygrove
Copy link
Member

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There is no memory management in shuffle writer

Describe the solution you'd like

Blaze implements file-based shuffle with memory management in https://github.com/blaze-init/blaze/blob/master/native-engine/datafusion-ext/src/shuffle_writer_exec.rs

Perhaps we can learn from that?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
See discussion in apache/datafusion#3493 (comment)

@andygrove andygrove added the enhancement New feature or request label Oct 5, 2022
@mingmwang
Copy link
Contributor

I will take care of this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants