Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Rust] [DataFusion] MemTable::load() should load partitions in parallel #26248

Closed
asfimport opened this issue Oct 9, 2020 · 1 comment
Closed

[Rust] [DataFusion] MemTable::load() should load partitions in parallel #26248

asfimport opened this issue Oct 9, 2020 · 1 comment
Assignees
@andygrove
Labels
Component: Rust - DataFusion Component: Rust Type: enhancement
Milestone
2.0.0

Comments

@asfimport
Copy link

MemTable::load() should load partitions in parallel using async tasks, rather than loading one partition at a time.

Also, we should make batch size configurable. It is currently hard-coded to 1024*1024 which can be quite inefficient.

Reporter: Andy Grove / @andygrove
Assignee: Andy Grove / @andygrove

PRs and other links:

Note: This issue was originally created as ARROW-10251. Please see the migration documentation for further details.
@asfimport
Copy link
Author

Andy Grove / @andygrove:
Issue resolved by pull request 8428
#8428

@asfimport asfimport added this to the 2.0.0 milestone Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
Component: Rust - DataFusion Component: Rust Type: enhancement
Projects
None yet
Milestone
2.0.0
Development

No branches or pull requests
2 participants
@andygrove @asfimport