Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Insert and delete workloads on large datasets (~1B rows) can cause OOM #4

Open
kaustavc opened this issue Aug 12, 2021 · 0 comments
Open

Insert and delete workloads on large datasets (~1B rows) can cause OOM #4

kaustavc opened this issue Aug 12, 2021 · 0 comments
Assignees
@kaustavc
Labels
enhancement New feature or request

Comments

@kaustavc
Copy link
Collaborator

Currently we create a shuffled slice of all the possible row indexes. For 1B rows this can take GBs of memory and cause OOM. The way to assign unique random row ids to each worker can be rethought.
@kaustavc kaustavc added the enhancement New feature or request label Aug 12, 2021
@kaustavc kaustavc self-assigned this Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kaustavc kaustavc

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kaustavc