Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize the batch_write operation #1

Open
naschorr opened this issue Jan 24, 2022 · 1 comment
Open

Optimize the batch_write operation #1

naschorr opened this issue Jan 24, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@naschorr
Copy link
Owner

Writing the DynamoDB specified 25 items per batch is super slow, however the docs mention that using multiple threads for the batch operation is totally fine.

Alternatively, the native batch_writer seems to be able to handle many more than 25 items at once. More experimentation is needed.

@naschorr naschorr added the enhancement New feature or request label Jan 24, 2022
@naschorr naschorr self-assigned this Jan 27, 2022
@naschorr
Copy link
Owner Author

As a follow up, it looks like even if you pass batch_writer N > 25 items, it still processes it in chunks of 25. That said (anecdotally), I did notice that data copying seemed to be happening slightly quicker. Maybe a couple minutes faster per chunk of ~7000 items from the AWS instance.

That said, I didn't robustly profile the operation, so it could just be a fluke.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant