Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add chunking to serializing models into Store #192

Open
bjester opened this issue Jul 19, 2023 · 0 comments
Open

Add chunking to serializing models into Store #192

bjester opened this issue Jul 19, 2023 · 0 comments

Comments

@bjester
Copy link
Member

bjester commented Jul 19, 2023

Overview

Serializing huge amounts of data into the Store could cause large increases in memory usage. Currently _serialize_into_store processes batches by model, but loads all of the model's data into memory before saving the bulk writes to the Store model.

Deliverables

  • Relocate and split up _serialize_into_store into smaller functions
  • Have the serialization process the model objects in chunks, or via a queryset iterator() (ref)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant