Use iterator() when rehashing to save memory #4139
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Label error. Requires at least 1 of: Feature, Bug, Enhancement, Maintenance, Documentation, Performance, Do not publish. Found: |
nllong
commented
Jul 12, 2023
|
|
||
| for state in property_states: | ||
| print(f"Re-hashing {property_count} Property States ") | ||
| for idx, state in enumerate(PropertyState.objects.all().iterator()): |
There was a problem hiding this comment.
this is the only real change
perryr16
approved these changes
Jul 12, 2023
axelstudios
approved these changes
Jul 12, 2023
yeah, I think the memory pressure probably started with the |
dhaley
pushed a commit
that referenced
this pull request
Jul 21, 2023
use iterator() when going through all the states to save memory
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Any background context you want to provide?
The
0200_rehashmigration was failing due to memory pressure when migrating the production database with over 14E6 states.What's this PR do?
Convert the
.all()call to.all().iterator()to prevent all records from returing at one time, rather force a database iterator.This migration still takes a long time to run--around 10 hours. We might need to parallalize this in the future.
How should this be manually tested?
If CI passes, then there are no syntax errors. The PR does not change the core functionality.
What are the relevant tickets?
n/a
Screenshots (if appropriate)