Major dereferencing optimizations and fix for de-pickling outdated documents [migrated 921] #947
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #888:
Dereferencing Optimizations:
x in dict
instead of listx in list
.Below is a preview of how the dereferencing optimizations improved loading 500 documents:
(This output was generated by GreenletProfiler)
The left shows the master branches performance when retrieving 500 documents all of which have 2 referenced fields.
The right shows the same when using this pull request.
Please note that the left displays O(n^2) where as the right displays O(n). Although the average time for each operation is
0.000005
(not shown above) as n increases this time becomes incredibly large.I found this issue when compiling 67,000 records took 1 hour