Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major dereferencing optimizations and fix for de-pickling outdated documents [migrated 921] #947

Merged
merged 1 commit into from
Apr 9, 2015

Conversation

reallistic
Copy link
Contributor

Fixes #888:

  • When de-pickling a document that is not dynamic, the _fields_ordered of the current type should be used
    • When de-pickling a document that is dynamic, the pickled _fields_ordered should be used for the current document ONLY

Dereferencing Optimizations:

  • Compiles the set of dbrefs as a set instead of a list so that conversions don't need to be made later
  • Checks for resolved objects via dict x in dict instead of list x in list.

Below is a preview of how the dereferencing optimizations improved loading 500 documents:

name # of operations name # or operations
..bson/objectid.py:288 ObjectId.eq 250998 ..bson/objectid.py:288 ObjectId.eq 998

(This output was generated by GreenletProfiler)

The left shows the master branches performance when retrieving 500 documents all of which have 2 referenced fields.
The right shows the same when using this pull request.
Please note that the left displays O(n^2) where as the right displays O(n). Although the average time for each operation is 0.000005 (not shown above) as n increases this time becomes incredibly large.
I found this issue when compiling 67,000 records took 1 hour

Review on Reviewable

Optimize dereferencing map by using sets.
@landscape-bot
Copy link

Code Health
Repository health decreased by 0.01% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 73.06% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

4 similar comments
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 73.06% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 73.06% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 73.06% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 73.06% when pulling a6a7cba on YoApp:optimization_issue888 into 1ce6a7f on MongoEngine:master.

thedrow added a commit that referenced this pull request Apr 9, 2015
Major dereferencing optimizations and fix for de-pickling outdated documents [migrated 921]
@thedrow thedrow merged commit d600ade into MongoEngine:master Apr 9, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Un-pickling a document removes new fields from the global document Class
4 participants