Dramatically speed up MongoMapper::Plugins::Keys#load_from_database #450

wants to merge 3 commits into

6 participants


This pull request significantly enhances the performance of mongo-mapper when dealing with a large number of documents that may also have a large number of keys. When profiling our app, we noticed that MongoMapper::Plugins::Keys#load_from_database was taking a VERY long time. In fact, much longer than the time it took to get a response from mongo. For example, a 5kb response from mongo for a single flat document with about 40 keys would arrive in under 1ms, but the "load_from_database" method took over 17ms to complete. This method is only working with stuff that is already in memory and it was surprising that it took so long.

The first thing we noticed was the Keys#key? method. This was generating an array of keys and calling include? on it, instead of just calling has_key?. This was called quite a lot and the changes cut about 8ms off the single load_from_database call for that one example object; basically cutting the time in half. In our large test page, this was being called 34,667 times for 623 documents.

In the load_from_database method, we broke the different types of keys into separate lists, which saved doing all the respond_to? calls and allows us to more directly do what needs to be done for each key type. This also eliminated almost all calls to key?.

Then we deferred building the associations until needed. The Keys#load_from_database method will simply store the value for associations being loaded into an instance variable, "@association#{key}", for future evaluation.

These changes alone reduced the average response time for a particular page of ours which loaded 623 documents from 8.1 seconds to 1.4 seconds. keys#load_from_database went from 4.617 seconds to 0.23 seconds and these changes also helped other aspects of the app. Naturally, your mileage may vary.

I have re-run the mongo-mapper tests and they all still pass.


Heads up, the travis build is broken due to changes on Travis's side.

This pull request fixes it if you want to cherry-pick the commit to get your PR build passing:


@jnunemaker jnunemaker commented on the diff Sep 21, 2012
@@ -12,10 +21,19 @@ def replace(values)
def find_target
- (@_values || []).map do |hash|
- child = polymorphic_class(hash).load(hash)
- assign_references(child)
- child
+ if !@_from_db
jnunemaker Sep 21, 2012

The if and else of this seem to be doing the same thing. Am I missing something?

jonpollock Sep 21, 2012

In one case it is doing a load and setting the _from_db to false and in the other it is doing a new.

jnunemaker Sep 21, 2012

I understand that the boolean switch, but how does that affect things?

jonpollock Oct 8, 2012

load would load the proxy the same way as load_from_database. The new, would end up setting the _new property to true, which would be wrong.

We truly are loading it from the database, just not evaluating what we received from the database right away.

@jnunemaker jnunemaker commented on the diff Sep 21, 2012
@@ -72,10 +73,20 @@ def build_proxy(association)
def get_proxy(association)
- unless proxy = self.instance_variable_get(association.ivar)
- proxy = build_proxy(association)
+ value_from_db = instance_variable_get(:"@_association_#{association.name}")
+ if value_from_db
+ instance_variable_set :"@_association_#{association.name}", nil
+ unless proxy = self.instance_variable_get(association.ivar)
+ proxy = build_proxy(association)
+ end
+ association.embeddable? ? proxy.load_from_database(value_from_db) : proxy.replace(value_from_db)
jnunemaker Sep 21, 2012

What is the benefit of this? Seems like load_from_database and replace are identical.

jonpollock Oct 8, 2012

In this particular case, it turns out that association should always be embeddable? and we can reduce that to just proxy.load_from_database.

In any case, the main difference is that load_from_database sets the @_from_db attribute to true and replace sets it to false.

In general, when calling the proxy's find_target, @_from_db controls whether it will do a load or a new. load will generally go to the one defined in keys, which may be replaced with the identity map. It is the method that would have been called by load_from_database if we didn't skip loading the associations.

If we used new, the _new property would get set to true, which we don't want.

The whole point is to defer the evaluating the proxy until it is needed. This saved us quite a bit of time when loading lists of documents. Especially, when it contained embedded documents we didn't need for the particular operation we were doing.

@jnunemaker jnunemaker commented on the diff Sep 21, 2012
@@ -13,6 +13,9 @@ module Keys
module ClassMethods
def inherited(descendant)
+ descendant.instance_variable_set(:"@key_names", key_names.dup)
jnunemaker Sep 21, 2012

We definitely should be doing this stuff.

@jnunemaker jnunemaker commented on the diff Sep 21, 2012
def key?(key)
- keys.keys.include?(key.to_s)
+ keys.has_key?(key.to_s)
jnunemaker Sep 21, 2012

Good catch. I'll fix this in master right now.


I really appreciate the effort towards improving performance. Performance hasn't been really focused on the past several releases and it needs to be again. I've picked a few of these changes into master already.

Some of the changes here are hard to read through and create some duplication problems (date conversion outside of Key#get/set).

Any chance you could paste an example document here? Or maybe put a mongodump somewhere that I could paste through. Also, what was the overall method for testing? Did you just start adding timing stuff and see what took the longest?

I'm not surprised that load from database takes the most time as that is where everything happens (typecasting, etc.). You definitely have me interested in speeding things up, but I want to do it in a way that doesn't affect duplication or readability. Does that make sense?

Again, thanks for bringing this up and any more help you can provide.


The special Date handling may no longer be necessary. We weren't doing a final key.get(read_value) for the value we put into the @read{key} instance variable. I noticed that just before making the pull request and fixed that, but didn't go back and remove the date handling. I'll experiment with that and update the pull request. This would certainly make it cleaner.

For the most part we used newrelic's profiling capability to see these hotspots. In other cases, I just put timing into the code.

I'll see what I can do about getting you a sample document. The one that I noticed the keys.key? problem was our site settings document, which had 131 keys defined on it.


jonpollock: Don't forget that we went back and validated the performance gains at the web page level using jmeter after these changes went in. It made a huge difference on overall site performance. Some pages got 10x faster. Not every page, but particularly database-heavy ones did.

MongoMapper member

I'm immensely interested in this changeset, as #read_key is the lion's share of my profiling efforts right now, but it breaks the test suite six ways from sunday. Do you all see the same issues?


The biggest improvement is just caching read attribute in an instance variable. Right now MM stupidly retypcasts all the time instead of just reading. I definitely want to fix that. I suppose I could bring in just the read attribute caching, but overall I'm not a huge fan of this changeset as I find it really hard to follow.

MongoMapper member

I'm actually working on just the read attribute caching in a separate branch, as I also had a hard time following this changeset, but the read attribute caching has very obvious benefits.

MongoMapper member

https://github.com/cheald/mongomapper/commits/faster_read_keys/ has my changes so far. I'm working on some further experiments, though, as a not-insignificant amount of time is spent in String#intern and Keys#keys, both of which I'm trying to whittle down. My gut is that this can all be significantly faster.

I'll submit a pull request once I've concluded this round of experiments.

MongoMapper member

This PR has been mostly deprecated by the recent overhauls to Keys. However, I still like the idea of deferring embedded document loads until they're actually invoked. The following does that nicely.

Thoughts? This is an easy enough change, and quick tests seem to indicate that it substantially improves load performance in cases where you have embedded documents that you don't actually touch (for obvious reasons).

diff --git a/lib/mongo_mapper/plugins/associations.rb b/lib/mongo_mapper/plugins/associations.rb
index d69a8c7..af7ae5a 100644
--- a/lib/mongo_mapper/plugins/associations.rb
+++ b/lib/mongo_mapper/plugins/associations.rb
@@ -73,13 +73,20 @@ module MongoMapper
       def build_proxy(association)
         proxy = association.proxy_class.new(self, association)
         self.instance_variable_set(association.ivar, proxy)

       def get_proxy(association)
         proxy = self.instance_variable_get(association.ivar) if instance_variable_defined?(association.ivar)
         proxy ||= build_proxy(association)
+        realize_association(association.name, proxy)
+      end
+      def realize_association(name, proxy)
+        if @__deferred_associations && @__deferred_associations.key?(name)
+          proxy.replace @__deferred_associations.delete(name)
+        end
+        proxy

       def save_to_collection(options={})
diff --git a/lib/mongo_mapper/plugins/keys.rb b/lib/mongo_mapper/plugins/keys.rb
index 3d3de71..de43ed4 100644
--- a/lib/mongo_mapper/plugins/keys.rb
+++ b/lib/mongo_mapper/plugins/keys.rb
@@ -289,9 +289,13 @@ module MongoMapper

           # Init the keys ivar. Due to the volume of times this method is called, we don't want it in a method.
           @_mm_keys ||= self.class.keys
+          @__deferred_associations ||= {}

           attrs.each do |key, value|
-            if !@_mm_keys.key?(key) && respond_to?(:"#{key}=")
+            s_key = key.to_sym
+            if associations.key?(s_key)
+              @__deferred_associations[s_key] = value
+            elsif !@_mm_keys.key?(key) && respond_to?(:"#{key}=")
               self.send(:"#{key}=", value)
               internal_write_key key, value, false

The keys.rb file has certainly changed since I last looked at it. Yes, this seems like an appropriate way to solve it. We have found that one of the biggest performance gains was in deferring the loading of these embedded documents.


Does that last patch work when #attributes() is called?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment