Skip to content
This repository

loadMany 40X performance improvement, array observer notifications only on load completion. #800

Closed
wants to merge 2 commits into from

7 participants

Alex Speller Stefan Penner Conrad VanLandingham Peter Wagenet Ray Tiley Kris Selden Yehuda Katz
Alex Speller

So, this started as an attempt to improve load performance, which it does, however it also solves another annoying bug, namely that observers fire for every single record loaded into a store.

The performance is way better in this version, for a large number of records (~10,000) load time is about ~500ms vs ~20s in chrome on a recent MacBook Pro. It also solves a really annoying problem of computed properties / observers, namely that when they observe an array that is loaded from ember data they fire for every object loaded into the store, not just when all the objects are loaded.

This problem can be demonstrated here in these jsFiddles.

Fiddle with ember-data master version

Fiddle with this version of ember-data

As you can see in the fiddles, my version only logs a property change when all models are loaded into the store, wheras for the current master there is a log for each individual object loaded into the store. When there are a large number of models this quickly becomes a problem (even for smaller numbers of models it is a pain, as if you have processing that takes e.g. 500ms in an array observer, and you then load 10 objects into that array, the app will freeze for 5 seconds).

There are failing tests in this pull request - I have not been able to fix them and they seem to run fine individually with rackup but not together. I don't have more time to work on fixing the tests at the moment and am using a different approach now, however these performance improvements and bugfix work great apart from the tests, so I'm filing this pull request in the hope that someone more experienced with the tests can take a look.

Stefan Penner
Owner

Seems like a good direction to take. I will take some time this evening to thoroughly review.

A high level concern: it does look like many new lines of code where added, but no test coverage.

Alex Speller

Awesome.

As indicated in this comment, the new code is basically a refactoring of the code in the updateManyArrays and updateManyArray methods, and thus covered by existing tests. Perhaps a new test for the bug mentioned in the jsFiddle would make sense, but ultimately removing the updateManyArray and updateManyArrays methods entirely probably makes more sense.

I didn't want to do that because there are tests that break when you do that. These tests explicitly expect arrayContentDidChange to be called with arguments specifying which record was added, and this doesn't make sense any more if you're updating the array in a batch like this. However as there are tests explicitly about this behaviour I didn't want to remove it as it might cause breaking changes to users if they rely on it. This is why the old code path is still taken when only one record is changed, which is probably not what we want.

Alex Speller

I cleaned this up a bit and made some more tests pass, I actually deleted the old method, there are still some tests failing but less now. If you would merge this if the tests were passing, let me know and I will keep bashing my head against the tests, but don't want to waste time if it's for nothing.

Conrad VanLandingham

I'm generally against +1 posts, but this would help me out a lot as well.

Peter Wagenet
Owner

@alexspeller Sorry for being so slow to comment on this. Ideally, we'd fix the behavior of arrays so this wouldn't even be an issue. That said, until we can do that, this might be a good stopgap. We'd also need to have all the tests passing of course :)

Ray Tiley

Any update to the status on this? I'm having big issues when trying to use ember data with endpoints that return 100 -> 5000 results. This sounds like it could be the reason. I'm willing to dive in and try making the tests pass if this is the direction the core team wants to go?

Alex Speller

I'm afraid that I haven't had any more time to work on this - in fact I've pretty much stopped using ember data altogether. If you're working with endpoints returning 5000 results, I would recommend doing the same presently, I spent a long time fighting performance issues before giving up for large datasets :-(

Stefan Penner
Owner

@wagenet should we get this in as a short-term fix? (cc: @tomdale) ?

Kris Selden
Owner

@stefanpenner tests fail, I think it was a WIP

Alex Speller

There are failing tests, and after spending about 2 days bashing my head against them I gave up and went for a different approach. This needs someone who understands ember internals better than me to look at, I think. Sorry, I tried my best and failed :)

Ray Tiley

But is the idea sound? With some work and passing tests could it be merged in? I have an app that needs to load up to a few thousand records when initiated, and it kills the UI. That is if chrome doesn't flat out crash.

Alex Speller

@raytiley I think the idea is sound, as far as I can tell - however, even with this performance enhancement, I think you will find ember-data too slow at the present time for multi-thousand recordsets. I was using this patched version in an app and even with this patch, using a custom solution was a lot faster.

Stefan Penner
Owner

@alexspeller alright, seems legit. We can treat this PR, as an identified problem, still pending proper solution.

Likely that perfect solution is making array observers themselves async.

Peter Wagenet
Owner

I think this will be addressed by async observers and possibly computed array properties: emberjs/ember.js#2711

Yehuda Katz
Owner

@alexspeller can you rebase this against master and compare this with Ember Data master? I'm definitely still interested in this.

Alex Speller

I will, I'm super busy this week but next week I'll post some up to date benchmarks

Peter Wagenet
Owner
Alex Speller

Sorry this took so long, unsurprisingly the code has changed massively and I had to basically start again.

In short, this is no longer relevant. A similar approach seems to have been taken to ensuring that record arrays are not updated inefficiently (see this function in RecordArrayManager).

So this approach to improving performance is obsolete and large performance gains have already been seen thanks to ongoing improvements. I am currently seeing times of 2-3 seconds after ajax response complete for 10,000 row recordset. Good work ED team!

Alex Speller alexspeller closed this
Alex Speller alexspeller deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Mar 13, 2013
Alex Speller Initial attempt 078c60e
Mar 14, 2013
Alex Speller Attempt 2 d4e4cc4
This page is out of date. Refresh to see the latest.
132  packages/ember-data/lib/system/store.js
@@ -1368,86 +1368,104 @@ DS.Store = Ember.Object.extend(DS._Mappable, {
1368 1368
         }
1369 1369
 
1370 1370
         if (shouldFilter) {
1371  
-          this.updateRecordArray(array, filter, type, clientId);
  1371
+          this.updateRecordArraysLater(type, clientId);
1372 1372
         }
1373 1373
       }
1374 1374
     }
1375 1375
   },
1376 1376
 
1377  
-  updateRecordArraysLater: function(type, clientId) {
1378  
-    Ember.run.once(this, function() {
1379  
-      this.updateRecordArrays(type, clientId);
1380  
-    });
  1377
+  clientIdsToUpdate: {},
  1378
+  typesToUpdate: [],
  1379
+
  1380
+  doRecordArrayUpdate: function() {
  1381
+    for (var i = 0; i < this.typesToUpdate.length; i++) {
  1382
+      var type = this.typesToUpdate[i];
  1383
+      var clientIds = this.clientIdsToUpdate[type] || [];
  1384
+      this.updateRecordArrays(type, clientIds);
  1385
+      delete this.clientIdsToUpdate[type];
  1386
+    }
  1387
+    this.typesToUpdate = [];
1381 1388
   },
1382 1389
 
1383 1390
   /**
1384 1391
     @private
1385 1392
 
1386  
-    This method is invoked whenever data is loaded into the store
1387  
-    by the adapter or updated by the adapter, or when an attribute
1388  
-    changes on a record.
1389  
-
1390  
-    It updates all filters that a record belongs to.
1391  
-
1392  
-    To avoid thrashing, it only runs once per run loop per record.
  1393
+    This method is invoked when loading a large number of records
  1394
+    into the store in a single runloop. It ensures that records
  1395
+    are loaded fast and only fires notifications once all records
  1396
+    are loaded.
1393 1397
 
1394 1398
     @param {Class} type
1395  
-    @param {Number|String} clientId
  1399
+    @param {Array} clientIds
1396 1400
   */
1397  
-  updateRecordArrays: function(type, clientId) {
1398  
-    var recordArrays = this.typeMapFor(type).recordArrays,
1399  
-        filter;
  1401
+  updateRecordArrays: function(type, clientIds) {
  1402
+    // ensure clientIds is an array
  1403
+    clientIds = [].concat(clientIds);
1400 1404
 
1401  
-    recordArrays.forEach(function(array) {
1402  
-      filter = get(array, 'filterFunction');
1403  
-      this.updateRecordArray(array, filter, type, clientId);
1404  
-    }, this);
  1405
+    var clientId, i, j;
  1406
+    var recordArrays = this.typeMapFor(type).recordArrays;
1405 1407
 
1406  
-    // loop through all manyArrays containing an unloaded copy of this
1407  
-    // clientId and notify them that the record was loaded.
1408  
-    var manyArrays = this.loadingRecordArrays[clientId];
  1408
+    var mapContentToClientIds = function (c) {
  1409
+      return c.clientId;
  1410
+    };
1409 1411
 
1410  
-    if (manyArrays) {
1411  
-      for (var i=0, l=manyArrays.length; i<l; i++) {
1412  
-        manyArrays[i].loadedRecord();
1413  
-      }
  1412
+    for (i = 0; i < recordArrays.length; i++) {
  1413
+      var array             = recordArrays[i];
  1414
+      var filter            = get(array, "filterFunction");
  1415
+      var content           = get(array, "content");
  1416
+      var contentClientIds  = map(content, mapContentToClientIds);
  1417
+      var shouldBeInArray   = true;
  1418
+
  1419
+      for (j = 0; j < clientIds.length; j++) {
  1420
+        clientId            = clientIds[j];
  1421
+        var record              = this.clientIdToData[clientId];
  1422
+        var clientRecordArrays  = this.recordArraysForClientId(clientId);
  1423
+        var reference           = this.referenceForClientId(clientId);
  1424
+
  1425
+        if (filter) {
  1426
+          record = this.findByClientId(type, clientId);
  1427
+          shouldBeInArray = filter(record);
  1428
+        }
1414 1429
 
1415  
-      this.loadingRecordArrays[clientId] = null;
  1430
+        if (shouldBeInArray) {
  1431
+          clientRecordArrays.add(array);
  1432
+          if (!~contentClientIds.indexOf(clientId)) {
  1433
+            content.push(reference);
  1434
+          }
  1435
+        } else {
  1436
+          clientRecordArrays.remove(array);
  1437
+          array.removeReference(reference);
  1438
+        }
  1439
+      }
  1440
+      content.arrayContentDidChange();
1416 1441
     }
1417  
-  },
1418 1442
 
1419  
-  /**
1420  
-    @private
1421  
-
1422  
-    Update an individual filter.
1423  
-
1424  
-    @param {DS.FilteredRecordArray} array
1425  
-    @param {Function} filter
1426  
-    @param {Class} type
1427  
-    @param {Number|String} clientId
1428  
-  */
1429  
-  updateRecordArray: function(array, filter, type, clientId) {
1430  
-    var shouldBeInArray, record;
1431  
-
1432  
-    if (!filter) {
1433  
-      shouldBeInArray = true;
1434  
-    } else {
1435  
-      record = this.findByClientId(type, clientId);
1436  
-      shouldBeInArray = filter(record);
  1443
+    for (i = 0; i < clientIds.length; i++) {
  1444
+      clientId = clientIds[i];
  1445
+      // loop through all manyArrays containing an unloaded copy of this
  1446
+      // clientId and notify them that the record was loaded.
  1447
+      var manyArrays = this.loadingRecordArrays[clientId];
  1448
+      if(manyArrays) {
  1449
+        for(j = 0; j < manyArrays.length; j++) {
  1450
+          manyArrays[j].loadedRecord();
  1451
+          this.loadingRecordArrays[clientId] = null;
  1452
+        }
  1453
+      }
1437 1454
     }
1438 1455
 
1439  
-    var content = get(array, 'content');
1440  
-
1441  
-    var recordArrays = this.recordArraysForClientId(clientId);
1442  
-    var reference = this.referenceForClientId(clientId);
  1456
+  },
1443 1457
 
1444  
-    if (shouldBeInArray) {
1445  
-      recordArrays.add(array);
1446  
-      array.addReference(reference);
1447  
-    } else if (!shouldBeInArray) {
1448  
-      recordArrays.remove(array);
1449  
-      array.removeReference(reference);
  1458
+  updateRecordArraysLater: function(type, clientId) {
  1459
+    if (this.typesToUpdate.indexOf(type) === -1) {
  1460
+      this.typesToUpdate.push(type);
  1461
+    }
  1462
+    if (!this.clientIdsToUpdate[type]) {
  1463
+      this.clientIdsToUpdate[type] = [];
  1464
+    }
  1465
+    if(this.clientIdsToUpdate[type].indexOf(clientId) === -1) {
  1466
+      this.clientIdsToUpdate[type].push(clientId);
1450 1467
     }
  1468
+    Ember.run.once(this, this.doRecordArrayUpdate);
1451 1469
   },
1452 1470
 
1453 1471
   /**
4  tests/ember_configuration.js
@@ -108,7 +108,9 @@
108 108
       didUpdateAttribute: syncForTest(),
109 109
       didUpdateAttributes: syncForTest(),
110 110
       didUpdateRelationship: syncForTest(),
111  
-      didUpdateRelationships: syncForTest()
  111
+      didUpdateRelationships: syncForTest(),
  112
+      updateRecordArrays: syncForTest(),
  113
+      updateRecordArraysLater: syncForTest()
112 114
     });
113 115
 
114 116
     DS.Model.reopen({
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.