Add Mongo Aggregation Framework to Meteor #644

Closed
wants to merge 1 commit into from

6 participants

@wihl

I've started to add support for Mongo's Aggregation framework to Meteor's mongo_driver. It should be able to return a cursor so it can be used with the rest of Meteor's Collections. However, I'm not sure the best place to wire it up. Please take a look at this pull request and if you provide a little guidance, I'll be happy to finish this off. Thanks.

@wihl

Here's how to use Mongo's Aggregation Framework without making any changes to Meteor.

Caveats:

  • This is subject to change as future versions of Meteor come out.
  • This is non-reactive. The client-side piece could be put into a reactive variable. The server-side could poll the database for changes if necessary.
  • The result is going into a Meteor method rather than a publish. It could be put into a publish if necessary.

Server:

var path = __meteor_bootstrap__.require('path');
var MongoDB = __meteor_bootstrap__.require('mongodb');
var Future = __meteor_bootstrap__.require(path.join('fibers', 'future'));

var Animals = new Meteor.Collection("animals");

Meteor.startup(function () {
  Animals.aggregate = function(pipeline) {
    var self = this;

    var future = new Future;
    self.find()._mongo.db.createCollection(self._name, function (err, collection) {
      if (err) {
        future.throw(err);
        return;
      }
      collection.aggregate(pipeline, function(err, result) {
        if (err) {
          future.throw(err);
          return;
        }
        future.ret([true, result]);
      });
    });
    var result = future.wait();
    if (!result[0])
      throw result[1];

    return result[1];
  };

});

Meteor.methods({
  myAggregationMethod: function() {
    return Animals.aggregate([ {$project: {dog:1, age:1}}, {$sort:{age:1}} ]);
  }
});

and the client would be:

 Meteor.startup(function () {
    Meteor.call('myAggregationMethod', function(err,result) {
      if (err === undefined) {
        Session.set("myAggregatedDogs", result);
      } else {
        console.log(err);
      }
    });
  });

  Template.animals.animal = function () {
    return Session.get("myAggregatedDogs");
  }
@wihl wihl closed this Jan 30, 2013
@wihl wihl deleted the wihl:mongo-aggregate branch Jan 30, 2013
@jrhone

Hey Wihl, I'm trying to get the server-side piece to poll the database for changes by moving things from a meteor method into a publish. Having trouble though. Animals.aggregate seems to return an array and I dont think Meteor can Publish arrays, only cursors...

Exception from sub 8NGX2T8JN5KECLSDc Error: Publish function returned an array of non-Cursors
at _.extend._runHandler (app/packages/livedata/livedata_server.js:866:20)
at _.extend._startSubscription (app/packages/livedata/livedata_server.js:724:9)
at _.extend.protocol_handlers.sub (app/packages/livedata/livedata_server.js:524:12)
at _.extend.processMessage.processNext (app/packages/livedata/livedata_server.js:488:43)

Any thoughts?

@jrhone

In case anyone's wondering, I used the 'counts-by-room' example from http://docs.meteor.com/#meteor_publish to observeChanges on the Animal's collection, and updated my 'custom aggregate' collection accordingly.

@oscardebos

I've been trying to get this working within publish as well but i get the same error as jrhone. You mentioned "It could be put into a publish if necessary." could you explain me how to do this?

@jrhone

You'll see that we're using publish with aggregate in the gist below.

Aggregation occurs on the Calendar Collection. Results of the aggregation are stored in the Calories Collection. We observe changes on the Calendar Collection and re-run the aggregation when necessary, updating Calories.

https://gist.github.com/jrhone/5935725

I modified my code to create the gist and have not tested so it might not run cleanly, but you'll see the idea.

@awatson1978
@jrhone

Originally I just wanted to get pub/sub going with my aggregated data as quickly as possible. Saw this as a solution and ran with it, but it's just for my prototype. It's simple. It's realtime.

It'd be nice to move the aggregations out of meteor, store em permanently and make it so that all meteor has to do is publish the agg'd collection as you suggested. You proposed to do it in bulk which is fine, but there'd be added complexity to make it realtime, which I wanted. Also in your approach you're agg'ing across all users, and in mine i'm only agg'ing for the user that changed.

I wonder what kind of penalty we get for keeping the agg's inside meteor though. Is it more of a hit on mongo than a hit on meteor?

Secondly, how expensive is the observe call? Is that also more of a mongo hit than a meteor hit? I imagine it's not that bad since that's a part of meteor's bread and butter but I dont know.

But yes, in a robust system I'd probably move the aggs outside of meteor, and use a worker queue to control the max number of agg query's hitting mongo at any point in time, maybe re-agg'ing on each user update. Then we can either let meteor pick up the change in 10 seconds or maybe use ddp to do the write through meteor.

@oscardebos

Well i got it working but i think its a bit slow/heavy. What i'm trying to do is: I have a collection filled with events. For each event there are a lot of people who are attending or unsure, etc. The aggregate call that i'm making is to make a list sorted by the sum of friends of a user. The list of friends is an array i got from another collection and i use this with $match $in after i $unwind all the event members. This results in an array of events sorted by the amount of friends of a user who are attending. I got this running, but only outside of publish because self.added can only be used for one item. Besides this problem, i ask myself if this approach is suitable for meteor (the aggregation call takes about 97 milliseconds). I love working with meteor and mongodb and i would hope to get this running smoothly.

Thanx for the great help by the way!

@Luciuz Luciuz referenced this pull request in arunoda/meteor-smart-collections Jan 8, 2014
Closed

Aggregate SmartCollection? #47

@timbrandin

Would be nice to know if Mongo's aggregate ever gets its way into Meteor.

@maxko87

To fix @wihl 's answer for Meteor 0.9.0, use this for the server code:

var path = Npm.require('path');
var Future = Npm.require(path.join('fibers', 'future'));

var Animals = new Meteor.Collection("animals");

Meteor.startup(function () {
  Animals.aggregate = function(pipeline) {
    var self = this;

    var future = new Future;
    self.find()._mongo._withDb(function(db) {
      return db.createCollection(self._name, function(err, collection) {
        if (err) {
          console.log(err);
          future.throw(err);
          return;
        }
        return collection.aggregate(pipeline, function(err, result) {
          if (err) {
            future.throw(err);
            return;
          }
          return future["return"]([true, result]);
        });
      });
    });
    var result = future.wait();
    if (!result[0])
      throw result[1];

    return result[1];
  };

});

Meteor.methods({
  myAggregationMethod: function() {
    return Animals.aggregate([ {$project: {dog:1, age:1}}, {$sort:{age:1}} ]);
  }
});

Notice: replace all __meteor_bootstrap__ with Npm.require and remove the "mongodb" dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment