Calling populate on embedded doc of embedded doc #601

Closed
dmmalam opened this Issue Nov 8, 2011 · 93 comments

Projects

None yet
dmmalam commented Nov 8, 2011

Hey,
I have the following schemas ( simplified for issue)

User = new Schema
login:String

A = new Schema
emb: [B]
comments: [Comments]

//and following embedded schemas

B = new Schema
comments: [Comments]

Comments = new Schema
creator:
type: ObjectId
ref: 'User'
message: String

Now while populate works on A
eg A.find().populate('comments.creator')

it doesn't work on the double nested embedded doc
eg A.find().populate('emb.comments.creator')

Any ideas?

Contributor
eneko commented Nov 9, 2011

The way I am doing that currently is populating the nested items and then iterating through them to populate the children. I was planing on implementing a solution for populating multiple nested levels, but it get very tricky.

Hi eneko,

How are you doing that?

When you do your first populate, you get returned the nested items but when you run through them to populate their children, are you able to save the populated children? It never sticks when I try to do it.

Some code would be awesome.

Thanks,
Paul

@eneko nevermind, just needed .toObject() ;)

Contributor
eneko commented Nov 15, 2011

Actually Paul, I figured populate() works on queries that return multiple elements, so there is no need to iterate through the child objects. But yes, you are right, you need to call .toObject if you want the properties to stick.

Here is an example:

// Ideally should be Parent.findOne().populate('children').populate('children.grandchildren').run();

function loadParentWithChildrenAndGrandChildren(parentId) {

  // Load parent without children references (children is array of ObjectId)
  Parent.findOne({ id: parentId }, { children: 0 }, function(err, parent) {
    if (err || !parent) return next(new Error("Parent not found: " + parentId));

    // Load children for this parent, populating grandchildren (no need to load parent reference)
    Children.find({ parent: parent._id }, { parent: 0 })
      .populate('grandchildren', [], { })
      .run(function(err, children) {
        if (err) return next(new Error("Could not load children: " + parentId));

        var result = parent.toObject();
        result.children = children;
        next(null, result);
      });
  });

}

Ah, thanks for your detailed answer. Your mongoose calls are more specific than mine and I could learn from your approach. As it happens my particular application required a recursive populate, so for now I may be stuck doing this manually.

I end up doing something like this.

var viewWithId_forDisplay = function(id, callback) {
if ( !id || typeof id === 'undefined' || id.toString().match(app_utils.emptyReg) ) {
    throw new Error('Bad user ID.');
}

View.findById(id)
.run(function(err, obj) {
    if ( err || !obj || !obj.subviews || !obj.subviews.length ) return callback(err, obj);

    obj = obj.toObject();

    // recursive subview fetch
    async.map( obj.subviews, viewWithId_forDisplay, function(err, results) {
        obj.subviews = results;

        return callback(err, obj);
    } );

})
}
dmmalam commented Dec 9, 2011

Does this now work in the latest mongoose release?

dbounds commented Dec 10, 2011

So I think this is the issue I'm experiencing. Each Comment object has an embedded User object. Comments are embedded as an array in Activity. A query for activities (as below) does not populate the User object on each comment.

var get_activities = function (callback) {
  var _this = this
  Activity.find({}, [], {sort:{ _id: -1 }})
  .populate('user')
  .populate('comments')
  .run(function (error, activities) {
    callback(error, activities)
  })
}

var User = new Schema({
    id: { type: String, required: true, lowercase: true, index: { unique: true } }
  , email_address: { type: String, required: true, index: true }
  , name: { type: String, required: true }
  , first_name: { type: String,  required: true } 
  , last_name: { type: String, required: true }
  , user_name : { type: String, required: true, lowercase: true, index: { unique: true } }
  , avatar_url: { type: String, required: true }
  , bio: String
  , following: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , followers: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
})

var Activity = new Schema({
    id: { type: Number, required: true, index: { unique: true } }
  , user: { type: Schema.ObjectId, ref: 'User', required: true }
  , recipients: [{ type: String, index: { unique: true } }]
  , type: String  
  , body: { type: String, required: true }
  , timestamp: { type: Date, required: true }
  , likes: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , comments: [{ type: Schema.ObjectId, ref: 'Comment' }]
})

var Comment = new Schema({
    id: { type: String, required: true, index: { unique: true } }
  , timestamp: { type: Date, required: true }
  , body: { type: String, required: true }
  , user: { type: Schema.ObjectId, ref: 'User', required: true } 
}) 
dethe commented Dec 13, 2011

Is there any plan for populate to support nested paths to handle cases like this?

Collaborator

yes we'd like to at some point. it could quickly become a performance issue but yeah we should support it.

dethe commented Dec 13, 2011

For nested objects I need to iterate over them manually, loading them, which seems like more of a performance issue and clutters up my server code. So I'm +1 for making my job easier ;-)

In the meantime I will continue, and also address it by splitting sections of the page into separately loadable objects that don't require such deeply nested data on each call.

Qard commented Dec 23, 2011

I'd be all for something like this;

Parent.findById(1).populate('child').run(function (err, parent) {
  parent.child.populate('grandchild', function (err) {
    console.log(parent.child.grandchild)
  })
})

It'd actually be pretty handy to have populate available on the document like that. You could keep diving as far as you need.

Collaborator

@Qard you read my mind.

Contributor
eneko commented Dec 23, 2011

@Qard that is how mongoose-relationships works. I used that module for a while at first, but ended up doing things manually. Maybe Mongoose should integrate it into the core, or at least parts of it like calling populate on arrays.

Has anything like #601 (comment) been implemented yet?

I'm wondering the same thing. I'm trying to figure out how to populate embedded doc child references. @dbounds I have an analogous models. Have you resolved this issue: #601 (comment). I can't find mongoose-relationships

The information in this thread has been very helpful in understanding the current state of .populate(), the current limitation related to nested population of grandchildren and how .toObject() in combination with manual lookup can be an option to workaround it.

I'm wondering if there is a technique we could use that does not require .toObject(). I ask b\c after we populate our model and the grandchildren we'd still like to have a Document and proper types so we can use functions like parent.childarray.id(x) or modify the values and call .save().

Any help here would be greatly appreciated.

note: We've tried using parent.set('child.grandchild', value), however that seems to cause some issues with the integrity of the Document and we are no longer able to read values from that path (error is Invalid ObjectId when trying to read parent.child or parent.child.grandchild).

vartana commented Jun 22, 2012

@aheckmann Any ideas on this yet?

Collaborator

been busy. i want it in 3.0 final

vartana commented Jun 22, 2012

@aheckmann Awesome cant wait. It's will be really useful in real applications. But I do want to give you kudos for the great job on what you have accomplished so far.

jsalonen commented Jul 9, 2012

Great awesomeness! I will use this feature as soon it is available.

Hey folks, I wrote my own layer around Mongoose that supports sub-population. It's not architecturally ideal since it's a wrapper around a wrapper, but it does support subpopulate the way we want it to work. Is anyone interested in seeing it if I clean it up and release it?

Definitely interested! I'm not sure if I'm the right person to do the patching though, but any ideas on how to work around this issue right now are very welcome!

Right now it's a hack, albeit a pleasant hack from my perspective :) Hopefully it can be used to patch Mongoose but it's a monkey-patch at the moment. I'll write some more tests, document my code, and come back here within the next week.

hackfrag commented Aug 9, 2012

Will this be added in version 3.0 ?

Collaborator

@hackfrag yes

Is this in 3.0 or is it still being added. If it is could we have an example please :)

Collaborator

it is not in 3.0. it will be added in an upcoming minor release.

stowns commented Aug 13, 2012

I'm having an incredibly tough time trying to hack this functionality. Do you have any estimate as to when this release might be available? I will ditch my ugly code and work on other areas of my application if true support is in the near future :) . And might I add, thanks for the awesome library.

Stowns, my hack is working nicely and I now feel good enough about releasing it. I'll get back to you within 24 hrs - need to test against Mongoose 3 first

-- Joshua Gross
Christian / Web Development Consultant / BA Candidate of Computer Science, UW-Madison 2013
414-377-1041 / http://www.joshisgross.com

On Aug 13, 2012, at 6:56 PM, stowns notifications@github.com wrote:

I'm having an incredibly tough time trying to hack this functionality. Do you have any estimate as to when this release might be available? I will ditch my ugly code and work on other areas of my application if true support is in the near future :)


Reply to this email directly or view it on GitHub.

madhums commented Aug 18, 2012

@JoshuaGross hey can you put your hack in a gist? my code is becoming too messy and ugly without this feature...

Hey @madhums, @stowns, @aheckmann, @farhanpatel, @hackfrag, @jsalonen, etc. My hack (only tested on Mongoose 2.7) is here: https://github.com/JoshuaGross/mongoose-subpopulate

Hope it's helpful.

WOW Joshua! Thanks a million!

I'm definitely gonna take a look it that right away!

stowns commented Aug 21, 2012

thanks @JoshuaGross , will check this out later today

That were to happen if any of the documents does not have the field to the "populate" is requesting to make the search? Would have errors? Technically there would be no error, but will be shown this documen(s)t?

chovy commented Oct 4, 2012

Do nested populate() calls work yet? It looks like there has been some activity, but I was unable to determine how to do it.

I have:

List.findById(req.params.id).populate('items').populate('items.user').exec(fn);

user is a deep nested...is it possible to populate?

@aheckmann can you comment on this? I've been using mongoose-subpopulate in production for quite a while to do subpopulates.

chovy commented Oct 4, 2012

mongoose-subpopulate does not work with express 3.

Collaborator

I have a branch started but been pretty busy. Its on its way.

On Wed, Oct 3, 2012 at 11:37 PM, Anthony Ettinger
notifications@github.comwrote:

mongoose-subpopulate does not work with express 3.


Reply to this email directly or view it on GitHubhttps://github.com/LearnBoost/mongoose/issues/601#issuecomment-9131991.

Aaron
@aaronheckmann https://twitter.com/#!/aaronheckmann

@chovy do you mean mongoose 3? I use mongoose-subpopulate with express 3.

chovy commented Oct 4, 2012

Yes, mongoose 3 I meant.

whito commented Oct 9, 2012

@aheckmann - Looking forward to see your branch!

chovy commented Oct 26, 2012

I could not figure out how to use mongoose-subpopulate

donnut commented Nov 2, 2012

+1 for this feature

+1 for this one

There is a way to populate an array child of an array?

var notifications = new Schema({
    _id             : { type : ObjectId }
    , from          : { type : ObjectId, ref: 'user' }
    , status                : { type : Number, default : 1 }
    , created_at            : { type : Date }
    , updated_at            : { type : Date }
});

var applications = new Schema({
    _id                     : { type : ObjectId, required : true, ref : 'application' }
    , notifications                 : [notifications]
});

var schema = new Schema({
    name                        : { type : String, required : true }
    , email                     : { type : String }
    , applications                      : [applications]
    , created_at                        : { type : Date }
    , updated_at                        : { type : Date }
});

var User = module.exports = mongoose.model('user', schema);

What I want to accomplish here its to get the name of the notification sender

// works like a charm
User.findOne({_id:'me'}).populate('applications');

// doesn't work
User.findOne({_id:'me'}).populate('applications.notifications.from');

Or if there is another way that do another query to get all the user names manually?

chovy commented Nov 16, 2012

you would need to make "from" an array like you're doing with notifications. The best thing to do is fetch the user for each notification in your callback.

yes, Im just doing the population manually, bull will be cool if populate works with deep references too...

+1, was hoping I would get to the bottom and see this was in and I just wasn't doing it right. :(

+1, any timeline estimate for this feature? It would be insanely helpful.

Collaborator

@winduptoy looking like 3.6

+1 here too.

More importantly, we should at least be able to populate the children ourselves without having to call toObject() on the parent object first. I can understand the desire to keep the field types consistent after the set call, but mongoose already breaks this rule with the populate function to start with. We should be able to set a referenced field to the complete object without mongoose changing it to merely the mongo id.

Other than this snafu, love mongoose.

koenoe commented Dec 12, 2012

I'm so looking forward to this feature.

+1 I also need this feature...

Contributor
Mikxail commented Dec 14, 2012

+1 for this future

+1 - looking forward to this one

+1 This would be awesome

+1 That came to software problems, but if you put another symbol would help, for example

User.findOne({_id:'me'}).populate('applications$notifications.from');
// Arrays $
// Simple document point "."

So do not avoid problems ... No?

chovy commented Dec 16, 2012

+2

+1 This would be helpful.

+1 That would be super good of you to do.

skotchio commented Jan 3, 2013

plese release that feature in 2013

devnieL commented Jan 12, 2013

+1 I need this right now, it will be very useful.

For those of you that need this right now, you might want to check out mongoose-subpopulate. I've been using it for several months: https://github.com/JoshuaGross/mongoose-subpopulate (I'm the maintainer)

Collaborator

Here's the plan for populate in 3.6: #1292

dexcell commented Jan 23, 2013

+1 We need this. Hopefully released soon, (because the issue already started year ago)

I think by popular vote, won this proposal!

@aheckmann aheckmann closed this Mar 5, 2013
jsalonen commented Mar 5, 2013

@aheckmann Thank you a lot!!

Also love lean() too so this seems really superb so far!

Excited for 3.6, this is a good fix, thanks!

Thanks mr!

On Mar 5, 2013, at 5:26 PM, Andy Burke notifications@github.com wrote:

Excited for 3.6, this is a good fix, thanks!


Reply to this email directly or view it on GitHub.

donnut commented Mar 6, 2013

@aheckmann Great, thank you!

Thanks!!! Great Job!!!

That's awesome! I believe it can help with my case. How should I do in this scenario:

var UserSchema, WineRating, WineSchema, mongoose;

UserSchema = new mongoose.Schema({
  wine_ratings: {
    type: [
      {
        type: mongoose.Schema.Types.ObjectId,
        ref: 'WineRating'
      }
    ]
  }
});
mongoose.model("User", UserSchema);

WineRating = new mongoose.Schema({
  wine: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Wine'
  }
});
mongoose.model("WineRating", WineRating, 'wine_ratings');


WineSchema = new mongoose.Schema({
  name: String
});
mongoose.model("Wine", WineSchema);

mongoose.model("User").findById(user._id).populate('wine_ratings.wine').exec(function(err, user) {});
/*
gets exception: 
TypeError: Cannot call method 'path' of undefined
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1830:28)
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1849:22)
    at Function._getSchema (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1856:5)
    at populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1594:22)
    at Function.Model.populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1573:5)
    at Query.findOne (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/query.js:1633:11)
    at exports.tick (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/utils.js:393:16)
*/

Is there something wrong? I am on "3.6.0rc0"

I have the same issue: #1377

yangsu commented Apr 4, 2013

@flockonus @vovan22 I ran to the same issue. This is how I resolved it.

I was trying to get post.populate("comments comments._creator") working in our project, and had a little success with the following tweaks.

It doesn't seem to work for deeper queries, and the first change breaks some of the existing tests, but I hope it may be of interest for anyone working on this.

https://gist.github.com/joeytwiddle/6129653

Since my patch kinda sucks, I have tried to contribute by writing a test case instead! #1603

joeytwiddle, i am having the exact same problem and I hope you get it fixed - I am trying to recursively populate but I get the can't find path error too. I don't know if this is by design ...

Yes the infamous 601. It is by design, at least for the moment. The latest Mongoose release does support deep population, but only within one schema.

In our project we need to do deep population across different models quite often, so we have written a helper function:

https://gist.github.com/joeytwiddle/6129676

This allows you to populate descendants of one document to any depth you like! It still requires one extra callback after you have fetched the document, but only one. For example:

deepPopulate(blogPost, "comments comments._creator comments._creator.blogposts", {sort:{title:-1}}, callback);

I hope the use of doc.constructor to get the model will not prove problematic in the future!

(Thanks to Sunride and the German government!)

merhawie commented Aug 1, 2013

@joeytwiddle - thank you so much! It worked like a charm. Now I will try to figure out a way to have it do n-deep populate.

xizhao commented Sep 15, 2013

Any updates on this?

@aheckmann I understand your theory on this, but I do disagree that it is an anti-pattern.

Sometimes you do need to perform expensive operations, its inevitable in complex systems and you do need to collect disparate documents together. For example, you may want to do one expensive operation when an document VERY VERY rarely changes and cache the results for performance.

Why do you believe one level of population is 'ok', but more than this is a sign of an anti-pattern?

@wprl wprl referenced this issue in wprl/baucis Jun 2, 2014
Closed

Nested populate does not work #193

Contributor

I know this is an old thread, but I've just created a plugin that make it very easy to populate models at any level of depth. I'm posting here in case anyone's interested: https://github.com/buunguyen/mongoose-deep-populate.

The usage is very straightforward, for example:

post.deepPopulate('votes.user, comments.user.followers, ...', cb);
Post.deepPopulate(posts, 'votes.user, comments.user.followers', cb);

Please check out the plugin repo for more information.

Contributor

@buunguyen nice job!

joaom182 commented Jan 4, 2015

Not working for me.

models/missionParticipation.js

var deepPopulate = require('mongoose-deep-populate');
var mongoose = require('mongoose');
var Types = mongoose.Schema.Types;

var missionParticipationSchema = new mongoose.Schema({
        user: {
            type: String,
            default: ''
        },
        mission: {
            type: Types.ObjectId,
            ref: 'Mission'
        },
        images: [{
            type: Types.ObjectId,
            ref: 'Image'
        }]
    }, {
        toJSON: {
            getters: true,
            virtuals: true
        },
        toObject: {
            getters: true,
            virtuals: true
        }
    });

    missionParticipationSchema.plugin(deepPopulate, {
        whitelist: [
            'images',
            'mission',
            'mission.images.poster',
            'mission.images.banner'
        ]
    });

var MissionParticipation = mongoose.model('MissionParticipation', missionParticipationSchema);

module.exports = MissionParticipation;

services/missionParticipationService.js

MissionParticipation.find({user: userID}).deepPopulate('mission.images.poster mission.images.banner').exec(function (err, missionParticipationsDocs) {
    // do the magic.
});

And i get this error on console

TypeError: Object #<Query> has no method 'deepPopulate'

@joaom182
I did not yet use deepPopulate, but recording to the docs on http://npm.taobao.org/package/mongoose-deep-populate I would assume that the correct call should be:

MissionParticipation.find({user: userID}, function (err, participations) {
  MissionParticipation.deepPopulate(participations, 'mission.images.poster mission.images.banner', function(err) {
    if (err) {
      //handle it
      return void 0;
    }
    //do your magic stuff. with participations, which are populated in place in the examples 
  })
})

Regards

joaom182 commented Jan 4, 2015

I found another way, but I'm concerned about the performance, I try to make a comparison.

The other way uses the async module

MissionParticipation.find({
   user: userID
}).populate('mission').exec(function (err, missionParticipationsDocs) {
   if (err)
      return; // handle error

   async.forEach(missionParticipationsDocs, function (mp, callback) {
      mp.mission.populate('images.poster', 'images.banner', 'prize', function (err, result) {
         callback();
      });
   }, function (err) {
      // forEach async completed
      if(err)
         return; // handle error
      resolve(missionParticipationsDocs);
   });
});
Contributor

@joaom182 you're a bit too fast :). Although I added code to bring deepPopulate to Query, I delayed pushing a new version on NPM so that I could test a bit more.

I've just pushed the new version (0.0.7). So this syntax you used should work after you update the dependency:

MissionParticipation
  .find({user: userID})
  .deepPopulate('mission.images.poster mission.images.banner')
  .exec(cb);
joaom182 commented Jan 4, 2015

@buunguyen Awesome!

Collaborator

Can y'all open future issues in the mongoose-deep-populate repo please? Makes everyone's life a little easier :)

@buunguyen Awesome! The best thing that happened to my project and You made my day! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment