Hydrating browser-side Documents with populated SubDocuments using isomorphic schemas

[jubr]

In reaction to #3774, here's what I am trying to do. Currently I'm using a combination of riot.js, feathers.js and Primus to transport data from Node/Mongoose to the browser and back. But the case I am describing is generic. Client-side I am trying to fully instantiate/hydrate documents so I can use validation, embedded document and subdocument modifications, using isomorphic techniques as much as possible.

var categorySchema = new Schema({
    name: {type: String, maxlength: 20}
});

var thingTemplateSchema = new Schema({
    name: {type: String, maxlength: 50},
    unit: {type: String, enum: ['x', 'm2', 'm', 'kg']},
    category: {type: ObjectId, ref: 'categorySchema'}
});

var thingEstimatedSchema = new Schema({
    thingTemplate: {type: ObjectId, ref: 'thingTemplateSchema'},
    quantity: {type: Number, min: 0},
    cost: {type: Number, min: 0}
});

var thingCostsSchema = new Schema({
    hourlyRate: {type: Number, min: 0},
    categories: [{
        category: {type: ObjectId, ref: 'categorySchema'},
        thingsEstimated: [ thingEstimatedSchema ]
    }]
});

This is only a part of the data structure and simplified. Category and ThingTemplate are used in a couple of other places, so I think I need db-refs for these. Please note that I am pretty new to Mongoose & I wonder if I'm sticking enough to the 'No' in NoSQL with my approach. Creation, modification and validation of these isomorphic schemas is performed at both the client-side and the server-side.
Moving on to the server-side query. The actual query is initiated client-side, but results in something like this on the server, the only non-isomorphic part of the code:

var thingCosts = mongoose.model('thingCosts', thingCostsSchema);

thingCosts
    .findById(someId)
    .populate('categories.category categories.thingsEstimated.thingTemplate')
    .lean()
    .exec(function(error, data) {
        thingService.sendDataToClient('things', data);
    });

The data transmitted might look something like this:

{
  "_id": "56908565acaef75c0f046504",
  "hourlyRate": 85,
  "categories": [
    {
      "category": /* populated subdoc */ {
        "name": "Category number one",
        "_id": "568f15b6c348ed0000e6135f"
      },
      "_id": "5690a139803b810000c4fc65",
      "thingsEstimated": [
        {
          "thingTemplate": /* populated subdoc */ {
            "category": "568f15b6c348ed0000e6135f",
            "unit": "x",
            "name": "Cat1 Unit1",
            "_id": "5690a139803b810000c4fc63"
          },
          "quantity": 2,
          "cost": 234,
          "_id": "5690a139803b810000c4fc64"
        }
      ]
    },
    {
      "category": /* populated subdoc */ {
        "name": "The second category",
        "_id": "568f1590c348ed0000e6135c"
      },
      "_id": "5690ac5c85b80e0000cd7146",
      "thingsEstimated": [
        {
          "cost": 3,
          "quantity": 1,
          "thingTemplate": /* populated subdoc */ {
            "category": "568f1590c348ed0000e6135c",
            "unit": "x",
            "name": "Little Unit",
            "_id": "5690ac5c85b80e0000cd7144"
          },
          "_id": "5690ac5c85b80e0000cd7145"
        },
        {
          "cost": 89,
          "quantity": 5,
          "thingTemplate": /* populated subdoc */ {
            "name": "Heavy Unit",
            "unit": "kg",
            "category": "568f1590c348ed0000e6135c",
            "_id": "5698574c371b7100003eefbe"
          },
          "_id": "5698574c371b7100003eefbf"
        }
      ]
    }
  ]
}

On the client-side this is then received, instantiated and used in various ways:

thingService.on('things', function(err, data) {
    // make a proper Mongoose Document out of it, with Embedded Documents + SubDocuments
    // from populated data
    var thingCosts = mongoose.document(data, thingCostsSchema);
    // this document would then be used to render the page
    irrelevantRenderingEngine('thingCostsView', thingCosts);

    // and would want to use things like this directly:
    console.log(thingCosts.categories[0].category.name);
    thingCosts.categories[3].thingsEstimated[1].quantity = -1;
    thingCosts.categories[3].thingsEstimated[1].validate(...); // -1 should not validate ok
    thingCosts.categories[3].thingsEstimated[2].thingTemplate.name += ' (old)';
    thingCosts.categories[4].remove();

    // using a Model-like constructor, a wrapper around mongoose.document(obj, xSchema)
    var thingTpl = new ThingTemplate({
        name: 'Some Thing',
        unit: 'x',
        category: currentCategory._id
    });
    var thingEst = new ThingEstimated({
        thingTemplate: thingTpl,
        quantity: 42,
        cost: 4242
    });
    thingCosts.categories[3].thingsEstimated.push(thingEst);
});

Saving of the modified document is left out of scope for brevity. Brief - a foreign concept, indeed 😀
Save to say (pun intended) that the modification semantics are meant to be used for efficient saving at a later time.

For instantiation with subdocuments from populated data I've managed to modify Document's function init() to check for schema.options.ref and pass in the actual subdoc schema to cast to explicitly using self.set().

Then, for later modification with .push() or .set() I was able to fill val.constructor.modelName (from Model-land) by explicitly setting it on doc.constructor after construction in a custom wrapper around mongoose.document(obj, xSchema) for each schema.

I suspect that @robschley's proposed BrowserModel in #3774 will take care of most of these challenges. Plus the possibility of plugins to elegantly handle the actual queries/saving.

Yes, I am aware of the hackiness, just wanted to see if I could, it is semi-working at the moment. Still seeing a lot of $__ and schema properties in the toObject() result, probably due to my coercion of various objects. Is there a better way to do this currently?

[vkarpov15]

I think right now the only workaround would be to explicitly get categories.category and then set it on the client side. Category.find(), transport the categories to the client side, instantiate category docs, and then on the client do thingCosts.categories = categories. The isomorphic code and populate() don't play nicely together right now, which is one of the reasons why #3774 is important. Right now the isomorphic schema is a rudimentary workaround that's great for form validation, but more complex use cases are not easy.

Great choice on riot btw, I've been using it a bit myself. How are you liking it so far?

[jubr]

Yeah, I was afraid of that, I should stop trying to get it to work this way :)

I stumbled upon Riot when I was looking for an isomorphic framework. I'm actually really pleased with it. Decided to start with https://github.com/ListnPlay/riot-isomorphic: ES6 + Babel + Browserify + Feathers + Primus + Passport + Gulp + Livereload goodness from the get-go :D
I like the way for each Tag the html + css + js logic are bundled together in a single .js file using ES6 template strings. That, combined with the Observables and a central Dispatcher make for a really powerful combination.

Then started modifying it to my needs. Feathers-mongoose-advanced plays reasonably well with feathers-client over Primus, but if you are going to do complex interlinked schemas and more than CRUD, then you hit its limits pretty fast. I'm sticking with mongoose for sure, though. Kudos!

[lucafaggianelli]

@jubr I stumbled on the same problem, I can't populate documents' paths on the client-side (browser), were you able to find a solution? I'm not even able to manually populate the paths, saying thingCosts.categories = categories

[vkarpov15]

@lucafaggianelli can you please open a new issue and follow the issue template?

[lucafaggianelli]

I did #8605