Updating large embedded array

[gilles-yvetot]

With the latest version of mongoose (currently 5.0.4), document.save() takes way more time and sometimes times out when updating a large array of embedded documents. It used to work perfectly with the version 4 but not anymore. I know it is not recommended to have arrays that big, but you know how it is when they keep changing the specs

The workaround I found is to perform another query instead of updating the document and then calling .save()

I used to do that and it worked perfectly

Model.findOne({}).then(doc => {
  for (let i = 0; i < 1000; i++) {
    doc.push({
      example: true,
      count: i,
    })
  }
  return doc.save()
})

Model.findOne({}).then(doc => {
  const toInsert =[]
  for (let i = 0; i < 1000; i++) {
    toInsert.push({
      example: true,
      count: i,
    })
  }
  return Model.findOneAndUpdate(
    {_id: doc._id},
    {
      $addToSet:{
        $each: toInsert
      }
    },
    {new:true}
  )
})

My questions are:

• Is it normal to have delay now when performing this kind of operation?
• Does anyone face the same issue?
• Is there any options to pass to save to keep the "old" performances ?

[vkarpov15]

Did you use the usePushEach option in when you used 4.x? One big change in 5.x was that we started using $push: { $each: [] } instead of $pushAll, so that is one possibility.

Also, do you have a way to identify whether this extra time is in mongodb or in nodejs?

[gilles-yvetot]

Did you use the usePushEach option in when you used 4.x?
No
do you have a way to identify whether this extra time is in mongodb or in nodejs
I can try to reproduce your model.save method and see what's happening, what do you think ?

[vkarpov15]

"I can try to reproduce your model.save method and see what's happening, what do you think ?" can you clarify that statement? I don't understand

[vkarpov15]

I can confirm the below script consistently shows a 5-10% degradation in mongoose 5.0.6 vs 4.13.11:

const mongoose4 = require('mongoose');
const mongoose5 = require('mongoose5');

run().catch(error => console.error(error.stack));

async function run() {
  mongoose4.Promise = global.Promise;
  await mongoose4.connect('mongodb://localhost:27017/m4', { useMongoClient: true });
  await mongoose5.connect('mongodb://localhost:27017/m5');

  const Model4 = mongoose4.model('Test', new mongoose4.Schema({
    children: [{ example: Boolean, count: Number }]
  })); 
  const Model5 = mongoose5.model('Test', new mongoose5.Schema({
    children: [{ example: Boolean, count: Number }]
  }));

  const reps = 100;

  let m5total = 0;
  for (let i = 0; i < reps; ++i) {
    m5total += await profile(Model5);
  }

  console.log('m5', m5total, 'average', m5total / reps);

  let m4total = 0;
  for (let i = 0; i < reps; ++i) {
    m4total += await profile(Model4);
  }

  console.log('m4', m4total, 'average', m4total / reps);

  console.log('average diff', (m4total - m5total) / reps);
}

async function profile(Model) {
  let doc = await Model.create({});
  doc = await Model.findById(doc._id);

  const start = Date.now();
  for (let i = 0; i < 1000; ++i) {
    doc.children.push({ example: true, count: i });
  }
  await doc.save();

  return Date.now() - start;
}

No further insights right now, will investigate more.

[vkarpov15]

With some improvements we've managed to knock the performance degradation down to 0-5%, assuming that you don't have any validate hooks on your child subdocs the performance should be much better in 5.0.7.