Error on MongoDB insertion

[RomainJamet]

Node version: 10.15.1
Sails version (sails): 1.2.2
ORM hook version (sails-hook-orm): 2.1.1
Sockets hook version (sails-hook-sockets): 1.5.5
Organics hook version (sails-hook-organics): N/A
Grunt hook version (sails-hook-grunt): 3.1.0
Uploads hook version (sails-hook-uploads): N/A
DB adapter & version (e.g. sails-mysql@5.55.5): sails-mongo@1.0.1
Skipper adapter & version (e.g. skipper-s3@5.55.5): N/A

---

Hello,

I currently have a problem with the sails-mongo adapter.

We have a SailsJS that has been using Postgresql for a while, it was created in 0.10 and updated recently to 1.2.2. We halso have a MongoDB database that is primarily used by another application, but we need to periodically collect data from this MongoDB base.

Previously, we used the mongoose module to do any operation on this base, but we wish to use the built-in solution with Waterline, and add our MongoDB base as a datastore.

I added it to the list of datastores, declared my models, and I did override the default id with MongoDB's _id, as described in SailsJS's documentation.

When I try to find data: everything seems fine, all my data are correctly retrieved. But when I tried to create objects in the MongoDB datastore, it failed.

First I tried to insert an array of objects, which prompted the following error:

{ UsageError: Invalid initial data for new records.
Details:
  Could not use one of the provided new records: Could not use specified `id`.  Invalid primary key value.  Instead of a string (the expected pk type), the provided value is: null
 [?] See https://sailsjs.com/support for help.
    at Object.insertDummyData [as admin/insertdummydata] (C:\Users\rjame\botmind\platform\api\controllers\AdminController.js:242:53)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  cause:
   { UsageError: Invalid initial data for new records.
   Details:
     Could not use one of the provided new records: Could not use specified `id`.  Invalid primary key value.  Instead of a string (the expected pk type), the provided value is: null
    [?] See https://sailsjs.com/support for help.
       at Object.insertDummyData [as admin/insertdummydata] (C:\Users\rjame\botmind\platform\api\controllers\AdminController.js:242:53)
       at process._tickCallback (internal/process/next_tick.js:68:7)
     name: 'UsageError',
     code: 'E_INVALID_NEW_RECORDS',
     details:
      'Could not use one of the provided new records: Could not use specified `id`.  Invalid primary key value.  Instead of a string (the expected pk type), the provided value is: null' },
  isOperational: true,
  code: 'E_INVALID_NEW_RECORDS',
  details:
   'Could not use one of the provided new records: Could not use specified `id`.  Invalid primary key value.  Instead of a string (the expected pk type), the provided value is: null' }

Apparently, it expected that I put something in the id field, which is impossible, as I wish to use MongoDB's default id system (since all existing data uses it).

So I tried to insert elements once at a time, and I got an even weirder error:
I tried to insert 5 elements from an array, once at a time (with await create(element).toPromise()), the first element was inserted properly:

debug: { id: '5d07b353bd406935c854dce6',
  status: 1,
  startDate: 2019-06-17T15:35:47.980Z,
  type: 50,
  senderId: 'dummy-san',
  conversationId: 'dummy_conv_4',
  projectId: '5d03951afb9c8127b0c27fc4',
  initialAssigneeId: 'No one',
  initialMessageId: 'None',
  errorCount: 0,
  intent: 'dummy-chat',
  intentHistory: [],
  currentSchema: {},
  nextStep: '',
  context: {} }

But then for the following one:

error: { AdapterError: Would violate uniqueness constraint-- a record already exists with conflicting value(s).
    at Object.insertDummyData [as admin/insertdummydata] (C:\Users\rjame\botmind\platform\api\controllers\AdminController.js:238:53)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  cause:
   { AdapterError: Would violate uniqueness constraint-- a record already exists with conflicting value(s).
       at Object.insertDummyData [as admin/insertdummydata] (C:\Users\rjame\botmind\platform\api\controllers\AdminController.js:238:53)
       at process._tickCallback (internal/process/next_tick.js:68:7)
     code: 'E_UNIQUE',
     attrNames: [],
     toJSON: [Function: toJSON],
     name: 'AdapterError',
     adapterMethodName: 'create',
     modelIdentity: 'mongothread',
     raw:
      { MongoError: E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId('5d07b353bd406935c854dce6') }
          at Function.MongoError.create (C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb-core\lib\error.js:31:11)
          at toError (C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb\lib\utils.js:139:22)
          at C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb\lib\collection.js:739:67
          at C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb-core\lib\connection\pool.js:461:18
          at process._tickCallback (internal/process/next_tick.js:61:11)
        name: 'MongoError',
        message:
         'E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId(\'5d07b353bd406935c854dce6\') }',
        driver: true,
        index: 0,
        code: 11000,
        errmsg:
         'E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId(\'5d07b353bd406935c854dce6\') }',
        footprint: [Object] } },
  isOperational: true,
  code: 'E_UNIQUE',
  attrNames: [],
  toJSON: [Function: toJSON],
  adapterMethodName: 'create',
  modelIdentity: 'mongothread',
  raw:
   { MongoError: E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId('5d07b353bd406935c854dce6') }
       at Function.MongoError.create (C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb-core\lib\error.js:31:11)
       at toError (C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb\lib\utils.js:139:22)
       at C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb\lib\collection.js:739:67
       at C:\Users\rjame\botmind\platform\node_modules\sails-mongo\node_modules\mongodb-core\lib\connection\pool.js:461:18
       at process._tickCallback (internal/process/next_tick.js:61:11)
     name: 'MongoError',
     message:
      'E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId(\'5d07b353bd406935c854dce6\') }',
     driver: true,
     index: 0,
     code: 11000,
     errmsg:
      'E11000 duplicate key error index: botmind-bridge-dev.thread.$_id_ dup key: { : ObjectId(\'5d07b353bd406935c854dce6\') }',
     footprint: { identity: 'notUnique', keys: [] } } }

Apparently it tried to use the same id. And the third item was inserted properly with another id, but not the fourth which used the third's id and so on....

So createEach does not generates ids, and successive create calls fail once of two times.

Here's my model file:

module.exports = {
  datastore: 'mongodb', // Override of config/models.js
  tableName: 'thread', // Collection's name (cf mongoose instanciation above)
  attributes: {
    // Override of config/models.js
    id: { type: 'string', columnName: '_id' },
    createdAt: false,
    updatedAt: false,
    // Redefinition of schema's elements
    status: { type: 'number' },
    startDate: { type: 'ref', columnType: 'string' },
    type: { type: 'number' },
    senderId: { type: 'string'},
    conversationId: { type: 'string', allowNull: true },
    projectId: { type: 'string' },
    initialAssigneeId: { type: 'string' },
    initialMessageId: { type: 'string' },
    errorCount: { type: 'number' },
    intent: { type: 'string' },
    intentHistory: { type: 'ref', columnType: 'Array' },
    currentSchema: { type: 'json' },
    nextStep: { type: 'string', allowNull: true },
    context: { type: 'json' }
  },
  findAll: function(){
    return MongoThread.find({}).toPromise();
  }
};

My successive create attempts looks like:

for(let i = 0; i < fakeThreads.length; i++){
  console.log('===============================================================================');
  console.log(i);
  var [err, insertedThread] = await to(MongoThread.create(fakeThreads[i]).toPromise());
  if(err)
    sails.log.error(err);
  else
    sails.log(insertedThread);
}

With fakethreads being an array of objects containing all of my model's attributes except ids.

[sailsbot]

@RomainJamet Thanks for posting! We'll take a look as soon as possible.

In the mean time, there are a few ways you can help speed things along:

• look for a workaround. (Even if it's just temporary, sharing your solution can save someone else a lot of time and effort.)
• tell us why this issue is important to you and your team. What are you trying to accomplish? (Submissions with a little bit of human context tend to be easier to understand and faster to resolve.)
• make sure you've provided clear instructions on how to reproduce the bug from a clean install.
• double-check that you've provided all of the requested version and dependency information. (Some of this info might seem irrelevant at first, like which database adapter you're using, but we ask that you include it anyway. Oftentimes an issue is caused by a confluence of unexpected factors, and it can save everybody a ton of time to know all the details up front.)
• read the code of conduct.
• if appropriate, ask your business to sponsor your issue. (Open source is our passion, and our core maintainers volunteer many of their nights and weekends working on Sails. But you only get so many nights and weekends in life, and stuff gets done a lot faster when you can work on it during normal daylight hours.)
• let us know if you are using a 3rd party plugin; whether that's a database adapter, a non-standard view engine, or any other dependency maintained by someone other than our core team. (Besides the name of the 3rd party package, it helps to include the exact version you're using. If you're unsure, check out this list of all the core packages we maintain.)

---

Please remember: never post in a public forum if you believe you've found a genuine security vulnerability. Instead, disclose it responsibly.

For help with questions about Sails, click here.

[RomainJamet]

Oh, in case you're wondering, in my attempt example code, the function to() is a Promise wrapper, that looks like:

let to = (promise)=>{
  return promise.then((data)=>[null, data]).catch((error)=>[error, null]);
}

To avoid having to put a try-catch block at each await I'm using.

[RomainJamet]

Alright, I've found a workaround derived from the previously attempted solution. Instead of directly trying to successively call await create() from my loop in my controller, I wrote a method inside the model's file:

insertDummy: async function(projectId, projectMessaging){
    return new Promise(async function(resolve, reject){
      let thread = {
        status: 1,
        startDate: new Date().toISOString,
        /*...*/
      };
      var [err, res] = await to(MongoThread.create(thread).toPromise());
      if(err)
        return reject(err);
      return resolve(res);
    });
  }

Then I call this method from the controller in a loop:

for(let i = 0; i < threads; i++){
  let [err, thread] = await to(MongoThread.insertDummy(project.idmongo, project.messaging));
  if(err)
    sails.log.error(err);
  else
    sails.log(thread);
}

Basically, I'm awaiting for a promise that's awaiting for create's promise. Since the volume of data is not too big, this will do for now, but I'd like to know if the previously observed behaviors (reusing the same ids despide the await create() calls being successive, and await createEach() not generating ids automatically) are known bugs and if a cleaner solution exists...

[johnabrams7]

@RomainJamet Thanks for the great detail on this one and providing a workaround. I'll bring this up with the team to determine the best reason why it's reusing the same ids despite the await create() calls being successive (await createEach() not automatically generating ids.) If you have a repo, that will also help. Appreciate the efforts 👍

[laurenamy]

I'm experiencing this issue now as well, has this been resolved?

[eashaw]

Hi @laurenamy, can you provide us with more information about your version info? Are you using multiple adapters or just sails-mongo?