Mongoose converting undefined to null when sending the query to MongoDB

[msugas19]

When running a complex aggregate query, if there is a conditional check for a field to be undefined, mongoose debugger will print out the query containing undefined as the value to check against. When pointing mongoose to a mongodb instance in verbose mode, it can be seen that the value is converted to null. Because of this, conditional checks for non-existent fields do not work in Mongoose.

Example Input: MongooseModel.aggregate([{$match:{_id:0123}},{$group: {_id:"$_id", validCount:{$sum:{$cond: [{$ne: ["$myField", undefined]}, 1, 0]}}])

Example Query MongoDB Receives: MongooseModel.aggregate([{$match:{_id:0123}},{$group: {_id:"$_id", validCount:{$sum:{$cond: [{$ne: ["$myField", null]}, 1, 0]}}])

In the event that the field myField does not exist in the document, the check will fail as mongoDB knows that the field is undefined and not null, however mongoose will pass in null, resulting in incorrect results.

MongoDB Version - 3.4
Node.js Version - 3.6.10
Mongoose Version -Able to replicate with all versions. Versions tried - 4.5.5, 4.7.1, 4.12.5

[vkarpov15]

Unfortunately the mongodb node driver does not support the undefined BSON type, see Automattic/monk#146 , http://devblog.me/wtf-mongo, etc. Its unfortunate but the BSON type undefined has been deprecated for over 7 years so the driver tries hard to avoid allowing it.

[msugas19]

Hi @vkarpov15. Even though the mongodb node driver is not supposed to support this type, it still works. I am able to run through the mongodb drivers and I am receiving the correct results. Why would mongoose attempt to manipulate the structure of the query? Shouldn't it let the user's query fail if that is what it is designed to do?

[vkarpov15]

That's strange, I'll re-open this issue and investigate.

[sobafuchs]

I'm not able to reproduce this:

const mongoose = require('mongoose');
const co = require('co');
mongoose.Promise = global.Promise;
const GITHUB_ISSUE = `gh-5776`;

mongoose.set('debug', true);
exec()
  .then(() => {
    console.log('successfully ran program');
    process.exit(0);
  })
  .catch(error => {
    console.error(`Error: ${error}\n${error.stack}`);
  });


function exec() {
  return co(function* () {
    yield mongoose.connect(`mongodb://localhost:27017/${GITHUB_ISSUE}`, { useMongoClient: true });
    const schema = new mongoose.Schema({
      name: String
    });

    const Model = mongoose.model('Model', schema);

    yield Model.aggregate([
      { $match: { _id: 0123 } },
      {
        $group: {
          _id: "$_id",
          validCount: {
            $sum: {
              $cond: [
                { $ne: ["$myField", undefined] },
                1,
                0
              ]
            }
          }
        }
      }])
  });
}

The mongoose debug output:

Mongoose: models.aggregate([ { '$match': { _id: 83 } }, { '$group': { _id: '$_id', validCount: { '$sum': { '$cond': [ { '$ne': [ '$myField', undefined ] }, 1, 0 ] } } } } ], {})

I am testing this with node 6.11.5. Are you sure your version is 3.6.10? That's not a valid node version - they went directly from 0.x to 4.0: https://nodejs.org/en/download/releases/

[msugas19]

@varunjayaraman If you notice in the original question, I mentioned how you can only see in the MongoDB logs that the incorrect information is being passed. The debugger shows the correct query, but Mongoose changes the query before sending it to MongoDB. You can only see the problem when you start a MongoDB instance in verbose mode, when it will print off every query that it receives.

To put MongoDB into verbose mode, you need to type open the Mongo Shell and run the command db.setProfilingLevel(0,-1). You should see the issue now using your test script

[sobafuchs]

ah i see, yup i was able to reproduce by setting the profiling level correctly.

Looking at the mongo logs, i see:

2017-11-05T23:19:48.886-0500 I COMMAND  [conn3] command gh-5776.models command: aggregate { aggregate: "models", pipeline: [ { $match: { _id: 83 } }, { $group: { _id: "$_id", validCount: { $sum: { $cond: [ { $ne: [ "$myField", null ] }, 1, 0 ] } } } } ], cursor: { batchSize: 1000 } } planSummary: EOF keysExamined:0 docsExamined:0 cursorExhausted:1 numYields:0 nreturned:0 reslen:102 locks:{ Global: { acquireCount: { r: 6 } }, Database: { acquireCount: { r: 3 } }, Collection: { acquireCount: { r: 2 } } }

[vkarpov15]

Mongoose doesn't transform the data from the debug output before sending it to the server, so I don't think this is a mongoose issue. I get the same output by running the below script using the mongodb driver directly:

const mongodb = require('mongodb');
const co = require('co');
const GITHUB_ISSUE = `gh5776`;

exec()
  .then(() => {
    console.log('successfully ran program');
    process.exit(0);
  })
  .catch(error => {
    console.error(`Error: ${error}\n${error.stack}`);
  });


function exec() {
  return co(function* () {
    const db = yield mongodb.MongoClient.connect(`mongodb://localhost:27017/${GITHUB_ISSUE}`);

    yield db.collection('Model').aggregate([
      { $match: { _id: 0123 } },
      {
        $group: {
          _id: "$_id",
          validCount: {
            $sum: {
              $cond: [
                { $ne: ["$myField", undefined] },
                1,
                0
              ]
            }
          }
        }
      }]).toArray();
  });
}

Can you clarify your comment "Even though the mongodb node driver is not supposed to support this type, it still works. I am able to run through the mongodb drivers and I am receiving the correct results"? Above script uses the driver and it still transforms to null. Perhaps you're mixing up the mongodb node driver and the mongodb shell?

[PhantomRay]

I bumped into this nasty bug today exactly the same way. I can now confirm this bug. @vkarpov15, the debug output is correct. But after the output at some point, e.g. when stringify JSON object and send to mongodb, the following code
{ $ne: ["$myField", undefined] }
changed to
{ $ne: ["$myField", null] }.

It appears that undefined in array will be converted to null.
A simple test would be
JSON.stringify([1, undefined]) gives you "[1,null]"

This issue has to be reopened.

btw, I am using the latest mongoose 5.4.12.

[PhantomRay]

Apart from this bug, an alternative query inside $cond should be changing
{ $ne: ["$myField", undefined] }
to
{ $ifNull: ["$myField", true] }

[vkarpov15]

@PhantomRay that's because the underlying mongodb driver doesn't support the undefined bson type. Not much Mongoose can do about it.