match within .populate is working incorrectly.

[tar-aldev]

Do you want to request a feature or report a bug?
bug
What is the current behavior?
Given the following model (using typegoose but it can be easily reproduces with plain mongoose too)

export class User {
  ...
  @prop({ type: Access })
  access?: Access[];
}

Now in access array I have the following:

[
       {
         ...
          role: 'admin',
          organization: 5f507934b33f9eebb1f036c7,
          deletedAt: 2021-04-07T13:04:19.952Z
        },
        {
          ...
          role: 'admin',
          organization: 5f507934b33f9eebb1f036c7
        }
]

Then I have another model with virtual populate like this

export class Organization {
    @prop({
      ref: 'User',
      localField: '_id',
      foreignField: 'access.organization'
    })
    administrators?: Ref<User>[];
}

export default getModelForClass(Organization);

When I do

organizationModel
      .findByIdAndUpdate(orgId, updated, {
        new: true
      })
      .populate({
        path: 'administrators',
        match: {
          'access.role': 'admin'
          'access.deletedAt': { $exists: false }
        },
        select: 'firstName lastName email access.role access.deletedAt access.isMainAdmin role'
      });

I expect it to find the above user because it has one of access items where deletedAt does not exist;

Instead it filters out that user which seems very unintuitive and unexpected.
If the current behavior is a bug, please provide the steps to reproduce.

"compilerOptions": {
"outDir": "../../dist/out-tsc",
"types": ["node", "express"],
"target": "es5",
"sourceMap": true,
"alwaysStrict": true,
"noImplicitAny": false,
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"allowSyntheticDefaultImports": true,
"noResolve": false,
"moduleResolution": "node",
"typeRoots": ["./node_modules/@types"]
},
"exclude": ["/*.spec.ts", "/.d.ts", "**/tests/utils/.ts"],
"include": ["**/*.ts"]

What is the expected behavior?
match should keep user with access like this

[
       {
         ...
          role: 'admin',
          organization: 5f507934b33f9eebb1f036c7,
          deletedAt: 2021-04-07T13:04:19.952Z
        },
        {
          ...
          role: 'admin',
          organization: 5f507934b33f9eebb1f036c7
        }
]

because one of items matches with organization and. does not have deletedAt field
What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.
node 14
mongoose 5.10
mongodb 3.6.5

[DmitryIvanovIAMM]

Yes. Have the same issue.

[IslandRhythms]

what are you updating exactly when you run findByIdAndUpdate()? specifically, what is updated?

[vkarpov15]

We've been looking into this and found some more info.

Your query, as written, is not expected to work because 'access.deletedAt': { $exists: false } looks like only finds docs where access.deletedAt doesn't exist for every entry in access. That's just how MongoDB works: https://stackoverflow.com/questions/10907843/mongo-ne-query-with-an-array-not-working-as-expected/36484191 .

The more correct populate uses $elemMatch as shown below.

      await Organization.findById(org).populate({
        path: 'administrators',
        match: {
          access: {
            $elemMatch: { role: 'admin', deletedAt: { $exists: false } }
          }
        },
      });

However, this does come with a nuance because this will find users where access.organizationId matches the organization's id for at least one subdocument in access, and at least one subdocument in access has role: 'admin' and deletedAt: { $exists: false }. We need to investigate more to see if there's a workaround.

Also, it turns out there's an issue where the below script shows that res.administrators contains 3 documents below, including 'user3' twice. We'll fix that and post here when we know more.

'use strict';

const mongoose = require('mongoose');

mongoose.set('debug', true);
mongoose.set('useFindAndModify', false);

const { Schema } = mongoose;

run().catch(err => console.log(err));

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test', {
    useNewUrlParser: true,
    useUnifiedTopology: true
  });
  await mongoose.connection.dropDatabase();

  const User = mongoose.model('User', mongoose.Schema({
    name: String,
    access: [{ role: String, deletedAt: Date, organization: 'ObjectId' }]
  }));

  const organizationSchema = mongoose.Schema({ name: String });
  organizationSchema.virtual('administrators', {
    ref: 'User',
    localField: '_id',
    foreignField: 'access.organization'
  });
  const Organization = mongoose.model('Organization', organizationSchema);

  const org = await Organization.create({ name: 'test org' });
  await User.create([
    { name: 'user1', access: [{ role: 'admin', organization: org._id }] },
    { name: 'user2', access: [{ role: 'admin', organization: null }] },
    { name: 'user3', access: [{ role: 'admin', organization: org._id, deletedAt: new Date() }, { role: 'admin', organization: org._id }] }
  ]);

  const res = await Organization.findById(org).populate({
        path: 'administrators',
        match: {
          access: { $elemMatch: { role: 'admin', deletedAt: { $exists: false } } }
        },
      });
  console.log(res.administrators);
}