Cursor operator in Aggregate function not working..

[mithralaya]

Hi,

I have the latest mongoose 3.8.16 with mongodb 2.6.4

I am trying to do cursor operation on an aggregate pipeline.

My function looks like,

var where = {  "storeId": storeId };
MongoBasket.aggregate()
                            .match(where)
                            .group({
                                "basketProducts": {
                                    "$push": "$basketProduct"
                                },
                                "basketIds": {
                                    "$push": "$basketId"
                                },
                                "storeId": {
                                    "$addToSet": "$storeId"
                                },
                                "consumerIds": {
                                    "$push": "$consumerId"
                                },
                                "basketTransactionCardTotal": {
                                    "$push": "$basketTransactionCard"
                                },
                                "basketTransactionCashTotal": {
                                    "$push": "$basketTransactionCash"
                                },
                                "totalTip": {
                                    "$push": "$tip"
                                },
                                "_id": {
                                    "year": {
                                        "$year": "$completed"
                                    },
                                    "month": {
                                        "$month": "$completed"
                                    },
                                    "day": {
                                        "$dayOfMonth": "$completed"
                                    },
                                    "hour": {
                                        "$hour": "$completed"
                                    }
                                }
                            })
                            .sort({_id: 1})
                            .cursor({
                                batchSize: 200000
                            })
                            .exec(function(err, result) {
                                 console.log(err, result);
                            }, req.messageInstance);

I printed the raw mongodb query that is built by mongoose (using mongoose.set('debug', true)) which looks like

baskets.aggregate([
    {
        '$match': {
            storeId: 11
        }
    },
    {
        '$group': {
            _id: {
                hour: {
                    '$hour': '$completed'
                },
                day: {
                    '$dayOfMonth': '$completed'
                },
                month: {
                    '$month': '$completed'
                },
                year: {
                    '$year': '$completed'
                }
            },
            totalTip: {
                '$push': '$tip'
            },
            basketTransactionCashTotal: {
                '$push': '$basketTransactionCash'
            },
            basketTransactionCardTotal: {
                '$push': '$basketTransactionCard'
            },
            consumerIds: {
                '$push': '$consumerId'
            },
            storeId: {
                '$addToSet': '$storeId'
            },
            basketIds: {
                '$push': '$basketId'
            },
            basketProducts: {
                '$push': '$basketProduct'
            }
        }
    },
    {
        '$sort': {
            _id: 1
        }
    }
]){
    cursor: {
        batchSize: 200000
    }
}

Please note that cursor operator is outside the aggregate () enclosure

]){
    cursor: {
        batchSize: 200000
    }
}

Shouldn't it be like

],
{
    cursor: {
        batchSize: 200000
    }
})

My hand written raw mongodb query for mongo shell works really well and here is how it looks

db.baskets.aggregate([
    {
        "$match": {
            storeId: 11
        }

    },
    {
        "$group": {
            "_id": {
                "year": {
                    "$year": "$completed"
                },
                "month": {
                    "$month": "$completed"
                },
                "day": {
                    "$dayOfMonth": "$completed"
                },
                "hour": {
                    "$hour": "$completed"
                }
            },
            "totalTip": {
                "$push": "$tip"
            },
            "basketTransactionCashTotal": {
                "$push": "$basketTransactionCash"
            },
            "basketTransactionCardTotal": {
                "$push": "$basketTransactionCard"
            },
            "consumerIds": {
                "$push": "$consumerId"
            },
            "storeId": {
                "$addToSet": "$storeId"bas
            },
            "basketIds": {
                "$push": "$basketId"
            },
            "basketProducts": {
                "$push": "$basketProduct"
            }
        }
    },
    {
        "$sort": {
            _id: 1
        }
    }
],
{
    cursor: {
        batchSize: 2000000
    }
})

Is it a bug or am I missing something?

If someone could shred some light on it would be much appreciated.

Many thank,

Karthik

[webjay]

I have the same problem with the same versions.

This works:

return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).exec());

This doesn't work:

return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).cursor({ batchSize: 1000 }).exec());

[mithralaya]

Thanks for adding it to your milestone.

Many thanks,
Karthik

[mattbryson]

Hi, Any progress on this?

[vkarpov15]

Some progress, but not much unfortunately. We've figured out what's wrong, just haven't had the time to fix it yet.

[chrisjhoughton]

+1. Any progress on this?

[vkarpov15]

So this issue is closed, what sort of problem are you having?

[chrisjhoughton]

This works:

Model
.aggregate(pipeline)
.exec(function () {
  console.log(arguments);
})

And this doesn't: (never calls back)

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec(function () {
  console.log(arguments);
})

Also, the following doesn't return a promise:

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

Am I missing something?

[vkarpov15]

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

returns a cursor, as specified in 4.0 release notes. Admittedly this is a not-well-designed API, it's something that was thrown in at the last minute as a stopgap. The right way to do it would probably be something like this:

var stream = Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec().stream();

stream.on('data', function(doc) {
  // ...
});

See mongodb driver cursor docs

[chrisjhoughton]

Ok cool, thanks for that! For the callback approach, the docs currently say this:

This is incorrect, right?

[vkarpov15]

Yep those docs are incorrect, thanks for pointing that out

[timjnh]

Anyone looking at this may also be interested in #3160 which outlines another bug about getting an undefined response from exec when a cursor is used.

[radiumrasheed]

@vkarpov15 its almost 2018 and the docs are still misleading. someone help 🤔

[vkarpov15]

@Ra1da35ma well that's embarrassing, looks like we closed out #2955 by mistake. Thanks for pointing that out, re-opened and will be fixed 👍

[jaffarc]

I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6

[gucompi]

This works:

Model
.aggregate(pipeline)
.exec(function () {
  console.log(arguments);
})

And this doesn't: (never calls back)

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec(function () {
  console.log(arguments);
})

Also, the following doesn't return a promise:

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

Am I missing something?

This Answer + This One:

I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6

Solved my problem. Thanks!