Aggregation Freamwork - cursor id <NUMBER> didn't exist on server

[Roycohen]

Hi There,

I'm using the aggregation framework and most of the features are working very good.
There is one issue if there are long queries and there is no way to set the cursor timeout or something like that.

The Error I'm getting is: "cursor id xxxxxxx didn't exist on server".
Please help me resolve it.
maybe configuration on the Mongo server themselves.
Maybe you have some idea or insights whether Mongo going to address this issue.

I'm using the following version:
mongodb
mongodb support => enabled
mongodb version => 1.2.0alpha3
mongodb stability => alpha
libmongoc version => 1.5.0-rc0
libbson version => 1.5.0-rc0
mongodb.debug => no value => no value

Thanks Guys.

[jmikola]

There is one issue if there are long queries and there is no way to set the cursor timeout or something like that.

Can you elaborate on what a "long query" is in your case? According to the MongoDB manual, idle cursors will time out after an inactivity period of 10 minutes. I believe that inactivity period is the time between getmore requests for additional batches of results.

While the find command does have a noCursorTimeout option to prevent inactive cursors from timing out, there is no such option for the aggregate command. If you'd like to make a case for this, you could open a SERVER ticket in JIRA or inquire on the mongodb-user mailing list.

A possible work-around may be to lower your batchSize on the aggregate command in an attempt to cause the driver to issue getmore requests more frequently during cursor iteration.

[Roycohen]

OK.
Thanks for the response.
I understand that they are planing to make an option to support timeout or something like this. DO you know something about it?

In addition, assuming my query performs FULL TABLE SCAN in several shards, do you think reducing the batch size may help?

[lindelius]

Depending on your query you might want to change the filter and/or the sorting in order to return the results quicker, and then instead do any additional checks in your application or script.

[jmikola]

I understand that they are planing to make an option to support timeout or something like this. DO you know something about it?

The only related issue I found in JIRA was SERVER-6036, which looks relevant based on the linked Python driver issue. The feature is still not scheduled for implementation, and I don't know anything more than what is in the ticket itself.

In addition, assuming my query performs FULL TABLE SCAN in several shards, do you think reducing the batch size may help?

Batch size does not influence whether a full table scan is performed. That is more of a question of the index (if any) that is used to start your aggregation pipeline. If you're not starting the pipeline with some combination of a $match, $limit, $sort , or $geoNear, it is going to need to consider every document in the collection and perform a full scan.

[Roycohen]

OK, Thanks.
So there is nothing we can do on our end...

Another question about the batch size.
If I set it to lower number, the result will be with all the data that fits to the aggregation pipeline, meaning I don't need to do some loops and and and iterations.
The batch size is just how many results the driver will get from the server and will issue get more commands until it gets the entire data, right?

[jmikola]

The batch size is just how many results the driver will get from the server and will issue get more commands until it gets the entire data, right?

That is correct. We're only modifying the payload size between the driver and the server. This won't affect how you iterate on the results via a cursor object, since those getMore commands happen behind the scenes.

[jmikola]

Closing this, as the last question was answered. Feel free to re-open as needed if the issue isn't fully resolved.