New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model.find() is not returning all documents #609

Closed
gasteve opened this Issue Nov 10, 2011 · 36 comments

Comments

Projects
None yet
9 participants
@gasteve
Copy link

gasteve commented Nov 10, 2011

With a particular model, I have ~120 documents. When I run model.find({}, function(err, list) {…}); some documents for that model are not returned. If I add a clause to the query, I can make it return certain matching documents that are not returned with an empty query. The problem can also occur when I specify a query parameter that matches many documents (one or two are missing from the returned results).

I'm curious if others have experienced this problem and how they fixed it. Is it a mongoose issue, the mongo native module, or mongodb itself? What did others do to fix it?

I am running mongoose 1.3.7 and mongodb 1.8.1 on Ubuntu Linux.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 15, 2011

are you using a sparse index?

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 15, 2011

I am not using a sparse index, but I do have indexes.

I have 3 indexes for this schema…all three fields are of the type String, 2 are unique indexes and 1 is non unique... they are defined in the schema as follows:

field1: {type: String, unique: true},
field2: {type: String, unique: true},
field3: {type: String, index: true},

Note that "theModel.find({}, function(err, list) {…})" omits some of the documents for this model…however if I look up the document using one of the unique indexes, it finds returns it (so I know it's there). I don't believe the document will be returned if I query using one of the non indexed fields. So, it's behaving like the document appears in the index, but that it is not in whatever document collection that is being maintained for this schema as a whole. Is there any way to rebuild such a collection?

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 15, 2011

Actually, with further testing, I see that in some cases if I add a clause using a non-indexed field, more of the documents are returned…in one case, with a clause "{activated: true}" I'm getting 101 documents back, but with an empty clause, I only get 82…so there are a significant percentage of documents that are missing from the result set when an empty clause is used.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 15, 2011

care to share your schema and queries?

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 15, 2011

They are rather large and uninteresting…the relevant info is here. One question though…over time we've added fields to the schema…we've not removed any fields or indexes. We didn't do anything special to migrate instances in the DB. Could this cause these symptoms we're experiencing? Do we need to perform some kind of maintenance in the DB when changing the schema?

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 15, 2011

no maintenance should be necessary.

I can't help any further if more information is not provided b/c I cannot reproduce.

@aheckmann aheckmann closed this Nov 15, 2011

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 23, 2011

I upgraded to the latest mongoose, mongo plugin, and mongodb and I'm still having this issue. One thing I've noticed is that I am getting the following message in the mongodb console:

Wed Nov 23 13:42:09 [conn5] getMore: cursorid not found …

Also, I've confirmed that there are 166 documents for the collection using the mongo command line tool…however, the find is only returning 101 documents.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 23, 2011

So the mongo shell is returning 166 docs but mongoose only returns 101? If so, can you test the mongodb-native driver directly by running it like so:

model.collection.find({}, callback)

the docs returned won't be instances of your model but this will let us see if its a -native issue or a mongoose issue.

also, some googling makes me think it may be timing out? http://groups.google.com/group/mongodb-user/browse_thread/thread/88246dffd68dd117/8efb39c093e5c60e?show_docid=8efb39c093e5c60e

if thats the case you may wanna try Model.find(query, { timeout: 999999 }, callback)

this is a tricky one. hopefully we nail it.

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 23, 2011

Ok, I couldn't get model.collection.find({}, callback) to work…the second arg to the callback isn't an array of results…not sure what it was. However, I went to the mongo-native documentation and copied used this example (substituting my db and collection and bumping up the limit on results):

var mongodb = require('mongodb');
var server = new mongodb.Server("127.0.0.1", 27017, {});
new mongodb.Db('test', server, {}).open(function (error, client) {
  if (error) throw error;
  var collection = new mongodb.Collection(client, 'test_collection');
  collection.find({}, {limit:10}).toArray(function(err, docs) {
    console.dir(docs);
  });
});

I get 166 results when I do this and I do not get any messages about cursorid on the mongodb console…it doesn't appear that mongo-native is the culprit. I think I'll have a look at what the mongoose code is doing in the find() method and see if can spot anything. I can't help but think I must be doing something atypical, or else others would surely have encountered this.

Thanks for you help.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 23, 2011

ah yeah, forgot about -native returning cursor. cool, let me know

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 23, 2011

ok…I'm sneaking up on it…I switched to using the model.find().each() protocol to see if I hit the problem there as well. I did. Every time, after the same record, I get the cursorid not found output in the mongdb console and I get only 101 records. I looked at the code for the each() method and noticed the use of process.nextTick with the next function of the method. I decided to try it without using nextTick() and voila, I get all 166 records (and no message in the mongodb console). So, there must be something about the use of nextTick() that is causing mongodb to get confused about the cursorid (maybe it's a nodejs bug…searching).

@aheckmann aheckmann reopened this Nov 24, 2011

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 24, 2011

sounding more like a driver issue. wish I had a dump of your collection to test against this weekend. care to send it to me? aaron.heckmann@gmail.com

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 24, 2011

I mean "sounding more like a mongoose issue".

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

Can't provide the data…it has sensitive customer information. I have a few more clues though…and I think it is in the -native module. I went back to the find() method and couldn't find any similar use of nextTick …so I dug further into -native and it appears that the initial result is returning 101 documents…with that result, it gets a cursorId…it then tries to do a getMore(), but mongo doesn't appear to like the cursorId and silently fails…so the query just appears to return 101 results, but in reality, there was a failure. If I manually tweak things such that the initial query requests 1000 results, then it returns the full set. I don't have a good explanation as to why nextTick affected the behavior of each() …I suspect it's a bug in the handling of the cursorId and it only manifests itself when getMore() is need. You might be able to replicate it by forcing the numberToReturn to a low number in Cursor.prototype.generateQueryCommand (in cursor.js in the -native module). But, as I said, it seems like it's definitely a -native issue, not mongoose.

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

One other clue…I tried the iterative method again…with and without the use of nextTick() (in mongoose's query.each() method)…again when nextTick is used, I get 101 results and the bad cursorId message in the mongo console. When I removed nextTick(), I get the full 166 results…however, I noticed that in both cases the initial call fetches 101 results and in both cases, getMore() is being invoked. But, in one case, mongo is happy with the cursorId and the other it chokes. This must be some kind of concurrency issue causing the cursorId to get clobbered or closed. More digging.

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

Ok..this might be a mongoose issue after all. What I've discovered is that the problem goes away if I put a delay after connecting to the DB. I think what might be happening is a timing issue between the initial connection establishment (and the querying and index setup) and my query. If I give the mongoose schema related queries a bit of time to run before my query runs, everything seems ok. I wonder if some of the initial mongoose setup is clobbering open cursors? If that's the case, maybe an event "initialized" or something could be triggered to indicate it's safe to start interacting with the DB. Also, if something like that is the case, it would be good to check to see whether cursors associated with other client processes get clobbered (that would be bad). The context in which I'm using this is command line utilities that I run against a production DB. If those utilities result in cursors in the main production server processes to get clobbered, that would not be good.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 24, 2011

  1. there is an event fired after mongoose connected.

var db = mongoose.createConnection(...)
db.on('open', callback)

  1. when first connecting, mongoose ensures your indexes are created. I'll need to double check but it may be a race condition between this and the query that trips up mongodb.

so I can be certain I have the same setup for my tests, are you on mongoose 2.3.13 and mongodb 2.0.1?

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

Yes, I'm on 2.3.13 and 2.0.1. I do wait on the 'open' callback…but it seems things are still happening after that (like ensuring those indexes). I suspect ensuring those indexes causes open cursors to get closed. If that's the case and it closes all open cursors for all connections (even other processes that are connected), I would suggest adding a way to connect to the DB without ensuring the indexes (in my case I would only do that when I bounce the main server process). If it only affects the cursors for the current DB connection, then I think all that's necessary is to make sure the 'open' event doesn't fire until after the indexes have been ensured (otoh, you might still want to provide control over it…index creation could take significant time in some cases and not always be necessary depending on what you're doing).

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 24, 2011

try commenting out the indexes in your schema and test it. that should narrow things down a bit since ensureIndex won't get called at startup.

On Nov 24, 2011, at 7:48 AM, gastevereply@reply.github.com wrote:

Yes, I'm on 2.3.13 and 2.0.1. I do wait on the 'open' callback…but it seems things are still happening after that (like ensuring those indexes). I suspect ensuring those indexes causes open cursors to get closed. If that's the case and it closes all open cursors for all connections (even other processes that are connected), I would suggest adding a way to connect to the DB without ensuring the indexes (in my case I would only do that when I bounce the main server process). If it only affects the cursors for the current DB connection, then I think all that's necessary is to make sure the 'open' event doesn't fire until after the indexes have been ensured (otoh, you might still want to provide control over it…index creation could take significant time in some cases and not always be necessary depending on what you're doing).


Reply to this email directly or view it on GitHub:
#609 (comment)

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

That worked…I only commented out the indexes on the collection in question (though that doesn't necessarily mean cursors for other collections wouldn't get reset…since this is a timing issue). I'm going to look around in the mongodb documentation and see if it mentions anything about this.

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

I did a little more testing…I've noticed that one of my indexes (String/unique) always rebuilds…I have two other indexes (both String, one unique and the other non unique)…neither of those two indexes ever rebuilds…I've tried enabling just one at a time and it's always that one index that gets rebuilt, never the other two (going to try and find out why an index would always get rebuilt). So, it seems this is only an issue when an index actually gets rebuilt as a result of ensureIndex().

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Nov 24, 2011

Since we use ensureIndex it will attempt creation at each startup if it doesn't already exist. It must be failing for some reason, possibly a dup in the collection. If it fails, an error should be getting emitted on the connection, but it sounds like it may not be?

@gasteve gasteve closed this Nov 24, 2011

@gasteve

This comment has been minimized.

Copy link

gasteve commented Nov 24, 2011

There wasn't any indication of a failure in the console, however I just tried creating the index interactively via the mongo shell and discovered that indeed there is a duplicate and that caused it to fail. So, I guess for mongoose, the only suggestion I would have is to a) not fire the 'open' event until the ensureIndex() operations have completed and b) provide an option to open a connection without ensuring indexes (since I suspect the cursors for other clients could get clobbered by an index build and in a production environment you may need more control over when indexes do get built).

Thanks again for all your advice.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Dec 12, 2011

@diversario

This comment has been minimized.

Copy link

diversario commented Feb 9, 2012

Was this ever resolved in Mongoose or driver? I'm experiencing something similar to this.

I have a collection with the following model schema:

  var schema = new Schema({
    'userId': { 'type': String, 'required': true, 'index': { 'unique': true } }
  , 'props': [{
      'name': { 'type': String, 'index': true }
    , 'array': { 'type': [String], 'index': true }
  }]
  }, {'strict': true});

This model will be updated often and sometimes concurrently by multiple people. When running tests I found that find() returns null, null when I check for existing document with particular userId (so I know if I need to create one or to update existing doc). If I then run the same test on database from previous test run I get error 11000 about unique IDs and such.

I tried running mongoose.connection.db.ensureIndex('users', {'userId': 1}, {'unique': true, 'sparse': false}, function () {}); before find() but that didn't change anything.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Feb 10, 2012

its neither. see the MongoDB bug report above. its caused by the unique index creation failing and blowing away the current query cursor and happens at app startup.

@nmccready

This comment has been minimized.

Copy link

nmccready commented Jul 6, 2012

I can confirm this problem / bug on OSX and Ubuntu 12.04. Bug within MongoDB that is :P

@mitar

This comment has been minimized.

Copy link

mitar commented Jan 20, 2013

I can confirm that indeed this is the problem of setting an unique index on field which is not unique. The issue is that there is no error shown about that. But yes, query then returns 101 results.

@aheckmann

This comment has been minimized.

Copy link
Collaborator

aheckmann commented Jan 20, 2013

Please upvote/comment on the mongodb ticket to help raise its priority.

@mitar

This comment has been minimized.

Copy link

mitar commented Jan 20, 2013

I did.

@ramseydsilva

This comment has been minimized.

Copy link

ramseydsilva commented Apr 14, 2014

I can confirm, this is the case for me as well. If a unique index is set after creation of entries with non-unique/non-existent fields, the cursor doesn't return all values. It works if you narrow the query by querying on non unique fields.

@silverbucket

This comment has been minimized.

Copy link

silverbucket commented Jul 18, 2014

Can also confirm that I'm only receiving 101 records when a limit is not specified, either with a simple find() or with conditions.

@totty90

This comment has been minimized.

Copy link

totty90 commented Jan 20, 2015

Any news? I'm making tests and the indexes are run after the queries or inbetween. I found that the open event is ran after the indexes are ensured.
http://mongoosejs.com/docs/api.html#connection_Connection

Is this issue closed already?

Edit: The indexes are not ensured when the open event is fired.

@vkarpov15

This comment has been minimized.

Copy link
Collaborator

vkarpov15 commented Jan 20, 2015

@totty90 the server ticket unfortunately is still open. You can either disable index creation and just create them manually before your tests run, or you can wait for the Model.on('index') event to start your tests.

@totty90

This comment has been minimized.

Copy link

totty90 commented Jan 20, 2015

I've made a script to wait for the index event to fire. https://github.com/totty90/mongoose-connect

@vkarpov15

This comment has been minimized.

Copy link
Collaborator

vkarpov15 commented Jun 18, 2018

Instead of using Model.on('index'), in mongoose 5 you can do Model.init().then(), Model.init() returns a promise that is fulfilled after index build is done, or immediately if autoIndex is disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment