Experimental: expose the cursor in find() queries

[trenton42]

I have been thinking for quite a while about being able to run queries without loading the entire result set into memory. Here's what I came up with (view the Changes):

collection.find_with_cursor returns a deferred which results in a tuple: (docs, deferred). If there are no more documents remaining in the current query, a deferred will be returned which results in this tuple: ([], None)

For example:

docs, d = yield collection.find_with_cursor()
# collection.find(cursor=True) also works

while docs:
    # Do something with this page of docs here
    for i in docs:
        pass
    # Get the next set of docs from the query.
    docs, d = yield d

I have already implemented this (here), but let me know what you think, or if there are any improvements that could be made on this, then I will send over a pull request

[fiorix]

Seems interesting, but how does the cleanup work in case something fails (e.g. exception) and you have the cursor no longer accessible?

[trenton42]

That's a good question. It is basically the same as running one query per loop. If an exception occurs in the middle, it would be like failing in the middle of a generator. Mongo itself will expire a cursor after an amount of time (cursor expiration)

[psi29a]

@trenton42: Why not send a pull request? I'de like to see this.

[trenton42]

@psi29a Pull request submitted!