Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support async iteration with query cursors and aggregation cursors #6737

Closed
vkarpov15 opened this Issue Jul 17, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@vkarpov15
Copy link
Collaborator

vkarpov15 commented Jul 17, 2018

for/await/of is natively supported in Node.js 10.x, so ideally you should be able to do something like:

for await (const doc of Model.find().cursor({ asyncIterable: true })) {}

or

for await (const doc of Model.find().cursor().asyncIterable()) {}

@lineus any thoughts?

@vkarpov15 vkarpov15 added this to the 5.3 milestone Jul 17, 2018

@lineus

This comment has been minimized.

Copy link
Collaborator

lineus commented Jul 18, 2018

I'm SUPER inexperienced with async iterators, so I had to do some research and test this out (below).

If I understand correctly, for-await-of converts the the synchronous iterable to an async iterable for us. I found this in a blog post but haven't been able to find the explicit language that would guarantee this behavior in the spec yet.

Is there a performance benefit to be gained by explicitly creating the async iterator ourselves?

I tested this out with Node 10.6

test.js

#!/usr/bin/env node --no-warnings
'use strict';

const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/test', { useNewUrlParser: true });
const conn = mongoose.connection;
const Schema = mongoose.Schema;
var called = 0;
const schema = new Schema({
  name: String
});

const Test = mongoose.model('test', schema);

const tests = [];
for (let i = 0; i < 10000; i++) {
  tests.push({ name: `name${i}`});
}

async function run() {
  await conn.dropDatabase()
  await Test.collection.insertMany(tests)

  for await (const doc of Test.find().cursor()) {
    if (doc.name === 'name5000') {
      console.log(`called: ${called}`);
      process.exit(0)
    } else {
      called++
    }
  }
}

run()

Output:

Node: ./test.js
called: 5000
Node:
@lineus

This comment has been minimized.

Copy link
Collaborator

lineus commented Jul 18, 2018

heh, it figures that not even 5 minutes after my last comment, I would check my rss client and see this post. I understand now.

It seems like must-add feature to me.

@vkarpov15

This comment has been minimized.

Copy link
Collaborator Author

vkarpov15 commented Jul 22, 2018

Yeah we should be able to just add a Symbol.asyncIterator function to queries that means you can use for/await/of with Model.find(). Will be very nice to have

@sibelius

This comment has been minimized.

Copy link

sibelius commented Aug 18, 2018

it worked well for, just got this warning

ExperimentalWarning: Readable[Symbol.asyncIterator] is an experimental feature. This feature could change at any time

should we add this to the docs?

@vkarpov15

This comment has been minimized.

Copy link
Collaborator Author

vkarpov15 commented Aug 19, 2018

@sibelius that's strange, I've never seen that warning before. Can you show me the code you used to get that warning?

@sibelius

This comment has been minimized.

Copy link

sibelius commented Aug 19, 2018

This only happens on node 10, the same code as above in the comments

vkarpov15 added a commit that referenced this issue Sep 28, 2018

@vkarpov15 vkarpov15 added fixed? and removed fixed? labels Sep 28, 2018

@vkarpov15

This comment has been minimized.

Copy link
Collaborator Author

vkarpov15 commented Sep 28, 2018

It works for query cursors, still need to add support for aggregation cursors.

vkarpov15 added a commit that referenced this issue Sep 28, 2018

@vkarpov15 vkarpov15 added the fixed? label Sep 28, 2018

@vkarpov15 vkarpov15 closed this in 57ecaf4 Sep 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.