Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save articles in bulk. #37

Open
supersam654 opened this issue Feb 26, 2016 · 0 comments
Open

Save articles in bulk. #37

supersam654 opened this issue Feb 26, 2016 · 0 comments

Comments

@supersam654
Copy link
Contributor

Inserting a document in Mongo takes ~1/3 of a second. Most of that is overhead for starting to insert something and then finish inserting it. When we crawl a bunch of feeds, we usually save over 100 articles each round. It would save a surprising amount of time to save in batches (see bulk_op).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant