New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large number of operations hangs server #2668

Closed
davidworkman9 opened this Issue Sep 22, 2014 · 25 comments

Comments

Projects
None yet
10 participants
@davidworkman9
Contributor

davidworkman9 commented Sep 22, 2014

Originally discussed here.

As our application data and usage has begun to scale we've run into an issue where a large number of MongoDB operations performed at one time will lock our server entirely for extended periods. The application becomes non-responsive to all users. What seems to be happening is a few thousand operations go into the oplog and Meteor struggles with the workload to determine which of those operations need to be passed down to the client. This could become a deal breaker for using Meteor with a large class of really useful applications unless addressed.

For our app this is unavoidable; no schema redesign will save us. There are tasks that need to affect huge numbers of MongoDB documents. Today it's 5,000 and up to a dozen users, but in the coming months it could easily be 100,000 documents and hundreds of users. Given what we are seeing with today's performance we know we are going to be in trouble.

Our solution for the immediate future is to turn oplog tailing off and revert to the original polling approach. With oplog tailing off I can do thousands of operations (directly to Mongo, not through a Meteor.Collection) and the server doesn't DDOS's itself. But we lose the benefit of the oplog tailor.

We wonder if the following (or something else that accomplishes the same goal) could be built into Meteor core somehow:

  • Provide the ability to turn oplog tailing off
  • Then we could execute our big query
  • Turn oplog tailing back on, which would invoke a poll and diff of all the subscriptions to get the missed changes
@bowlesdr

This comment has been minimized.

bowlesdr commented Sep 24, 2014

This is definitely a show-stopper for certain types of applications, including ours as it scales.

We have worked with mongo tailable cursors and found that using them to populate a queue gives considerable flexibility.

Here's an idea based upon our experience that might work.
meteor_oplog 1

@n1mmy

This comment has been minimized.

Member

n1mmy commented Sep 29, 2014

Thanks, @davidworkman9. This is something we'd like to address.

The general strategy of moving to poll-and-diff when the oplog gets very full sounds workable, but has lots of edge cases to be careful of.

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Oct 2, 2014

Any chance of getting an idea of where this fits in on the roadmap?

I'm curious what edge cases my pull request doesn't address, there is an exception being thrown, but that doesn't seem to actually affect anything, and removing the code that throws the exception also doesn't affect anything.

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Oct 20, 2014

Seeing as there are 1.0 release candidates starting to surface this looks like it's not going to be addressed in the short term. I'd love to help get this moving, I fear for our ability to scale our app and now that you've hit 1.0, I think you'll start to see a larger number of people facing this same issue as the stigma around Meteor being in pre-release lifts and more people start to use it.

Any discussion on this would really help ease my mind.

@n1mmy

This comment has been minimized.

Member

n1mmy commented Oct 21, 2014

Hi @davidworkman9,

As you've noted, this is not going to make it into the 1.0 release. However, the scalability of Meteor apps is a big concern at MDG and is definitely something we're going to be putting more effort into in upcoming releases. I can't give you an exact timetable -- we're focused on 1.0 and haven't really planned out post-1.0 schedules -- but I can say this is definitely something we want to address soon! Hope that helps ease your mind =)

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Dec 17, 2014

Now that 1.0 has been out for almost 2 months, anything new on when this might get addressed? I see a lot of (IMO) nice-to-haves on trello but nothing related to this.

@rgoomar

This comment has been minimized.

Contributor

rgoomar commented Dec 17, 2014

@davidworkman9 From what was said by the CTO of MongoDB in the last devshop, my guess is that issues like this are going to be resolved in a collective effort with the Mongo team. Not entirely sure when that will start though.

@glasser

This comment has been minimized.

Member

glasser commented Jan 27, 2015

@davidworkman9 As you probably noticed, our main maintenance focuses since 1.0 have been on command-line tool performance and maintainability, and package documentation. But I'm about to start on a stretch of Mongo-related maintenance, and I'm definitely going to try to work on this in the next month or so!

As Nick mentioned, we've been considering fallback to poll and diff when oplog gets behind for a while, and I'd like to implement that. Hopefully this will help with your app's issues.

The best way to ensure that the work we do helps with your app's issues instead of maybe some related but not quite applicable issues is to help us reproduce your issues directly. If there's any way that you can help us directly reproduce your app's performance issues, that will allow us to focus more specifically on fixing the actual user concerns rather than an artificial benchmark.

For example, there are multiple stages in the oplog tailing process where we could insert the "are we behind? skip ahead" analysis. Doing it in later stages may be easier to implement, but it's possible that it won't give enough performance improvements for you. Having real use cases as benchmarks will really help here!

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Feb 5, 2015

So our use case is pretty simple. We have a tree structure in Mongo that uses the parent reference model. Each document knows about it's parent, if the parent property is missing were at the base of the tree. If there are no documents with a parent reference to the current document were at the end of a branch.

If a document gets deleted, we need to recursively delete everything beneath it as well. This can easily turn into thousands of delete operations on the database. I've resorted to turning off the oplog tailor in production and doing the update through a raw node Mongo connection rather than through an instance of Meteor.Collection. So Meteor picks up on the changes on next poll of the DB.

The same is true of update, sometimes in this same tree structure we need to recursively update all children when a parent changes.

And, also we sometimes do a bulk insert of records, and when that happens the same performance issues show up. Really all it takes to show the performance issue is a >1000 operations, all within a short period of time (<15 seconds).

@glasser

This comment has been minimized.

Member

glasser commented Feb 5, 2015

OK. So as I stated: I am going to build a benchmark setup that works like that, and put work into optimizing that benchmark. I hope that the benchmark will accurately represent the setup of users like you. In case it doesn't, hopefully there will be time to test any improvements I make against your real setup.

@wearhere

This comment has been minimized.

wearhere commented Feb 5, 2015

@davidworkman9 What have you found the downside to be of turning off oplog tailing? We need to find a fix for this too.

@mitar

This comment has been minimized.

Collaborator

mitar commented Feb 5, 2015

Have you thought about storing the tree in something like mptt structure?

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Feb 5, 2015

@wearhere, Downsides I've experienced is the 10 second delay for updates outside of Meteor (we have a few node processes that write to the database periodically), and you can't horizontally scale, which hasn't been a problem I've run into yet as the need hasn't came up yet.

@mitar I haven't seen that approach before. Looks like updates would be incredibly intensive work. But either way, it wouldn't solve my problem unless I'm missing something? If you needed to delete a high level tree object and all of it's descendants you'd still put N number of operations in the oplog, right?

@mitar

This comment has been minimized.

Collaborator

mitar commented Feb 6, 2015

If you needed to delete a high level tree object and all of it's descendants you'd still put N number of operations in the oplog, right?

I am not 100%, but I think you can make it one MongoDB remove query. You say something like "remove all objects which have left branch larger than X and right branch smaller than Y" and then it removes whole subtree. I am not 100%, it is some time ago when I was working with those trees, but the whole idea is that you can get members of a tree or subtrees with one query.

Not sure if this would help the oplog, but it would definitely help MongoDB to be able to do one query. (Also better for isolation.)

@mitar

This comment has been minimized.

Collaborator

mitar commented Feb 6, 2015

Oh, forget it. You then have to renumber all other nodes in a tree. So you get many update operations anyway.

@davidworkman9

This comment has been minimized.

Contributor

davidworkman9 commented Feb 6, 2015

I think you can make it one MongoDB remove query.

I can do that now if I figure out the IDs first, the problem is Mongo's oplog puts a separate entry in the oplog for every single document affected. Problem isn't the number of queries, it's the number of documents affected.

@engelgabriel

This comment has been minimized.

Contributor

engelgabriel commented Feb 9, 2015

Very happy too see this issue getting attention, we runned into that problem too. Not sure I can provide a ready-to-clone test case, but I'll ask our developers if they can try. Our problem is that we (like many MongoDB apps do) replicate information from documents in other collections where they are referenced to "simulate" what we would be able to do with JOINs. So when the original document changes, we have to update the information everywhere it is replicated.

For example: we have a property thats store the owner of each document in our collections.

_user: {
    _id: "qwertyu"
    name: "John"
    group: {
        _id: "asdfgh"
        name: "Alpha"
    }
} 

This way, we can easily list all document of any group, but sometimes, the user will change from one group to another, and we will have to update all the documents where the group info is replicated.

We have a separate meteor app observing the oplog for this kind of changes, and checking our metadata schemes to trigger any replication updates needed.

During these rare situations, it would be nice if the other meteor apps using the same database could either notice or be notified, to put on hold the oplog tailing, and re run the observers once the replication updates are finished.

Like said above, the problem isn't the number of queries, it's the number of documents affected.

glasser added a commit that referenced this issue Feb 10, 2015

If oplog tailing gets too far behind, just poll
The number of entries that is "too far" to be behind is configurable
with $METEOR_OPLOG_TOO_FAR_BEHIND and defaults to 2000.

Fixes #2668.

glasser added a commit that referenced this issue Feb 10, 2015

If oplog tailing gets too far behind, just poll
The number of entries that is "too far" to be behind is configurable
with $METEOR_OPLOG_TOO_FAR_BEHIND and defaults to 2000.

Fixes #2668.
@glasser

This comment has been minimized.

Member

glasser commented Feb 10, 2015

@davidworkman9 @cscott et al: Can you try running your apps (or a benchmark that has this sort of issue) with --release OPLOG-BACKLOG@1? This is a preview release of some work I've done to alleviate the issue. (It is identical to 1.0.3.1 except for this fix, so it lacks other recent changes on devel that will go into the next release.)

It implements the strategy of, when getting too far behind in the oplog, just dropping all the entries and re-polling all live queries. It has one tunable parameter: how many oplog entries counts as "too far behind". Set it with $METEOR_OPLOG_TOO_FAR_BEHIND (defaults to 2000).

I'd love to hear how this works for people facing this issue!

@glasser

This comment has been minimized.

Member

glasser commented Feb 10, 2015

Hmm, found at least one bug which I will dig into later tonight or tomorrow. Worth experimenting with, but definitely don't run this in production :)

@glasser

This comment has been minimized.

Member

glasser commented Feb 10, 2015

OK, the bug is not a major problem for real apps: it's a stall that happens until anybody writes to anything anywhere in the database. So it's a problem for my test app where everything is idle after I do the big write in question. Will fix tonight.

@glasser

This comment has been minimized.

Member

glasser commented Feb 10, 2015

OK, try --release OPLOG-BACKLOG@2.

@glasser glasser closed this in a4626f1 Feb 10, 2015

glasser added a commit that referenced this issue Feb 10, 2015

@glasser

This comment has been minimized.

Member

glasser commented Feb 10, 2015

And now, --release OPLOG-BACKLOG@3 with some minor improvements!

This has now been merged to devel. Will probably start the release process for 1.0.4 the week after next. Would love to get feedback from users who have been facing this issue about whether or not this fix works for you, if you need non-default $METEOR_OPLOG_TOO_FAR_BEHIND values, etc.

@ryw

This comment has been minimized.

Contributor

ryw commented Feb 10, 2015

nice work @glasser will test out

@mizzao

This comment has been minimized.

Contributor

mizzao commented Mar 11, 2015

I can't believe that I missed this thread, given, that I started the original one on meteor-talk. I think this is an elegant solution though, and one that hadn't occurred to me.

I'd just like to point out though - I'm not sure if the issue is just with the number of documents affected. My intuition is that it is the number of documents times the number of queries. Inserting a lot of documents with no users connected is no problem, and having a lot of users with few writes was also fine. But doing both at the same time causes the CPU load: in asymptotic terms it's O(mn), where m is the number of observers/multiplexers and n the number of documents affected. This is because each oplog entry needs to be potentially matched to every distinct query.

Here's hoping that this approach neatly sidesteps the issue.

I can't directly benchmark this, as I only observed the issue with 100+ users connected and a ton of writes hitting the database. Do we have testing tools to replicate that scenario these days? :)

@glasser

This comment has been minimized.

Member

glasser commented Mar 11, 2015

@mizzao I'd be happy to hear if 1.0.4 helps your apps!

@mikowals mikowals referenced this issue Apr 12, 2015

Closed

Bulk.insert #2

cscott added a commit to cjb/codex-blackboard that referenced this issue Jan 10, 2016

Upgrade to Meteor 1.0.4.2.
Meteor release notes:
http://info.meteor.com/blog/meteor-104-mongo-cordova-template-subscriptions
https://github.com/meteor/meteor/blob/release/METEOR@1.0.4.2/History.md

Important performance improvements included in this release:
* If the oplog observe driver gets too far behind in processing the oplog,
  skip entries and re-poll queries instead of trying to keep up.
  meteor/meteor#2668
* Optimize common cases faced by the "crossbar" data structure (used by
  oplog tailing and DDP method write tracking).
  meteor/meteor#3697
* The oplog observe driver recovers from failed attempts to apply the
  modifier from the oplog (eg, because of empty field names).
* Avoid unnecessary work while paused in minimongo.
* Fix bugs related to observing queries with field filters: changed
  callbacks should not trigger unless a field in the filter has
  changed, and changed callbacks need to trigger when a parent of an
  included field is unset.
  meteor/meteor#2254

There are also API improvements which can lead to more efficient/less
mistake-prone subscriptions and observe.  We'll migrate to these new
APIs in the next few commits.

Change-Id: I58e464750a03d9cfedfa7f07c9e25e8a6460fcde
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment