Large number of operations hangs server

[davidworkman9]

Originally discussed here.

As our application data and usage has begun to scale we've run into an issue where a large number of MongoDB operations performed at one time will lock our server entirely for extended periods. The application becomes non-responsive to all users. What seems to be happening is a few thousand operations go into the oplog and Meteor struggles with the workload to determine which of those operations need to be passed down to the client. This could become a deal breaker for using Meteor with a large class of really useful applications unless addressed.

For our app this is unavoidable; no schema redesign will save us. There are tasks that need to affect huge numbers of MongoDB documents. Today it's 5,000 and up to a dozen users, but in the coming months it could easily be 100,000 documents and hundreds of users. Given what we are seeing with today's performance we know we are going to be in trouble.

Our solution for the immediate future is to turn oplog tailing off and revert to the original polling approach. With oplog tailing off I can do thousands of operations (directly to Mongo, not through a Meteor.Collection) and the server doesn't DDOS's itself. But we lose the benefit of the oplog tailor.

We wonder if the following (or something else that accomplishes the same goal) could be built into Meteor core somehow:

• Provide the ability to turn oplog tailing off
• Then we could execute our big query
• Turn oplog tailing back on, which would invoke a poll and diff of all the subscriptions to get the missed changes

[bowlesdr]

This is definitely a show-stopper for certain types of applications, including ours as it scales.

We have worked with mongo tailable cursors and found that using them to populate a queue gives considerable flexibility.

Here's an idea based upon our experience that might work.

[n1mmy]

Thanks, @davidworkman9. This is something we'd like to address.

The general strategy of moving to poll-and-diff when the oplog gets very full sounds workable, but has lots of edge cases to be careful of.

[davidworkman9]

Any chance of getting an idea of where this fits in on the roadmap?

I'm curious what edge cases my pull request doesn't address, there is an exception being thrown, but that doesn't seem to actually affect anything, and removing the code that throws the exception also doesn't affect anything.

[davidworkman9]

Seeing as there are 1.0 release candidates starting to surface this looks like it's not going to be addressed in the short term. I'd love to help get this moving, I fear for our ability to scale our app and now that you've hit 1.0, I think you'll start to see a larger number of people facing this same issue as the stigma around Meteor being in pre-release lifts and more people start to use it.

Any discussion on this would really help ease my mind.

[n1mmy]

Hi @davidworkman9,

As you've noted, this is not going to make it into the 1.0 release. However, the scalability of Meteor apps is a big concern at MDG and is definitely something we're going to be putting more effort into in upcoming releases. I can't give you an exact timetable -- we're focused on 1.0 and haven't really planned out post-1.0 schedules -- but I can say this is definitely something we want to address soon! Hope that helps ease your mind =)

[davidworkman9]

Now that 1.0 has been out for almost 2 months, anything new on when this might get addressed? I see a lot of (IMO) nice-to-haves on trello but nothing related to this.

[rgoomar]

@davidworkman9 From what was said by the CTO of MongoDB in the last devshop, my guess is that issues like this are going to be resolved in a collective effort with the Mongo team. Not entirely sure when that will start though.

[glasser]

@davidworkman9 As you probably noticed, our main maintenance focuses since 1.0 have been on command-line tool performance and maintainability, and package documentation. But I'm about to start on a stretch of Mongo-related maintenance, and I'm definitely going to try to work on this in the next month or so!

As Nick mentioned, we've been considering fallback to poll and diff when oplog gets behind for a while, and I'd like to implement that. Hopefully this will help with your app's issues.

The best way to ensure that the work we do helps with your app's issues instead of maybe some related but not quite applicable issues is to help us reproduce your issues directly. If there's any way that you can help us directly reproduce your app's performance issues, that will allow us to focus more specifically on fixing the actual user concerns rather than an artificial benchmark.

For example, there are multiple stages in the oplog tailing process where we could insert the "are we behind? skip ahead" analysis. Doing it in later stages may be easier to implement, but it's possible that it won't give enough performance improvements for you. Having real use cases as benchmarks will really help here!

[davidworkman9]

So our use case is pretty simple. We have a tree structure in Mongo that uses the parent reference model. Each document knows about it's parent, if the parent property is missing were at the base of the tree. If there are no documents with a parent reference to the current document were at the end of a branch.

If a document gets deleted, we need to recursively delete everything beneath it as well. This can easily turn into thousands of delete operations on the database. I've resorted to turning off the oplog tailor in production and doing the update through a raw node Mongo connection rather than through an instance of Meteor.Collection. So Meteor picks up on the changes on next poll of the DB.

The same is true of update, sometimes in this same tree structure we need to recursively update all children when a parent changes.

And, also we sometimes do a bulk insert of records, and when that happens the same performance issues show up. Really all it takes to show the performance issue is a >1000 operations, all within a short period of time (<15 seconds).

[glasser]

OK. So as I stated: I am going to build a benchmark setup that works like that, and put work into optimizing that benchmark. I hope that the benchmark will accurately represent the setup of users like you. In case it doesn't, hopefully there will be time to test any improvements I make against your real setup.

[wearhere]

@davidworkman9 What have you found the downside to be of turning off oplog tailing? We need to find a fix for this too.

[mitar]

Have you thought about storing the tree in something like mptt structure?

[davidworkman9]

@wearhere, Downsides I've experienced is the 10 second delay for updates outside of Meteor (we have a few node processes that write to the database periodically), and you can't horizontally scale, which hasn't been a problem I've run into yet as the need hasn't came up yet.

@mitar I haven't seen that approach before. Looks like updates would be incredibly intensive work. But either way, it wouldn't solve my problem unless I'm missing something? If you needed to delete a high level tree object and all of it's descendants you'd still put N number of operations in the oplog, right?

[mitar]

If you needed to delete a high level tree object and all of it's descendants you'd still put N number of operations in the oplog, right?

I am not 100%, but I think you can make it one MongoDB remove query. You say something like "remove all objects which have left branch larger than X and right branch smaller than Y" and then it removes whole subtree. I am not 100%, it is some time ago when I was working with those trees, but the whole idea is that you can get members of a tree or subtrees with one query.

Not sure if this would help the oplog, but it would definitely help MongoDB to be able to do one query. (Also better for isolation.)

[mitar]

Oh, forget it. You then have to renumber all other nodes in a tree. So you get many update operations anyway.

[davidworkman9]

I think you can make it one MongoDB remove query.

I can do that now if I figure out the IDs first, the problem is Mongo's oplog puts a separate entry in the oplog for every single document affected. Problem isn't the number of queries, it's the number of documents affected.

[engelgabriel]

Very happy too see this issue getting attention, we runned into that problem too. Not sure I can provide a ready-to-clone test case, but I'll ask our developers if they can try. Our problem is that we (like many MongoDB apps do) replicate information from documents in other collections where they are referenced to "simulate" what we would be able to do with JOINs. So when the original document changes, we have to update the information everywhere it is replicated.

For example: we have a property thats store the owner of each document in our collections.

_user: {
    _id: "qwertyu"
    name: "John"
    group: {
        _id: "asdfgh"
        name: "Alpha"
    }
} 

This way, we can easily list all document of any group, but sometimes, the user will change from one group to another, and we will have to update all the documents where the group info is replicated.

We have a separate meteor app observing the oplog for this kind of changes, and checking our metadata schemes to trigger any replication updates needed.

During these rare situations, it would be nice if the other meteor apps using the same database could either notice or be notified, to put on hold the oplog tailing, and re run the observers once the replication updates are finished.

Like said above, the problem isn't the number of queries, it's the number of documents affected.

[glasser]

@davidworkman9 @cscott et al: Can you try running your apps (or a benchmark that has this sort of issue) with --release OPLOG-BACKLOG@1? This is a preview release of some work I've done to alleviate the issue. (It is identical to 1.0.3.1 except for this fix, so it lacks other recent changes on devel that will go into the next release.)

It implements the strategy of, when getting too far behind in the oplog, just dropping all the entries and re-polling all live queries. It has one tunable parameter: how many oplog entries counts as "too far behind". Set it with $METEOR_OPLOG_TOO_FAR_BEHIND (defaults to 2000).

I'd love to hear how this works for people facing this issue!

[glasser]

Hmm, found at least one bug which I will dig into later tonight or tomorrow. Worth experimenting with, but definitely don't run this in production :)

[glasser]

OK, the bug is not a major problem for real apps: it's a stall that happens until anybody writes to anything anywhere in the database. So it's a problem for my test app where everything is idle after I do the big write in question. Will fix tonight.

[glasser]

OK, try --release OPLOG-BACKLOG@2.

[glasser]

And now, --release OPLOG-BACKLOG@3 with some minor improvements!

This has now been merged to devel. Will probably start the release process for 1.0.4 the week after next. Would love to get feedback from users who have been facing this issue about whether or not this fix works for you, if you need non-default $METEOR_OPLOG_TOO_FAR_BEHIND values, etc.

[ryw]

nice work @glasser will test out

[mizzao]

I can't believe that I missed this thread, given, that I started the original one on meteor-talk. I think this is an elegant solution though, and one that hadn't occurred to me.

I'd just like to point out though - I'm not sure if the issue is just with the number of documents affected. My intuition is that it is the number of documents times the number of queries. Inserting a lot of documents with no users connected is no problem, and having a lot of users with few writes was also fine. But doing both at the same time causes the CPU load: in asymptotic terms it's O(mn), where m is the number of observers/multiplexers and n the number of documents affected. This is because each oplog entry needs to be potentially matched to every distinct query.

Here's hoping that this approach neatly sidesteps the issue.

I can't directly benchmark this, as I only observed the issue with 100+ users connected and a ton of writes hitting the database. Do we have testing tools to replicate that scenario these days? :)

[glasser]

@mizzao I'd be happy to hear if 1.0.4 helps your apps!