Use mgo.Collection.Bulk to prune docs #20

Merged
merged 1 commit into from Oct 31, 2016

Conversation

Projects
None yet
2 participants
Member

axw commented Oct 30, 2016

Use bulk document removal, rather than
a single RemoveAll with a query of "_id $in [...]".
The latter has performance problems in some
configurations of MongoDB. See https://bugs.launchpad.net/juju/+bug/1635311.

Excerpt from the bug:

Today I did the test again on the Azure instance where the issue was reproduced, instead of my local system where I did it first. On there, Mongo does not like the method used at all. In the logs from Comment 6, there's a log message stating that ~9000 transactions need pruning. Using the method from Juju, removing 10000 docs took about 2 minutes in one test. I wrote some more optimised code which does the same thing ~instantly.

(Review request: http://reviews.vapour.ws/r/5955/)

Use mgo.Collection.Bulk to prune docs
Use bulk document removal, rather than
a single RemoveAll with a query of "_id $in [...]".
The latter has performance problems in some
configurations of MongoDB. See lp:1635311.

mjs approved these changes Oct 31, 2016

This looks great for 2.0.

Before this gets used with 1.25 (MongoDB 2.4), it would be good to test the performance of mgo's Bulk fallback for older MongoDB versions.

Member

axw commented Oct 31, 2016

For posterity: I tested with mongo 2.4, and there is no performance penalty for the existing approach. Using Bulk isn't as fast there because 2.4 doesn't support bulk operations, and so mgo just turns it into a series of operations. Nevertheless, it only takes a few seconds to remove the 10000 documents, so we'll stick with just using Bulk across the board for simplicity.

Member

axw commented Oct 31, 2016

$$merge$$

@axw axw merged commit af2fa20 into juju:master Oct 31, 2016

axw added a commit to axw/mgopurge that referenced this pull request Feb 7, 2017

Apply juju/txn#20
Apply juju/txn#20,
fixing Issue #2.

@axw axw referenced this pull request in juju/mgopurge Feb 7, 2017

Closed

Apply https://github.com/juju/txn/pull/20 #4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment