HADOOP-213: Reuse DBObjects when inserting data #129
Conversation
| @@ -102,12 +105,12 @@ public void commitTask(final TaskAttemptContext taskContext) { | |||
| int mwType = inputStream.readInt(); | |||
| if (MongoWritableTypes.BSON_WRITABLE == mwType) { | |||
| bw.readFields(inputStream); | |||
| bulkOp.insert(new BasicDBObject(bw.getDoc().toMap())); | |||
| insert.putAll(bw.getDoc().toMap()); | |||
There was a problem hiding this comment.
putAll won't remove the fields that were in insert from a previous document, so if not all documents have exactly the same schema, then later documents will have extraneous fields in them. A similar problem applies to query and modifiers below as well. If you made these all BasicDBObjects instead of DBObjects, you could call the clear method before putAll.
|
Thanks for the pull request! This change sounds fine to me after addressing the one comment. |
|
New pull request submitted with changes. |
|
I'm just checking in with this... did you see the comment I made on https://jira.mongodb.org/browse/HADOOP-213? |
|
Closing this PR for now, since the design of this pull request introduces some problems of its own that outweigh the performance benefits. Perhaps reusing a pool of DBObjects can be investigated at a later date for performance gain. |
In the MongoOutputCommiter a new DBObject is created for each insert resulting in unnecessary object allocation.
Informal testing has seen noticeable performance improvement (5-10%) for tens of millions of inserts.