Skip to content

Commit

Permalink
Support UPDATE_ROW operation
Browse files Browse the repository at this point in the history
  • Loading branch information
kdkeck committed Apr 14, 2014
1 parent f87d01f commit b90c687
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ public class MongoDBRiver extends AbstractRiverComponent implements River {
public final static String OPLOG_UPDATE = "o2";
public final static String OPLOG_OPERATION = "op";
public final static String OPLOG_UPDATE_OPERATION = "u";
public final static String OPLOG_UPDATE_ROW_OPERATION = "ur";
public final static String OPLOG_INSERT_OPERATION = "i";
public final static String OPLOG_DELETE_OPERATION = "d";
public final static String OPLOG_COMMAND_OPERATION = "c";
Expand Down
3 changes: 3 additions & 0 deletions src/main/java/org/elasticsearch/river/mongodb/Operation.java
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ public static Operation fromString(String value) {
return operation;
}
}
if (MongoDBRiver.OPLOG_UPDATE_ROW_OPERATION.equalsIgnoreCase(value)) {
return Operation.UPDATE;
}
}
return Operation.UNKNOWN;
}
Expand Down
1 change: 1 addition & 0 deletions src/main/java/org/elasticsearch/river/mongodb/Slurper.java
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ class SlurperException extends Exception {
private final String gridfsOplogNamespace;
private final String cmdOplogNamespace;
private final ImmutableList<String> oplogOperations = ImmutableList.of(MongoDBRiver.OPLOG_DELETE_OPERATION,
MongoDBRiver.OPLOG_UPDATE_ROW_OPERATION, // from TokuMX
MongoDBRiver.OPLOG_UPDATE_OPERATION, MongoDBRiver.OPLOG_INSERT_OPERATION, MongoDBRiver.OPLOG_COMMAND_OPERATION);
private final Client client;
private Mongo mongo;
Expand Down

5 comments on commit b90c687

@cheald
Copy link

@cheald cheald commented on b90c687 Apr 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might be incomplete. Toku uses this when it performs find_and_modify updates, so there may not necessarily be an o2 field available.

https://github.com/kdkeck/elasticsearch-river-mongodb/blob/master/src/main/java/org/elasticsearch/river/mongodb/Slurper.java#L426

There, Slurper expects to read an o2 field out of the oplog entry, but in the case that the op is ur, the fields from the oplog entry might be o and m, rather than o and o2.

I'm working on figuring out the correct handling for it, but it should be some parallel to runUpdateModsWithRowFromOplog.

@kdkeck
Copy link
Owner Author

@kdkeck kdkeck commented on b90c687 Apr 17, 2014 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cheald
Copy link

@cheald cheald commented on b90c687 Apr 18, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've fixed it in our branch. I picked up a few clues from this issue which pointed me at the fact that we just need the ID from the document, not the full document state rebuilt. From there it was just a matter of making sure that we can read the ID from o or o2 as appropriate.

There was a separate issue (which I think Anthony first talked to you about) in which there are namespaceless oplog entries that point to tokumx's oplog.refs collection. They just contain on object ID, which is pointer to the right ref; these are used for TokuMX transaction support per this blog post (near the bottom). Full transaction support will require handling those refs, but I think that shouldn't be too hard, since we only really care about the object IDs from the transaction, rather than replaying the full set of changes on our end.

The commit is here - mashable@6c6caf3 - I'm still learning the codebase, so I'm not sure if it's a very good fix or not, but if you like it, I'll submit a PR for it.

It needs tests, of course, but I haven't written those just yet - I'm grossly unfamiliar with the Mongo Java driver, which slowed me down a good bit there.

@kdkeck
Copy link
Owner Author

@kdkeck kdkeck commented on b90c687 Apr 18, 2014 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cheald
Copy link

@cheald cheald commented on b90c687 Apr 18, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, the oplog.refs need to be handled long-term. In our case, nothing ends up there that we care about, but it should definitely be handled in the general case.

That's a good point about the inner slurper loop. We might just be able to check the status in the inner loop, though - as written, it continually obtains a new cursor, which isn't what we want - we want it to be running the same cursor continually.

I look forward to the unit tests! Thanks! :)

Please sign in to comment.