forked from richardwilly98/elasticsearch-river-mongodb
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
5 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
b90c687
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this might be incomplete. Toku uses this when it performs
find_and_modify
updates, so there may not necessarily be ano2
field available.https://github.com/kdkeck/elasticsearch-river-mongodb/blob/master/src/main/java/org/elasticsearch/river/mongodb/Slurper.java#L426
There, Slurper expects to read an
o2
field out of the oplog entry, but in the case that the op isur
, the fields from the oplog entry might beo
andm
, rather thano
ando2
.I'm working on figuring out the correct handling for it, but it should be some parallel to runUpdateModsWithRowFromOplog.
b90c687
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b90c687
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've fixed it in our branch. I picked up a few clues from this issue which pointed me at the fact that we just need the ID from the document, not the full document state rebuilt. From there it was just a matter of making sure that we can read the ID from
o
oro2
as appropriate.There was a separate issue (which I think Anthony first talked to you about) in which there are namespaceless oplog entries that point to tokumx's
oplog.refs
collection. They just contain on object ID, which is pointer to the right ref; these are used for TokuMX transaction support per this blog post (near the bottom). Full transaction support will require handling those refs, but I think that shouldn't be too hard, since we only really care about the object IDs from the transaction, rather than replaying the full set of changes on our end.The commit is here - mashable@6c6caf3 - I'm still learning the codebase, so I'm not sure if it's a very good fix or not, but if you like it, I'll submit a PR for it.
It needs tests, of course, but I haven't written those just yet - I'm grossly unfamiliar with the Mongo Java driver, which slowed me down a good bit there.
b90c687
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b90c687
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, the oplog.refs need to be handled long-term. In our case, nothing ends up there that we care about, but it should definitely be handled in the general case.
That's a good point about the inner slurper loop. We might just be able to check the status in the inner loop, though - as written, it continually obtains a new cursor, which isn't what we want - we want it to be running the same cursor continually.
I look forward to the unit tests! Thanks! :)