Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update parse start script to handle the case that fsimage txid is cov… #78

Merged

Conversation

fengnanli
Copy link
Contributor

Sometimes the edit logs doesn't end exactly with the txid from fsimage, so we need to find the proper range which covers this id and thus to pick the right edit log

@fengnanli
Copy link
Contributor Author

@xkrogen Please review this one and thanks in advance!

@xkrogen
Copy link
Collaborator

xkrogen commented Feb 25, 2019

Hey @fengnanli , thanks for reporting this! I am curious, when will it occur that the edits file does not match the fsimage txn ID? I was under the impression that it always will; I've never seen it differ on our cluster. Maybe there are changes in newer versions of Hadoop which cause this?

The code looks good. I have two minor comments:

  • If we already use awk for processing, can we do away with the cut and simply do awk -F'-' to grab only the ending txn ID?
  • When using awk, I'm used to seeing the single quotes appear outside of the curly braces like '{ if .... }'. Is there any difference with the way you have specified it? I actually was under the impression that they way you specified it would not work because bash would interpret the curly braces as special characters. I should note that I am by no means a bash expert, though.

@fengnanli
Copy link
Contributor Author

@xkrogen In our cluster it seems there is never an edit log ending with the fsimage id. I am not sure why, and we are using 2.8.2.
Comments applied. I am not bash expert either :)

Copy link
Collaborator

@xkrogen xkrogen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @fengnanli . Interesting to hear that your edit log txn IDs never align. Maybe a change in 2.8+...

@xkrogen xkrogen merged commit 7e5ec90 into linkedin:master Feb 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants