-
Notifications
You must be signed in to change notification settings - Fork 543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not fail on write error #1547
Do not fail on write error #1547
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The refactoring makes the code diff difficult to read, I wasn't sure the purpose of making those new methods. Looks like the main purpose of the PR is to add mBasMessages, you can just insert this logic in place. Also what kind of errors do you see during messageWriter.write and you want to skip those errors?
@@ -223,37 +224,12 @@ protected boolean consumeNextMessage() { | |||
} | |||
|
|||
parsedMessage = mMessageParser.parse(transformedMessage); | |||
final double DECAY = 0.999; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did we lose the reset of DECAY after refactoring?
|
||
private void writeMessage(Message rawMessage, ParsedMessage parsedMessage) throws Exception { | ||
mMessageWriter.write(parsedMessage); | ||
mBadMessages *= DECAY; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DECAY
is final and used to decrease mBadMessages
as before. It does not make sense to declare new DECAY
variable on each writeMessage
call, so I moved it to static final constant.
Hey Henry, thanks for taking time to review my Pull Requests :). I do understand that it is hard to review this code, since the method was not that small.
To simplify the idea, before code looked like this: //...
try {
// parse partition
} catch (Exception e) {
// increment consumer.message_errors.count metric
// handle fail, re-throw if there were > 1000 errors
}
try {
// write message
} catch (Exception e) {
// just re-throw
} After this change there is only one try {
// parse partition
// write message
} catch (Exception e) {
// increment consumer.message_errors.count metric
// handle fail, re-throw if there were > 1000 errors
} This allows us to configure secor to not fail on writes and also
Our error is very similar to one, described in #1505. We use custom |
Your explanation makes sense, I will merge in your PR, thanks. |
Fixes #1505
This issue occurred to us too. Found existing issue about it and made this PR.
try/catch
blockconsumeNextMessage
to multiple smaller methods to make it more readable.