New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAFKA-9769: Finish operations for leaderEpoch-updated partitions up to point ZK Exception #8479
Conversation
…xception occurs 1 - Remove fetchers for partitions whose leader epoch is updated. 2 - Finish delayed fetch and produce requests for those same partitions 3 - Re-add fetchers for those same partitions. 4 - Don't throw exception, but rather log it as an error about this occurrence. Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
@mjsax Hello Matthias! I would appreciate your review. Thanks in advance! |
I am not really familiar with this part of the code. Maybe @hachikuji or @cmccabe can help? |
Thanks for referring Matthias. Would appreciate your review @hachikuji @cmccabe |
@mjsax Hi Matthias, would you happen to know if there were any other reviewers available? I don't mind waiting, but was curious what the ETA usually appears to be. |
Feel to reach out to the dev mailing list and call for review. |
@andrewchoi5 Just want to make sure I understand the problem. The scenario is that we lose the zk session while handling a LeaderAndIsr request. Current LeaderAndIsr handling works like this:
So the problem occurs when we hit an error loading the topic configs in step 3). This potentially causes us to miss the needed state changes from that request and also prevents us from being able to retry them because the epoch has already been updated. Is that right? I guess the fix here is only a partial fix. We would still be left with the one failed partition, right? Off the top of my head, I am wondering whether we should be retrying the request at a lower level. For example, maybe cc @junrao |
@andrewchoi5 : Thanks for finding this issue. We fixed https://issues.apache.org/jira/browse/KAFKA-9932 recently. So, the |
Hi @junrao. Thank you for your review -- I have further synced up with @jjkoshy on this PR. The partition's new leader epoch update is actually happening after the point at which ZooKeeper Exception is thrown. Therefore, when the In the catch case for ZooKeeperClientException, I have populated the responseMap with the topic partition and the |
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
I haven't looked at this code in a while so I may not have enough context at this point, but I don't think we should use the network exception error code - i.e., this isn't a network issue between the coordinator and broker but between the broker and zk. Also, there doesn't seem to be any active retry attempt from the controller to resend the request in this scenario. |
Correct -- I wasn't able to find the best, close enough Errors exception to populate, especially since there was none related to ZooKeeper in that class. |
@andrewchoi5 : Since the controller only checks KAFKA_STORAGE_ERROR in LeaderAndIsrResponse now, perhaps we can just log an error without sending an error code back for now. |
…into andchoi_zkCatch
…KAFKA_STORAGE_ERROR in LeaderAndIsrResponse now, just log an error without sending an error code back for now. Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
Thanks, @junrao . |
Hello @junrao @hachikuji -- I have made some updates to address the comments. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewchoi5 : Thanks for the updated PR. One more comment.
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewchoi5 : Thanks for the updated PR. A few more comments below.
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewchoi5 : Thanks for the updated PR. One more comment below.
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
Thanks for the review @junrao -- let me know if there's anything else for revision. |
Closed by accident. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewchoi5 : Thanks for the new update. Still one more comment below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@junrao -- thanks for the comment. left a reply.
Signed-off-by: Andrew Choi <li_andchoi@microsoft.com>
@junrao -- let me know if anything else needs your attention. Thanks! |
ok to test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewchoi5 : Thanks for the latest PR. LGTM
KAFKA-9769: Finish operations for leaderEpoch-updated partitions up to point ZK Exception occurs.
https://issues.apache.org/jira/browse/KAFKA-9769
For example, in such case, we will have the following mechanism :
1 - P1 and P2 succeeds. leaderEpoch for them are incremented because no ZkException occurs
2 - while making follower for P3, ZkException occurs and the leaderEpoch is not updated and thus thepartitionsToMakeFollower += partition isn’t executed. We catch this ZkException in line 1498 and log it as an error. No Exception is thrown.
3 - After catching the exception, makeFollower for P4 is then not executed.
4 - so the partitionsToMakeFollower only contains P1, P2. And fetchers are added to these partitionsToMakeFollower
Signed-off-by: Andrew Choi li_andchoi@microsoft.com
Committer Checklist (excluded from commit message)