HDFS-17267. Client send the same packet multiple times when method markSlowNode throws IOException. #6311
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
e817f5c to
6dea947
Compare
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
6dea947 to
c3694bf
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
c3694bf to
17f321a
Compare
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
the failed unit tests were passed in my local. It seems unreleated with this PR. |
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
17f321a to
ec10b91
Compare
Contributor
Author
|
@Hexiaoqiao @tomscut @zhangshuyan0 Sir, could you please help me review this code when you have free time? Thanks a lot. |
|
🎊 +1 overall
This message was automatically generated. |
hfutatzhanghb
changed the title
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
ec10b91 to
e689dec
Compare
|
🎊 +1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
2 times, most recently
from
e9c2121 to
3c0d085
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
2 times, most recently
from
318bb6c to
9b31b76
Compare
…rkSlowNode throws IOException.
hfutatzhanghb
force-pushed
the
HDFS-17267
branch
from
9b31b76 to
67870e4
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Since we have HDFS-16348, we can kick out SLOW node in pipeline when writing data to pipeline.
And I think it introduced a problem, that is the same packet will be sent twice or more times when we kick out SLOW node.
The flow are as below:
1、 DFSPacket p1 is pushed into dataQueue.
2、DataStreamer takes DFSPacket p1 from dataQueue.
3、Remove p1 from dataQueue and push p1 into ackQueue.
4、sendPacket(p1).
5、In ResponseProcessor#run, read pipelineAck for p1.
6、We meet SlOW node, so method markSlowNode throw IOException and does not execute
ackQueue.removeFirst();.7、In next loop of DataStreamer#run, we come into method processDatanodeOrExternalError and execute
dataQueue.addAll(0, ackQueue);.8、the p1 will be sent repeatedly.
We can debug the unit test method testPipelineRecoveryWithSlowNode to verify this PR.
Set breakpoint in DataStreamer#run :
LOG.debug("{} sending {}", this, one);.We can see the DFSPacket with seq=3 sends twice.
BTW, on datanode side. It will not write packet data twice, because if will compare the onDiskLen and offsetInBlock in method receivePacket(). if onDiskLen >= offsetInBlock, there will not happen writing data behavior.