HDFS-17372. CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command #6530
+31
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Hexiaoqiao @zhangshuyan0 @tasanuma @tomscut Hi, sir. Could you please take a look at this problem? If needed, I will post an UT soonly. |
|
Thanks @hfutatzhanghb for your report. Is |
@ZanderXu Hi, sir. Yes, DNA_ACCESSKEYUPDATE blocked by BlockCommand. Good idea, but still have some problems to solve. For example, the heartbeat response may contain commands with kind of command type, we can't know whether it contains DNA_ACCESSKEYUPDATE except we iterate |
|
💔 -1 overall
This message was automatically generated. |
|
I am a bit confused about this. How often do you update the block key? @hfutatzhanghb |
|
we use default, 10 hours.
…---- Replied Message ----
| From | ***@***.***> |
| Date | 02/06/2024 19:20 |
| To | ***@***.***> |
| Cc | ***@***.***>***@***.***> |
| Subject | Re: [apache/hadoop] HDFS-17372. CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command (PR #6530) |
I am a bit confused about this. How often do you update the block key? @hfutatzhanghb
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Has command DNA_ACCESSKEYUPDATE not been executed for ten hours? |
|
yes, it was blocked by block command。
…---- Replied Message ----
| From | ***@***.***> |
| Date | 02/06/2024 19:42 |
| To | ***@***.***> |
| Cc | ***@***.***>***@***.***> |
| Subject | Re: [apache/hadoop] HDFS-17372. CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command (PR #6530) |
we use default, 10 hours.
Has command DNA_ACCESSKEYUPDATE not been executed for ten hours?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
zhangshuyan0
reviewed
Feb 7, 2024
There was a problem hiding this comment.
Agree with @ZanderXu . I think we should try not to modify namenode or protocol. Actually, there is a trick that key update command can only appear in the last two position of the cmds according to DatanodeManager#handleHeartbeat, so we do not need to iterate all commands on datanode.
|
@zhangshuyan0 Sir, thanks a lot for your valuable suggestion, Agree with you and @ZanderXu . I will try to use the trick you mentioned above to modify this PR. |
Hi, sir. I have some doubts that what should we do if we add new CMD type here? The access key update cmd will not be the last two in array. |
hfutatzhanghb
force-pushed
the
HDFS-17372
branch
5 times, most recently
from
9d1d14b
to
1f6fe12
Compare
|
@zhangshuyan0 @ZanderXu @tasanuma @tomscut Hi, sir. Have updated this PR.
|
hfutatzhanghb
force-pushed
the
HDFS-17372
branch
from
2b46a60
to
47d3f2e
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
HDFS-17372
branch
from
2e2de6d
to
2d39b50
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
…que to prevent high priority command blocked by low priority command.
hfutatzhanghb
force-pushed
the
HDFS-17372
branch
from
c2a43cc
to
53cb056
Compare
|
@Hexiaoqiao @zhangshuyan0 @ZanderXu Sir, this PR has run rightly for about two months. Could you please help me push this PR forward ? thanks a lot. |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Refer to HDFS-17372.
Recently, we met a critical problem in our production cluster which have lots of small files. In that cluster, per datanode has almost six million blocks.
After deleting large dir or recommision, some datanodes's SumOfActorCommandQueueLength metrics became very high as below picture shows.
After a while, we found some datanodes's write block ops became zero. That means client can not write to those datanodes. We found some logs on those datanodes:
That is to say, DNA_ACCESSKEYUPDATE command was blocked in CommandProcessingThread#queue. This can be deadly.
So, we should guarantee that command with high-priority should be processed in time.