-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDFS-16502. Reconfigure Block Invalidate limit #4064
Conversation
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly look reasonable to me.
Do we have a doc somewhere that lists the available configuration keys that can be reconfigured? It would be nice for users to understand what can be reconfigured.
We don't seem to have a doc as such but operator/admin can use |
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM +1
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 1c0bc35)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Description of PR
Based on the cluster load, it would be helpful to consider tuning block invalidate limit (dfs.block.invalidate.limit). The only way we can do this without restarting Namenode as of today is by reconfiguring heartbeat interval
Math.max(heartbeatInt*20, blockInvalidateLimit)
, this logic is not straightforward and operators are usually not aware of it (lack of documentation), also updating heartbeat interval is not desired in all the cases.We should provide the ability to alter block invalidation limit without affecting heartbeat interval on the live cluster to adjust some load at Datanode level.
We should also take this opportunity to keep (heartbeatInterval * 20) computation logic in a common method.
How was this patch tested?
Locally and with UT.
For code changes: