HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS #3346
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
fengnanli
reviewed
Aug 28, 2021
| * @param addr Provided service address. | ||
| * @param lifelineAddr Provided service address. | ||
| */ | ||
| public NNHAServiceTarget(Configuration conf, |
There was a problem hiding this comment.
Instead of overloading the constructor like this, can we have the common part extracted as a function and call it inside? Having a lot of null check for parameters feels not ideal:
NNHAServiceTarget(Configuration conf, String nsId, String nnId) {
// do addrs assignment from config
.....
assignAutofailoverAndFencer();
}
NNHAServiceTarget(Configuration conf, String nsId, String nnId, String serviceAddr, addr, lifelineAddr) {
// do assignment from param
....
assignAutofailoverAndFencer();
}
Also please add a simple test for it.
There was a problem hiding this comment.
Yeah this will look cleaner. I will need to remove the final key word from some vars as the assignment is out of the constructor.
There was a problem hiding this comment.
Added a simple test per comment.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Aug 30, 2021
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
Outdated
Show resolved
Hide resolved
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
Outdated
Show resolved
Hide resolved
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
Show resolved
Hide resolved
.../src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterNamenodeHeartbeat.java
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Sep 1, 2021
| } | ||
| } catch (UnknownHostException e) { | ||
| LOG.error("Failed to resolve address: " + address); | ||
| ret.putAll(getResolvedAddressesForNnId( |
There was a problem hiding this comment.
I would extract the output of the method before doing putAll().
If something breaks, it is easier to debug if it points to the exact line too.
| * This is mainly used when configured nn is | ||
| * a domain record that has multiple hosts behind it. | ||
| */ | ||
| static String getConcatNnId(String nsId, String nnId, String hostname, int port) { |
| if (this.serviceAddress == null) { | ||
| LOG.error("Cannot locate RPC service address for NN {}, " + | ||
| "using RPC address {}", nnDesc, this.rpcAddress); | ||
| this.serviceAddress = this.rpcAddress; | ||
| } | ||
| if (resolvedHost != null) { |
There was a problem hiding this comment.
We do the same thing over and over for the lifeline and the others.
Maybe do all of them in a single shot?
The way to extract the port might also be expensive to be honest; creating a socket address is usually bad.
There was a problem hiding this comment.
Sure, that will be more clear.
Maybe I will just do a String split with ":"? The format coming from config should be ok.
There was a problem hiding this comment.
There might be already a function to do the split and make it resilient without having to set a socket.
Take a look at NetUtils to see if there is something.
If there is not, please create a static function that does the split and adds some checks
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Sep 1, 2021
| if (resolvedHost != null) { | ||
| // Get the addresses from resolvedHost plus the configured ports. | ||
| rpcAddress = resolvedHost + ":" | ||
| + rpcAddress.split(":")[1]; |
There was a problem hiding this comment.
Try to use something from NetUtils that doesn't open a socket and if it's not there create a method for extractPort or something.
There was a problem hiding this comment.
Sure, will add a simple method there to parse the host:port string.
goiri
approved these changes
Sep 1, 2021
| assertEquals(1002, NetUtils.getPortFromHostPortString("testHost:1002")); | ||
| try { | ||
| NetUtils.getPortFromHostPortString("testHost"); | ||
| fail("Should throw exception for wrong format"); |
There was a problem hiding this comment.
LambdaTestUtils#intercept does this.
There was a problem hiding this comment.
thx for the pointer, its pretty handy!
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Thanks for the contribution from @LeonGao91 and review from @goiri ! |
|
thanks for the reviews @goiri @fengnanli ! |
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
… DNS (apache#3346) Contributed by Leon Gao * Router to support resolving monitored namenodes with DNS * Style * fix style and test failure * Add test for NNHAServiceTarget const * Resolve comments * Fix test * Comments and style * Create a simple function to extract port * Use LambdaTestUtils.intercept * fix javadoc * Trigger Build
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?