HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points #2266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fengnanli
changed the title
goiri
reviewed
Sep 1, 2020
...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java
Outdated
Show resolved
Hide resolved
...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java
Show resolved
Hide resolved
| setUpMocks(); | ||
| } | ||
|
|
||
| private static void setUpMocks() throws IOException { |
| RouterRpcServer spyRpcServer = | ||
| Mockito.spy(routerContext.getRouter().createRpcServer()); | ||
| Whitebox | ||
| .setInternalState(routerContext.getRouter(), "rpcServer", spyRpcServer); | ||
| Mockito.doReturn(null).when(spyRpcServer).getFileInfo(Mockito.anyString()); | ||
|
|
||
| mockRpcClient = Mockito.spy(spyRpcServer.getRPCClient()); | ||
| Whitebox |
There was a problem hiding this comment.
I think whitebox needed to be deprecated.
...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java
Show resolved
Hide resolved
| * | ||
| * @param entry the new mount points added, could be from add or update. | ||
| * @return destination nameservices where the file doesn't exist. | ||
| * @throws IOException |
|
💔 -1 overall
This message was automatically generated. |
fengnanli
changed the title
goiri
reviewed
Sep 2, 2020
| * @return destination nameservices where the file doesn't exist. | ||
| * @throws IOException | ||
| */ | ||
| private List<String> verifyFileInDestinations(MountTable entry) |
There was a problem hiding this comment.
I would unit tests just for this function if possible.
There was a problem hiding this comment.
Yeah, I can add that.
This logic change will break a lot of current tests listed by yetus above. What is a good practice here? I have two ideas in mind:
- make the logic based on a command option when adding/updating mount tables.
- add mock (if possible) like what I did for TestRouterAdmin.java to fake out the dirs in namenodes.
I like 2 since I think this should be the default check, what's your opinion?
There was a problem hiding this comment.
I would fix those tests to have those paths existing yes.
There was a problem hiding this comment.
Thanks for the suggestion. I want to involve more people as well since when I started to fix the tests, I found there are quite a few tests targeting/testing cases for dangling mount points.
@aajisaka Can you share your thoughts as well?
There was a problem hiding this comment.
@goiri Uploaded an early version of trying to fix all tests. This is pretty tedious work so before I spend more time on this, let me know your thoughts.
There are mainly two types of tests when dealing with mount table:
- Use mock RouterRpcServer and so on, this way no downstream namenode calls are made. I put the mock as well, see the change for TestRouterAdmin.java
- Use real downstream namenode interaction, see TestRouterMountTable.java. I created the paths before calling mount points change.
I kept thinking a much easier way is to add a Router server side config to turn this on and the default is on. In the tests I can just turn the config off explicitly and this way I don't need to deal with individual tests.
There was a problem hiding this comment.
Eventually we may want to make all the tests be correct, but for now I'm fine setting up the config for just the new tests.
There was a problem hiding this comment.
If that's the case, I will try to fix all in one batch.
There was a problem hiding this comment.
Had a discussion with @ayushtkn in the jira ticket and he suggested just making the config to preserve the current workflow and enable this feature optionally. I updated the diff as that and it is not necessary to fix a lot of tests in that case.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
goiri
changed the title
goiri
approved these changes
Sep 9, 2020
...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java
Outdated
Show resolved
Hide resolved
goiri
reviewed
Sep 9, 2020
...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
The TestRouterRpcMultiDestination test passed locally. @goiri Can you help commit it? Thanks a lot! |
|
@ayushtkn Can you help commit the change? Thanks a lot! |
bilaharith
pushed a commit
to bilaharith/hadoop
that referenced
this pull request
Sep 27, 2020
…fore adding/updating mount points (apache#2266). Contributed by Fengnan Li.
Xushaohong
pushed a commit
to Xushaohong/hadoop
that referenced
this pull request
Jul 20, 2022
…inations before adding/updating mount points (apache#2266). Contributed by Fengnan Li.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…t points
NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute