Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS-11242. Support refreshTopology #3790

Closed
wants to merge 2 commits into from
Closed

HDFS-11242. Support refreshTopology #3790

wants to merge 2 commits into from

Conversation

secfree
Copy link
Contributor

@secfree secfree commented Dec 12, 2021

Description of PR

  1. Support refreshing topology of clients/datanodes in NameNode
  2. Usage: hdfs dfsadmin -refreshTopology <ipAddr>
  3. Based on the patch HDFS-11242.002.patch at https://issues.apache.org/jira/browse/HDFS-11242 with the following changes
    • Rebase to trunk
    • Change from refreshing all nodes to a specific node, as the former holds the lock for a long time

How was this patch tested?

  1. Tested with UT
  2. Tested in a real HDFS cluster
$ hdfs dfsadmin -fs hdfs://test -refreshTopology 1.2.3.4
Refresh topology successful at active-namenode/1.1.1.1:8020 for 1.2.3.4
Refresh topology successful at standby-namenode/1.1.1.2:8020 for 1.2.3.4
Refresh topology successful at observer-namenode/1.1.1.3:8020 for 1.2.3.4

Note

  1. It is also ok to pass in a hostname instead of an IP address, as keys of the cache may be hostnames sometimes.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 12m 54s Maven dependency ordering for branch
+1 💚 mvninstall 24m 43s trunk passed
+1 💚 compile 6m 1s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 5m 32s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 17s trunk passed
+1 💚 mvnsite 2m 59s trunk passed
+1 💚 javadoc 2m 18s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 3m 1s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 7m 11s trunk passed
+1 💚 shadedclient 23m 16s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 2m 40s the patch passed
+1 💚 compile 6m 2s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 cc 6m 2s the patch passed
+1 💚 javac 6m 2s the patch passed
+1 💚 compile 5m 32s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 cc 5m 32s the patch passed
+1 💚 javac 5m 32s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 7s /results-checkstyle-hadoop-hdfs-project.txt hadoop-hdfs-project: The patch generated 1 new + 453 unchanged - 0 fixed = 454 total (was 453)
+1 💚 mvnsite 2m 42s the patch passed
-1 ❌ javadoc 0m 34s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 97 unchanged - 2 fixed = 99 total (was 99)
+1 💚 javadoc 2m 43s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 7m 28s the patch passed
+1 💚 shadedclient 23m 34s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 18s hadoop-hdfs-client in the patch passed.
+1 💚 unit 316m 25s hadoop-hdfs in the patch passed.
-1 ❌ unit 31m 12s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 41s The patch does not generate ASF License warnings.
496m 40s
Reason Tests
Failed junit tests hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/1/artifact/out/Dockerfile
GITHUB PR #3790
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat
uname Linux d5388bb2ee1d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 3f80508
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/1/testReport/
Max. process+thread count 2452 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 2s Maven dependency ordering for branch
+1 💚 mvninstall 27m 16s trunk passed
+1 💚 compile 7m 25s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 6m 34s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 25s trunk passed
+1 💚 mvnsite 3m 31s trunk passed
+1 💚 javadoc 2m 40s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 3m 19s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 8m 38s trunk passed
+1 💚 shadedclient 25m 0s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
+1 💚 mvninstall 3m 12s the patch passed
+1 💚 compile 7m 0s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 cc 7m 0s the patch passed
+1 💚 javac 7m 0s the patch passed
+1 💚 compile 6m 29s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 cc 6m 29s the patch passed
+1 💚 javac 6m 29s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 18s the patch passed
+1 💚 mvnsite 3m 15s the patch passed
+1 💚 javadoc 2m 19s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 3m 7s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 8m 44s the patch passed
+1 💚 shadedclient 26m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 28s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 358m 12s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 6m 57s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
531m 53s
Reason Tests
Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/2/artifact/out/Dockerfile
GITHUB PR #3790
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat
uname Linux 1ba9c026ce65 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 380f81a
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/2/testReport/
Max. process+thread count 2442 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3790/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@secfree secfree closed this Jun 14, 2022
@hfutatzhanghb
Copy link
Contributor

hi, @secfree , why was this PR closed ?

@secfree
Copy link
Contributor Author

secfree commented Oct 26, 2022

hi, @secfree , why was this PR closed ?

@hfutatzhanghb I closed it because no one feedback/review in 6 months.The PR itself works well in our clusters.

@hfutatzhanghb
Copy link
Contributor

hi, @secfree , why was this PR closed ?

@hfutatzhanghb I closed it because no one feedback/review in 6 months.The PR itself works well in our clusters.

Very Nice, thks @secfree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@secfree @hadoop-yetus @hfutatzhanghb