Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-24195 : Admin.getRegionServers() should return live servers exc… #1523

Merged
merged 2 commits into from
Apr 16, 2020

Conversation

virajjasani
Copy link
Contributor

…luding decom RS optionally

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 39s Maven dependency ordering for branch
+1 💚 mvninstall 3m 26s master passed
+1 💚 checkstyle 1m 34s master passed
+1 💚 spotbugs 2m 55s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for patch
+1 💚 mvninstall 3m 9s the patch passed
-0 ⚠️ checkstyle 0m 28s hbase-client: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 10m 41s Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚 spotbugs 3m 16s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 26s The patch does not generate ASF License warnings.
35m 27s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #1523
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname Linux 217b48260cf1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 38954ed
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt
Max. process+thread count 94 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 33s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 39s Maven dependency ordering for branch
+1 💚 mvninstall 3m 31s master passed
+1 💚 compile 1m 18s master passed
+1 💚 shadedjars 4m 49s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 58s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 3m 22s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 shadedjars 4m 49s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 57s the patch passed
_ Other Tests _
+1 💚 unit 0m 56s hbase-client in the patch passed.
+1 💚 unit 138m 0s hbase-server in the patch passed.
163m 46s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #1523
Optional Tests javac javadoc unit shadedjars compile
uname Linux 5359120d89b7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 38954ed
Default Java 1.8.0_232
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/testReport/
Max. process+thread count 3949 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 33s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 38s Maven dependency ordering for branch
+1 💚 mvninstall 4m 15s master passed
+1 💚 compile 1m 37s master passed
+1 💚 shadedjars 5m 42s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 28s hbase-client in master failed.
-0 ⚠️ javadoc 0m 39s hbase-server in master failed.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 4m 1s the patch passed
+1 💚 compile 1m 33s the patch passed
+1 💚 javac 1m 33s the patch passed
+1 💚 shadedjars 6m 13s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 32s hbase-client in the patch failed.
-0 ⚠️ javadoc 0m 52s hbase-server in the patch failed.
_ Other Tests _
+1 💚 unit 1m 21s hbase-client in the patch passed.
+1 💚 unit 137m 56s hbase-server in the patch passed.
169m 3s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #1523
Optional Tests javac javadoc unit shadedjars compile
uname Linux 2dcb4f35cbea 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 38954ed
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/testReport/
Max. process+thread count 3781 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks useful. +1 (non-binding) after comments are addressed.

waitForServerDecom(serverToDecommission, true);

Assert.assertEquals(2, ADMIN.getRegionServers(true).size());

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Assert.assertEquals(3, ADMIN.getRegionServers(false).size()); for paranoid reviewers (cough cough me).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, definitely

ADMIN.recommissionRegionServer(serverToDecommission, Collections.emptyList());

Assert.assertEquals(3, ADMIN.getRegionServers(true).size());
waitForServerDecom(serverToDecommission, false);
Copy link
Member

@liuml07 liuml07 Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wait for decommission this server in the end?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the assertion Assert.assertEquals(3, ADMIN.getRegionServers(true).size()); should have made sure all RS are fine now. I understand in case that is not enough, you explicitly make sure the mini cluster is ready for other test cases. OK, good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wait for decommission this server in the end?

Actually, we are waiting for server to be recommissioned (maybe method name not correct, last arg determines). Let me change the order.

return allServers;
}
List<ServerName> decommissionedRegionServers = listDecommissionedRegionServers();
if (CollectionUtils.isNotEmpty(decommissionedRegionServers)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

 if (CollectionUtils.isNotEmpty(decommissionedRegionServers)) {
    allServers = new ArrayList<>(allServers);
    allServers.removeIf(decommissionedRegionServers::contains);
    return Collections.unmodifiableList(allServers);
  }
 return allServers;

to

  return allServers.stream()
    .filter(s -> !decommissionedRegionServers.contains(s))
    .collect(ImmutableList.toImmutableList());

I think this will save one time full copy (previous code copy twice, and this copies just once). Need to confirm.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason behind copy (new ArrayList<>(allServers)) was making allServers unmodifiable list. So ImmutableList.toImmutableList() seems to be solving our problem.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 37s Maven dependency ordering for branch
+1 💚 mvninstall 4m 52s master passed
+1 💚 checkstyle 1m 58s master passed
+1 💚 spotbugs 3m 50s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 4m 31s the patch passed
+1 💚 checkstyle 2m 2s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 14m 32s Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚 spotbugs 4m 10s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
47m 38s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #1523
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname Linux 35274c71e872 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 94b4101
Max. process+thread count 95 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 35s Maven dependency ordering for branch
+1 💚 mvninstall 3m 23s master passed
+1 💚 compile 1m 18s master passed
+1 💚 shadedjars 4m 45s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 59s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 3m 12s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 shadedjars 4m 46s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 56s the patch passed
_ Other Tests _
+1 💚 unit 0m 53s hbase-client in the patch passed.
+1 💚 unit 117m 35s hbase-server in the patch passed.
142m 54s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #1523
Optional Tests javac javadoc unit shadedjars compile
uname Linux 02d3226261b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 94b4101
Default Java 1.8.0_232
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/testReport/
Max. process+thread count 4334 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 26s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for branch
+1 💚 mvninstall 4m 30s master passed
+1 💚 compile 1m 36s master passed
+1 💚 shadedjars 5m 55s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 26s hbase-client in master failed.
-0 ⚠️ javadoc 0m 38s hbase-server in master failed.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 4m 17s the patch passed
+1 💚 compile 1m 41s the patch passed
+1 💚 javac 1m 41s the patch passed
+1 💚 shadedjars 6m 44s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 27s hbase-client in the patch failed.
-0 ⚠️ javadoc 0m 42s hbase-server in the patch failed.
_ Other Tests _
+1 💚 unit 1m 16s hbase-client in the patch passed.
+1 💚 unit 174m 14s hbase-server in the patch passed.
205m 33s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #1523
Optional Tests javac javadoc unit shadedjars compile
uname Linux 6a18cbfd0a7b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 94b4101
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/testReport/
Max. process+thread count 3029 (vs. ulimit of 12500)
modules C: hbase-client hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1523/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@virajjasani virajjasani merged commit d212dc4 into apache:master Apr 16, 2020
@virajjasani virajjasani deleted the HBASE-24195-master branch April 16, 2020 14:43
virajjasani added a commit that referenced this pull request Apr 16, 2020
#1523)

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
virajjasani added a commit that referenced this pull request Apr 16, 2020
#1523)

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
ddupg pushed a commit to ddupg/hbase that referenced this pull request Apr 20, 2020
apache#1523)

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants