Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-26157 Expose some IA.LimitedPrivate interface in TestingHBaseCl… #3643

Merged
merged 2 commits into from Sep 2, 2021

Conversation

Apache9
Copy link
Contributor

@Apache9 Apache9 commented Aug 28, 2021

…uster

@Apache9 Apache9 self-assigned this Aug 28, 2021
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 18s master passed
+1 💚 compile 0m 24s master passed
+1 💚 shadedjars 8m 43s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 22s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 59s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 shadedjars 8m 39s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 19s the patch passed
_ Other Tests _
-1 ❌ unit 1m 48s hbase-testing-util in the patch failed.
30m 39s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3643
Optional Tests javac javadoc unit shadedjars compile
uname Linux 89b22aaf98e6 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-testing-util.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/testReport/
Max. process+thread count 1978 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 18s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 5m 3s master passed
+1 💚 compile 0m 23s master passed
+1 💚 shadedjars 9m 8s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 20s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 50s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 shadedjars 9m 7s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 18s the patch passed
_ Other Tests _
-1 ❌ unit 1m 37s hbase-testing-util in the patch failed.
33m 44s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3643
Optional Tests javac javadoc unit shadedjars compile
uname Linux be54b8a92d85 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-testing-util.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/testReport/
Max. process+thread count 1836 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 6s master passed
+1 💚 compile 0m 35s master passed
+1 💚 checkstyle 0m 16s master passed
+1 💚 spotbugs 0m 42s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 38s the patch passed
+1 💚 compile 0m 32s the patch passed
+1 💚 javac 0m 32s the patch passed
+1 💚 checkstyle 0m 14s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 16s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 0m 51s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
37m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3643
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 2ed8a2265b66 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 29s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 54s master passed
+1 💚 compile 0m 24s master passed
+1 💚 shadedjars 8m 45s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 20s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 57s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 shadedjars 8m 38s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 19s the patch passed
_ Other Tests _
+1 💚 unit 1m 51s hbase-testing-util in the patch passed.
30m 18s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3643
Optional Tests javac javadoc unit shadedjars compile
uname Linux 553579dcf4bd 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/testReport/
Max. process+thread count 2049 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 20s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 47s master passed
+1 💚 compile 0m 23s master passed
+1 💚 shadedjars 9m 10s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 19s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 54s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
+1 💚 shadedjars 9m 12s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 18s the patch passed
_ Other Tests _
+1 💚 unit 1m 35s hbase-testing-util in the patch passed.
33m 31s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3643
Optional Tests javac javadoc unit shadedjars compile
uname Linux 99a2aff6120f 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/testReport/
Max. process+thread count 1910 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 3m 40s master passed
+1 💚 compile 0m 31s master passed
+1 💚 checkstyle 0m 14s master passed
+1 💚 spotbugs 0m 38s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 38s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 checkstyle 0m 14s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 13s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 0m 49s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 12s The patch does not generate ASF License warnings.
37m 4s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3643
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 6ef07fd13de3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 07c0eee
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 95 (vs. ulimit of 30000)
modules C: hbase-testing-util U: hbase-testing-util
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3643/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@gjacoby126
Copy link
Contributor

Thanks for these, @Apache9 . These will be helpful for the cell tag tests I mentioned earlier.

Another API sometimes needed when testing a coproc is access to the Coprocessor and/or CoprocessorEnvironment (or WALCoprocessor / WAL) to assert on some state.

All of these classes are LP(COPROC), but to get to them from the minicluster even today requires using IA.Private methods (HRegion.getCoprocessorHost, not present on the Region interface, which returns the IA.Private RegionCoprocessorHost. Same for WAL which is IA.Private but required to get to WALCoprocessor / WALObserver.

I'll keep looking to find other candidates for useful helper methods.

@Apache9
Copy link
Contributor Author

Apache9 commented Aug 31, 2021

Thanks for these, @Apache9 . These will be helpful for the cell tag tests I mentioned earlier.

Another API sometimes needed when testing a coproc is access to the Coprocessor and/or CoprocessorEnvironment (or WALCoprocessor / WAL) to assert on some state.

All of these classes are LP(COPROC), but to get to them from the minicluster even today requires using IA.Private methods (HRegion.getCoprocessorHost, not present on the Region interface, which returns the IA.Private RegionCoprocessorHost. Same for WAL which is IA.Private but required to get to WALCoprocessor / WALObserver.

I'll keep looking to find other candidates for useful helper methods.

Got the mission. I will also try to see how to expose these things and propose new PRs to add support for them. Mind giving me some pointer on what is your current usage? Maybe in Phoenix?

And, if the approach here is OK, do you want to approve or I need to find others to get approval? Thanks.


@Override
public Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName) {
return Optional.ofNullable(util.getMiniHBaseCluster().getRegionServer(serverName));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getRegionServer returns an HRegionServer...don't we need to keep going to the HRegionServer's RS coprocessor host to get the OnlineRegions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HRegionServers extends RegionServerServices, RegionServerServices extends MutableOnlineRegions, and finally MutableOnlineRegions extends OnlineRegions, so HRegionServer itself is 'OnlineRegions', we do not need to go to RS coprocessor to get it. The design here is not give CP implementations a way to get the Region interface, but do not want to give the CP implementations the ability to add or remove from the online regions directly as it could mess up the state.

* And also you could use the returned interface to get all regions on this region server, etc.
*/
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: does the "Interface" in the method name add any information? Seems like it could be "getOnlineRegions" to return an OnlineRegions. That's just like above, and in general throughout HBase, we have "getRegion" not "getRegionInterface" when we return a Region.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why I added the 'Interface' suffix is because I assumed the name 'getOnlineRegions ' is a bit confusing. It could also be explained as get the online regions, where we should return a list of Regions.
WDYT? I'm not a native English speaker so I'm fine with changing the name, if you think 'getOnlineRegions' is enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that because the return type is Optional(OnlineRegions), the method name getOnlineRegions is clear without the Interface, and we usually don't append Interface. But it's not a big deal either way.

try (RegionLocator locator = CONN.getRegionLocator(NAME)) {
loc = locator.getRegionLocation(info.getStartKey()).getServerName();
}
OnlineRegions onlineRegionsInterface = CLUSTER.getOnlineRegionsInterface(loc).get();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused why this isn't a ClassCastException going between HRegionServer and OnlineRegions but I see the test is passing in at least some of the test runs. Is there something interesting going on that I'm not understanding?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said above, the HRegionServer just extends the OnlineRegions interface, so there is no problem. So actually in the CP implementation, when you get an OnlineRegions instance, it is actually a HRegionServer :)

@gjacoby126
Copy link
Contributor

@Apache9 - No objection to limiting the review on this PR to just the methods you already added. I'll approve this PR when the review on those are done, and we can do additional methods in later JIRAs / PRs.

@Apache9
Copy link
Contributor Author

Apache9 commented Sep 1, 2021

@Apache9 - No objection to limiting the review on this PR to just the methods you already added. I'll approve this PR when the review on those are done, and we can do additional methods in later JIRAs / PRs.

Thank you @gjacoby126 . I've replied all your comments. Waiting for your response. Thanks.

Copy link
Contributor

@gjacoby126 gjacoby126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

* And also you could use the returned interface to get all regions on this region server, etc.
*/
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
Optional<OnlineRegions> getOnlineRegionsInterface(ServerName serverName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that because the return type is Optional(OnlineRegions), the method name getOnlineRegions is clear without the Interface, and we usually don't append Interface. But it's not a big deal either way.

@Apache9
Copy link
Contributor Author

Apache9 commented Sep 2, 2021

Thanks @gjacoby126 ! Let me merge.

@Apache9 Apache9 merged commit f022692 into apache:master Sep 2, 2021
Apache9 added a commit that referenced this pull request Sep 2, 2021
…uster (#3643)

Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants