Skip to content

HBASE-23941 : FilterBy operator support in get_slowlog_responses API#1793

Merged
virajjasani merged 2 commits intoapache:masterfrom
virajjasani:HBASE-23941-master
May 30, 2020
Merged

HBASE-23941 : FilterBy operator support in get_slowlog_responses API#1793
virajjasani merged 2 commits intoapache:masterfrom
virajjasani:HBASE-23941-master

Conversation

@virajjasani
Copy link
Contributor

No description provided.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 prototool 0m 0s prototool was not available.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 3m 36s master passed
+1 💚 checkstyle 1m 55s master passed
+0 🆗 refguide 4m 57s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 spotbugs 14m 26s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 3m 21s the patch passed
-0 ⚠️ checkstyle 2m 0s root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76)
-0 ⚠️ rubocop 0m 11s The patch generated 7 new + 326 unchanged - 1 fixed = 333 total (was 327)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+0 🆗 refguide 4m 48s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 hadoopcheck 11m 8s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 hbaseprotoc 6m 2s the patch passed
+1 💚 spotbugs 15m 9s the patch passed
_ Other Tests _
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
77m 52s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #1793
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool rubocop refguide
uname Linux bee38ded3f0c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 79d5649
refguide https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/branch-site/book.html
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt
rubocop https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt
refguide https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/patch-site/book.html
Max. process+thread count 137 (vs. ulimit of 12500)
modules C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 26s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for branch
+1 💚 mvninstall 4m 58s master passed
+1 💚 compile 3m 16s master passed
+1 💚 shadedjars 6m 35s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 14s root in master failed.
-0 ⚠️ javadoc 0m 26s hbase-client in master failed.
-0 ⚠️ javadoc 0m 42s hbase-server in master failed.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for patch
+1 💚 mvninstall 4m 35s the patch passed
+1 💚 compile 3m 6s the patch passed
+1 💚 javac 3m 6s the patch passed
+1 💚 shadedjars 6m 24s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 25s hbase-client in the patch failed.
-0 ⚠️ javadoc 0m 45s hbase-server in the patch failed.
-0 ⚠️ javadoc 0m 13s root in the patch failed.
_ Other Tests _
-1 ❌ unit 214m 11s root in the patch failed.
250m 4s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #1793
Optional Tests javac javadoc unit shadedjars compile
uname Linux 84166ac79140 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 79d5649
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/testReport/
Max. process+thread count 2847 (vs. ulimit of 12500)
modules C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 45s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for branch
+1 💚 mvninstall 4m 34s master passed
+1 💚 compile 3m 3s master passed
+1 💚 shadedjars 6m 36s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 44s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 3m 58s the patch passed
+1 💚 compile 2m 46s the patch passed
+1 💚 javac 2m 46s the patch passed
+1 💚 shadedjars 6m 56s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 54s the patch passed
_ Other Tests _
-1 ❌ unit 329m 1s root in the patch failed.
368m 57s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #1793
Optional Tests javac javadoc unit shadedjars compile
uname Linux e37c3fc218b4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 79d5649
Default Java 1.8.0_232
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/testReport/
Max. process+thread count 5337 (vs. ulimit of 12500)
modules C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@bharathv bharathv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to make the make it a little concise, otherwise looks fine.

AdminProtos.SlowLogResponseRequest request,
List<TooSlowLog.SlowLogPayload> slowLogPayloadList) {
List<TooSlowLog.SlowLogPayload> filteredSlowLogPayloads = new ArrayList<>();
int totalFilters = getTotalFiltersCount(request);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (filterCount == 0) {
return ...
}

int totalFilters = getTotalFiltersCount(request);
for (TooSlowLog.SlowLogPayload slowLogPayload : slowLogPayloadList) {
int totalFilterMatches = 0;
if (StringUtils.isNotEmpty(request.getRegionName())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply only the filters that are actually in the request, why check if a filter exists in every iteration?


static List<TooSlowLog.SlowLogPayload> getFilteredLogs(
AdminProtos.SlowLogResponseRequest request, List<TooSlowLog.SlowLogPayload> logPayloadList) {
if (isFilterProvided(request)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for this if you implement the above logic I suggested..

return logPayloadList.subList(0, limit);
}

private static boolean isFilterProvided(AdminProtos.SlowLogResponseRequest request) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and you can remove this..

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 prototool 0m 0s prototool was not available.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for branch
+1 💚 mvninstall 4m 14s master passed
+1 💚 checkstyle 2m 24s master passed
+0 🆗 refguide 5m 26s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 spotbugs 17m 45s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for patch
+1 💚 mvninstall 4m 18s the patch passed
-0 ⚠️ checkstyle 2m 40s root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76)
-0 ⚠️ rubocop 0m 16s The patch generated 7 new + 326 unchanged - 1 fixed = 333 total (was 327)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+0 🆗 refguide 6m 11s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 hadoopcheck 13m 18s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 hbaseprotoc 7m 8s the patch passed
+1 💚 spotbugs 19m 51s the patch passed
_ Other Tests _
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
96m 53s
Subsystem Report/Notes
Docker Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #1793
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool rubocop refguide
uname Linux 1f2e423d6af6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / bda2094
refguide https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/branch-site/book.html
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt
rubocop https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt
refguide https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/patch-site/book.html
Max. process+thread count 124 (vs. ulimit of 12500)
modules C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@ndimiduk ndimiduk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good to me.

Reading through your examples, I realize it would have been nice if we supported filter operations other than equality. String prefix and regex come to mind, for matching tables, say, by namespace, or regions by rowkey range. Another idea is CIDR matching of client IP addresses by subnet.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 8s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for branch
+1 💚 mvninstall 4m 45s master passed
+1 💚 compile 3m 4s master passed
+1 💚 shadedjars 6m 21s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 13s root in master failed.
-0 ⚠️ javadoc 0m 25s hbase-client in master failed.
-0 ⚠️ javadoc 0m 39s hbase-server in master failed.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for patch
+1 💚 mvninstall 4m 27s the patch passed
+1 💚 compile 3m 4s the patch passed
+1 💚 javac 3m 4s the patch passed
+1 💚 shadedjars 6m 49s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 28s hbase-client in the patch failed.
-0 ⚠️ javadoc 0m 45s hbase-server in the patch failed.
-0 ⚠️ javadoc 0m 14s root in the patch failed.
_ Other Tests _
-1 ❌ unit 214m 29s root in the patch failed.
250m 49s
Subsystem Report/Notes
Docker Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #1793
Optional Tests javac javadoc unit shadedjars compile
uname Linux a821839371d8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / bda2094
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/testReport/
Max. process+thread count 2854 (vs. ulimit of 12500)
modules C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@bharathv bharathv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, lgtm.

@virajjasani
Copy link
Contributor Author

This change looks good to me.

Reading through your examples, I realize it would have been nice if we supported filter operations other than equality. String prefix and regex come to mind, for matching tables, say, by namespace, or regions by rowkey range. Another idea is CIDR matching of client IP addresses by subnet.

That's quite interesting idea 👍
Table search is already included with regionName.startsWith() sort of expression. Similarly, allowing search with operators other than just equality should help make life even simpler. Let me have a follow up Jira.

Thanks @bharathv @ndimiduk for the review.

@virajjasani virajjasani merged commit f0f9a20 into apache:master May 30, 2020
virajjasani added a commit that referenced this pull request May 30, 2020
…1793)

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
virajjasani added a commit that referenced this pull request May 30, 2020
…1793)

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
clarax pushed a commit to clarax/hbase that referenced this pull request Nov 15, 2020
…pache#1793)

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants