Skip to content

HBASE-26088 Fix thread leaks in conn#getBufferedMutator(tableName) method call#3506

Merged
anoopsjohn merged 3 commits intoapache:branch-2from
shahrs87:HBASE-26088-branch-2
Jul 21, 2021
Merged

HBASE-26088 Fix thread leaks in conn#getBufferedMutator(tableName) method call#3506
anoopsjohn merged 3 commits intoapache:branch-2from
shahrs87:HBASE-26088-branch-2

Conversation

@shahrs87
Copy link
Copy Markdown
Contributor

No description provided.

@shahrs87
Copy link
Copy Markdown
Contributor Author

@saintstack @anoopsjohn @apurtell This PR is just for branch-2. The behavior is different in master branch. In master brach, there is only AsyncConnection and I don't see it creating a new ThreadPoolExecutor but still studying master code base. Please review.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 37s Docker mode activated.
-0 ⚠️ yetus 0m 7s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 3m 43s branch-2 passed
+1 💚 compile 0m 26s branch-2 passed
+1 💚 shadedjars 6m 2s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 27s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 21s the patch passed
+1 💚 compile 0m 27s the patch passed
+1 💚 javac 0m 27s the patch passed
+1 💚 shadedjars 5m 58s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 25s the patch passed
_ Other Tests _
+1 💚 unit 2m 22s hbase-client in the patch passed.
25m 12s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux 44df0013242f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/testReport/
Max. process+thread count 346 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 10s Docker mode activated.
-0 ⚠️ yetus 0m 8s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 19s branch-2 passed
+1 💚 compile 0m 31s branch-2 passed
+1 💚 shadedjars 6m 47s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 30s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 0s the patch passed
+1 💚 compile 0m 30s the patch passed
+1 💚 javac 0m 30s the patch passed
+1 💚 shadedjars 6m 45s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 29s the patch passed
_ Other Tests _
+1 💚 unit 2m 42s hbase-client in the patch passed.
29m 11s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux d9a81630bbf8 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/testReport/
Max. process+thread count 292 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 46s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 41s branch-2 passed
+1 💚 compile 1m 22s branch-2 passed
+1 💚 checkstyle 0m 49s branch-2 passed
+1 💚 spotbugs 1m 41s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 29s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 checkstyle 0m 41s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 14m 36s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 spotbugs 1m 38s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 14s The patch does not generate ASF License warnings.
42m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3506
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux b3d2c31b3cad 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 86 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 11s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 3m 59s branch-2 passed
+1 💚 compile 0m 33s branch-2 passed
+1 💚 shadedjars 6m 41s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 28s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 0s the patch passed
+1 💚 compile 0m 32s the patch passed
+1 💚 javac 0m 32s the patch passed
+1 💚 shadedjars 6m 52s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 29s the patch passed
_ Other Tests _
+1 💚 unit 2m 50s hbase-client in the patch passed.
29m 0s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux f46173bf6ac8 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/testReport/
Max. process+thread count 298 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 5s Docker mode activated.
-0 ⚠️ yetus 0m 7s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 47s branch-2 passed
+1 💚 compile 0m 32s branch-2 passed
+1 💚 shadedjars 7m 18s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 30s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 0s the patch passed
+1 💚 compile 0m 26s the patch passed
+1 💚 javac 0m 26s the patch passed
+1 💚 shadedjars 6m 48s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 25s the patch passed
_ Other Tests _
+1 💚 unit 2m 56s hbase-client in the patch passed.
31m 13s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux 88ca43cee7ff 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/testReport/
Max. process+thread count 236 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

@anoopsjohn anoopsjohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit.
This issue is not there in branch-1 anyways right?
So for all branch-2 branches, we are sticking with the existing behaviour to have the compatibility and correcting the Javadoc. Pls mark this issue accordingly and we can address master branch (3.x also?) in another task. There we can even change to use the current javadoc way of using Connection's pool (If bug is there still)

* use the Connection's ExecutorService. This object can be used for long lived operations.
* {@link BufferedMutator} returned by this method is thread-safe.
* This accessor will create a new ThreadPoolExecutor and will be shutdown once we close the
* BufferedMutator. This object can be used for long lived operations.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: May be for API getBufferedMutator(BufferedMutatorParams params) we can say clearly that if user passes a ThreadPool in params, we will use that or else same as this method (new one and handled by us on close)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anoopsjohn Done ! Please review again !

@Reidddddd
Copy link
Copy Markdown
Contributor

Just a nit.
This issue is not there in branch-1 anyways right?
So for all branch-2 branches, we are sticking with the existing behaviour to have the compatibility and correcting the Javadoc. Pls mark this issue accordingly and we can address master branch (3.x also?) in another task. There we can even change to use the current javadoc way of using Connection's pool (If bug is there still)

branch-1 seems no such issue, because there's no this flag cleanupPoolOnClose, so the pool no matter created by default or provided by applications will be shutdown

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 36s Docker mode activated.
-0 ⚠️ yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 19s branch-2 passed
+1 💚 compile 0m 31s branch-2 passed
+1 💚 shadedjars 6m 50s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 30s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 3s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 shadedjars 6m 47s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 29s the patch passed
_ Other Tests _
+1 💚 unit 2m 49s hbase-client in the patch passed.
28m 54s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux b9d60aa2c2a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/testReport/
Max. process+thread count 302 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚 mvninstall 3m 26s branch-2 passed
+1 💚 compile 1m 3s branch-2 passed
+1 💚 checkstyle 0m 33s branch-2 passed
+1 💚 spotbugs 1m 6s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 18s the patch passed
+1 💚 compile 1m 3s the patch passed
+1 💚 javac 1m 3s the patch passed
+1 💚 checkstyle 0m 32s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 11m 44s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 spotbugs 1m 25s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 11s The patch does not generate ASF License warnings.
32m 39s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3506
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux d4f56114ed70 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 37s Docker mode activated.
-0 ⚠️ yetus 0m 7s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 5m 12s branch-2 passed
+1 💚 compile 0m 38s branch-2 passed
+1 💚 shadedjars 7m 54s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 31s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 18s the patch passed
+1 💚 compile 0m 33s the patch passed
+1 💚 javac 0m 33s the patch passed
+1 💚 shadedjars 7m 11s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 29s the patch passed
_ Other Tests _
+1 💚 unit 3m 1s hbase-client in the patch passed.
32m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #3506
Optional Tests javac javadoc unit shadedjars compile
uname Linux f51b88b7ae3c 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / cabf0dc
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/testReport/
Max. process+thread count 233 (vs. ulimit of 12500)
modules C: hbase-client U: hbase-client
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3506/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

@saintstack saintstack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great. Needs release note.

@anoopsjohn anoopsjohn merged commit 253b83d into apache:branch-2 Jul 21, 2021
anoopsjohn pushed a commit that referenced this pull request Jul 21, 2021
…thod call (#3506)

Co-authored-by: Rushabh Shah <rushabh.shah@rushabh-ltmflld.internal.salesforce.com>
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@duboce.net>
Signed-off-by: Reid Chan <reidchan@apache.org>
anoopsjohn pushed a commit that referenced this pull request Jul 21, 2021
…thod call (#3506)

Co-authored-by: Rushabh Shah <rushabh.shah@rushabh-ltmflld.internal.salesforce.com>
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@duboce.net>
Signed-off-by: Reid Chan <reidchan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants