HBASE-24075: Fix a race between master shutdown and metrics (re)init #1394

bharathv · 2020-03-31T07:37:19Z

No description provided.

Apache-HBase · 2020-03-31T08:53:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 40s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 41s	master passed
+1 💚	checkstyle	2m 39s	master passed
+1 💚	spotbugs	14m 52s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 26s	the patch passed
+1 💚	checkstyle	2m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	11m 14s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚	spotbugs	15m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 44s	The patch does not generate ASF License warnings.
		68m 7s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1394
JIRA Issue	HBASE-24075
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle xml
uname	Linux 9c85616f9fd7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3d4124c`
Max. process+thread count	83 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-03-31T10:54:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 41s	Maven dependency ordering for branch
+1 💚	mvninstall	6m 52s	master passed
+1 💚	compile	4m 8s	master passed
+1 💚	shadedjars	6m 21s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 17s	hbase-hadoop-compat in master failed.
-0 ⚠️	javadoc	0m 16s	hbase-hadoop2-compat in master failed.
-0 ⚠️	javadoc	0m 18s	root in master failed.
-0 ⚠️	javadoc	0m 41s	hbase-server in master failed.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 34s	the patch passed
+1 💚	compile	4m 3s	the patch passed
+1 💚	javac	4m 3s	the patch passed
+1 💚	shadedjars	6m 10s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 15s	hbase-hadoop-compat in the patch failed.
-0 ⚠️	javadoc	0m 17s	hbase-hadoop2-compat in the patch failed.
-0 ⚠️	javadoc	0m 39s	hbase-server in the patch failed.
-0 ⚠️	javadoc	0m 19s	root in the patch failed.
		_ Other Tests _
+1 💚	unit	147m 28s	root in the patch passed.
		189m 25s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1394
JIRA Issue	HBASE-24075
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2587548a5c58 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3d4124c`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop2-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop2-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/testReport/
Max. process+thread count	5751 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

saintstack

Good.

saintstack · 2020-03-31T15:02:31Z

Perhaps related, I wonder if getFromCacheOrFetch should check if master is stopped and if so, NOT trying fetching clusterid from fs. Another issue?

bharathv · 2020-03-31T19:56:09Z

Perhaps related, I wonder if getFromCacheOrFetch should check if master is stopped and if so, NOT trying fetching clusterid from fs. Another issue?

I fixed it. I wanted to see how the metrics change alone affects the patch, hence marked it as a draft. The latest version has this change that makes getClusterId() a no-op if the server is shutting down.

Apache-HBase · 2020-03-31T21:08:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 41s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 55s	master passed
+1 💚	checkstyle	2m 39s	master passed
+1 💚	spotbugs	14m 51s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 26s	the patch passed
+1 💚	checkstyle	2m 36s	the patch passed
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	11m 18s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚	spotbugs	15m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		68m 13s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1394
JIRA Issue	HBASE-24075
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle xml
uname	Linux ca8d2fd7ab20 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e5a8435`
Max. process+thread count	83 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-03-31T21:53:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 44s	Maven dependency ordering for branch
+1 💚	mvninstall	6m 8s	master passed
+1 💚	compile	3m 37s	master passed
+1 💚	shadedjars	5m 35s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 18s	hbase-hadoop-compat in master failed.
-0 ⚠️	javadoc	0m 18s	hbase-hadoop2-compat in master failed.
-0 ⚠️	javadoc	0m 18s	root in master failed.
-0 ⚠️	javadoc	0m 40s	hbase-server in master failed.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 49s	the patch passed
+1 💚	compile	3m 42s	the patch passed
+1 💚	javac	3m 42s	the patch passed
+1 💚	shadedjars	5m 29s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 17s	hbase-hadoop-compat in the patch failed.
-0 ⚠️	javadoc	0m 17s	hbase-hadoop2-compat in the patch failed.
-0 ⚠️	javadoc	0m 39s	hbase-server in the patch failed.
-0 ⚠️	javadoc	0m 19s	root in the patch failed.
		_ Other Tests _
-0 ⚠️	unit	74m 59s	root in the patch failed.
		113m 4s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1394
JIRA Issue	HBASE-24075
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f57e4523f113 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e5a8435`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop2-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop2-compat.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/testReport/
Max. process+thread count	6781 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-03-31T23:14:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 55s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 43s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 52s	master passed
+1 💚	compile	3m 22s	master passed
+1 💚	shadedjars	5m 14s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	3m 45s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 37s	the patch passed
+1 💚	compile	3m 16s	the patch passed
+1 💚	javac	3m 16s	the patch passed
+1 💚	shadedjars	5m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	3m 44s	the patch passed
		_ Other Tests _
+1 💚	unit	150m 38s	root in the patch passed.
		193m 46s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1394
JIRA Issue	HBASE-24075
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4e86c11cfa8e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e5a8435`
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/testReport/
Max. process+thread count	5926 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1394/2/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

JMXCacheBuster resets the metrics state at various points in time. These events can potentially race with a master shutdown. When the master is tearing down, metrics initialization can touch a lot of unsafe state, for example invalidated FS objects. To avoid this, this patch makes the getMetrics() a no-op when the master is either stopped or in the process of shutting down. Additionally, getClusterId() when the server is shutting down is made a no-op. Simulating a test for this is a bit tricky but with the patch I don't locally see the long stacktraces from the jira. Signed-off-by: Michael Stack <stack@apache.org>

bharathv changed the title ~~HBASE-16211: Fix a race between master shutdown and metrics (re)init~~ HBASE-24075: Fix a race between master shutdown and metrics (re)init Mar 31, 2020

bharathv force-pushed the HBASE-16211 branch from 891b23f to 1678624 Compare March 31, 2020 07:40

saintstack approved these changes Mar 31, 2020

View reviewed changes

bharathv force-pushed the HBASE-16211 branch from 1678624 to 6d3c601 Compare March 31, 2020 19:53

bharathv marked this pull request as ready for review March 31, 2020 19:56

bharathv force-pushed the HBASE-16211 branch from 6d3c601 to b1dd0e6 Compare March 31, 2020 23:36

bharathv merged commit 6f213e9 into apache:master Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-24075: Fix a race between master shutdown and metrics (re)init #1394

HBASE-24075: Fix a race between master shutdown and metrics (re)init #1394

Uh oh!

bharathv commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

saintstack left a comment

Uh oh!

saintstack commented Mar 31, 2020

Uh oh!

bharathv commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HBASE-24075: Fix a race between master shutdown and metrics (re)init #1394

HBASE-24075: Fix a race between master shutdown and metrics (re)init #1394

Uh oh!

Conversation

bharathv commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

saintstack commented Mar 31, 2020

Uh oh!

bharathv commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Apache-HBase commented Mar 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants