HBASE-24377 MemStoreFlusher throw NullPointerException #1721

binlijin · 2020-05-15T08:32:45Z

No description provided.

Apache-HBase · 2020-05-15T09:15:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 59s	master passed
+1 💚	checkstyle	1m 11s	master passed
+1 💚	spotbugs	2m 9s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 42s	the patch passed
+1 💚	checkstyle	1m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 4s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 10s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		34m 38s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux d07ebd34b6e7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a73132c`
Max. process+thread count	84 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

wchevreuil

Can you share more context of when you see such NPE? Is it reproducible, should we have test conditions? Also, which version were you testing it, line numbers on the stack trace don't seem to match with master version of MemStoreFlusher.

ramkrish86

LGTM.

binlijin · 2020-05-15T11:16:52Z

@wchevreuil
{code}
2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2] regionserver.MemStoreFlusher: Cache flusher failed for entry null
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
at java.lang.Thread.run(Thread.java:745)
{code}
We use version 2.2.2, see it on production cluster. And current have no idea how to test it.

Apache9 · 2020-05-15T11:59:04Z

Mind explaining more?

anoopsjohn · 2020-05-15T12:01:53Z

@binlijin , @wchevreuil - I think I got exactly when we will see this issue.

Method reclaimMemStoreMemory() Will be called from RSRpcServices every time during a write op (mutate, multi etc). This will check for the memory global barrier and do actions. You can see in this method the flushType will be set and then we add a WAKEUPFLUSH_INSTANCE entry so that the FlushHandler threads get this entry. This will again do watermark check and call flushOneForGlobalPressure but by then the flushType is been set already.

But there is another possible thread flow which can also check this barrier. When the FlushHandler wake up time reached and so it poll and see 'fqe' as null, still it check the barrier. Ya by then already the barrier might have reached. (But the other thread did not call reclaimMemStoreMemory and set the state). This time the call to flushOneForGlobalPressure will result in NPE.

Apache9 · 2020-05-15T12:08:05Z

Wait a minute, FlushType is a field of the MemStoreFlusher? This means we can only have one flush ongoing at the same time?

anoopsjohn · 2020-05-15T12:11:24Z

I think we dont even need this flushType state variable. The FlushHandler is anyway checking the type by calling the isAboveXXX() method (Even if waked up ). So we can pass in the flushType as local variable. Did not read so carefully. But at high level seems we can avoid the state.

Apache9 · 2020-05-15T12:11:56Z

I think we can just remove the field, and pass the flush type to flushOneForGlobalPressure? The write to this field in reclaimMemStoreMemory is just a dumb one, we will always check it in the flush thread...

Apache9 · 2020-05-15T12:12:22Z

I think we dont even need this flushType state variable. The FlushHandler is anyway checking the type by calling the isAboveXXX() method (Even if waked up ). So we can pass in the flushType as local variable. Did not read so carefully. But at high level seems we can avoid the state.

Oh yep, exactly. Agree.

ramkrish86 · 2020-05-15T12:19:12Z

I think we can just remove the field, and pass the flush type to flushOneForGlobalPressure? The write to this field in reclaimMemStoreMemory is just a dumb one, we will always check it in the flush thread...

Yes. Good one.

ramkrish86 · 2020-05-15T12:19:39Z

I think we dont even need this flushType state variable. The FlushHandler is anyway checking the type by calling the isAboveXXX() method (Even if waked up ). So we can pass in the flushType as local variable. Did not read so carefully. But at high level seems we can avoid the state.

Yes good points here.

Apache-HBase · 2020-05-15T12:44:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 47s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 12s	master passed
+1 💚	compile	1m 15s	master passed
+1 💚	shadedjars	6m 41s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 46s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 41s	the patch passed
+1 💚	compile	1m 13s	the patch passed
+1 💚	javac	1m 13s	the patch passed
+1 💚	shadedjars	6m 37s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 52s	hbase-server in the patch failed.
		_ Other Tests _
+1 💚	unit	212m 44s	hbase-server in the patch passed.
		244m 4s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux aa3eac083b81 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a73132c`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/testReport/
Max. process+thread count	2893 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-05-15T12:53:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 47s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 25s	master passed
+1 💚	compile	1m 0s	master passed
+1 💚	shadedjars	6m 17s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 58s	the patch passed
+1 💚	compile	1m 0s	the patch passed
+1 💚	javac	1m 0s	the patch passed
+1 💚	shadedjars	6m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	the patch passed
		_ Other Tests _
+1 💚	unit	224m 50s	hbase-server in the patch passed.
		252m 48s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux cd734709c9ef 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a73132c`
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/testReport/
Max. process+thread count	3376 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

binlijin · 2020-05-17T07:40:21Z

@Apache9 @anoopsjohn @ramkrish86 ok, remove the flushType field.

Apache-HBase · 2020-05-17T08:18:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 22s	master passed
+1 💚	checkstyle	1m 14s	master passed
+1 💚	spotbugs	2m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 51s	the patch passed
+1 💚	checkstyle	1m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 36s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 49s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		38m 35s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux ac4b874fe124 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `15627bb`
Max. process+thread count	84 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-05-17T10:19:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 44s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	shadedjars	5m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 22s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	5m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
-1 ❌	unit	135m 1s	hbase-server in the patch failed.
		159m 6s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8a6be9747578 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `15627bb`
Default Java	1.8.0_232
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/testReport/
Max. process+thread count	4841 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-05-17T11:08:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 38s	master passed
+1 💚	compile	1m 8s	master passed
+1 💚	shadedjars	6m 18s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 42s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 31s	the patch passed
+1 💚	compile	1m 8s	the patch passed
+1 💚	javac	1m 8s	the patch passed
+1 💚	shadedjars	6m 16s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 39s	hbase-server in the patch failed.
		_ Other Tests _
+1 💚	unit	180m 12s	hbase-server in the patch passed.
		208m 25s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1721
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2f904b4e3b83 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `15627bb`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/testReport/
Max. process+thread count	3283 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Ramkrishna <ramkrishna@apache.org> Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Ramkrishna <ramkrishna@apache.org> Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com> Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 9b3fe09) Change-Id: I3cba54b8c53775ccbf7f35c9a85c340af9e09663

HBASE-24377 MemStoreFlusher throw NullPointerException

6ed2525

wchevreuil reviewed May 15, 2020

View reviewed changes

ramkrish86 approved these changes May 15, 2020

View reviewed changes

anoopsjohn approved these changes May 15, 2020

View reviewed changes

remove field flushType

d9a3f44

Apache9 approved these changes May 17, 2020

View reviewed changes

binlijin merged commit 6ce1e14 into apache:master May 18, 2020

asfgit pushed a commit that referenced this pull request May 18, 2020

HBASE-24377 MemStoreFlusher throw NullPointerException (#1721)

9fd61f4

Signed-off-by: Ramkrishna <ramkrishna@apache.org> Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>

asfgit pushed a commit that referenced this pull request May 18, 2020

HBASE-24377 MemStoreFlusher throw NullPointerException (#1721)

b4f613d

Signed-off-by: Ramkrishna <ramkrishna@apache.org> Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>

asfgit pushed a commit that referenced this pull request May 18, 2020

HBASE-24377 MemStoreFlusher throw NullPointerException (#1721)

9b3fe09

Signed-off-by: Ramkrishna <ramkrishna@apache.org> Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-24377 MemStoreFlusher throw NullPointerException #1721

HBASE-24377 MemStoreFlusher throw NullPointerException #1721

binlijin commented May 15, 2020

Apache-HBase commented May 15, 2020

wchevreuil left a comment

ramkrish86 left a comment

binlijin commented May 15, 2020

Apache9 commented May 15, 2020

anoopsjohn commented May 15, 2020

Apache9 commented May 15, 2020

anoopsjohn commented May 15, 2020

Apache9 commented May 15, 2020

Apache9 commented May 15, 2020

ramkrish86 commented May 15, 2020

ramkrish86 commented May 15, 2020

Apache-HBase commented May 15, 2020

Apache-HBase commented May 15, 2020

binlijin commented May 17, 2020

Apache-HBase commented May 17, 2020

Apache-HBase commented May 17, 2020

Apache-HBase commented May 17, 2020

HBASE-24377 MemStoreFlusher throw NullPointerException #1721

HBASE-24377 MemStoreFlusher throw NullPointerException #1721

Conversation

binlijin commented May 15, 2020

Apache-HBase commented May 15, 2020

wchevreuil left a comment

Choose a reason for hiding this comment

ramkrish86 left a comment

Choose a reason for hiding this comment

binlijin commented May 15, 2020

Apache9 commented May 15, 2020

anoopsjohn commented May 15, 2020

Apache9 commented May 15, 2020

anoopsjohn commented May 15, 2020

Apache9 commented May 15, 2020

Apache9 commented May 15, 2020

ramkrish86 commented May 15, 2020

ramkrish86 commented May 15, 2020

Apache-HBase commented May 15, 2020

Apache-HBase commented May 15, 2020

binlijin commented May 17, 2020

Apache-HBase commented May 17, 2020

Apache-HBase commented May 17, 2020

Apache-HBase commented May 17, 2020