HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7249

kgeisz · 2025-08-26T21:15:09Z

Summary

https://issues.apache.org/jira/browse/HBASE-29544

This is a cherry-pick from PR #7243.

IntegrationTestBackupRestore runs the backup and restore processes in a separate thread (or threads). PR #7203 (HBASE-29503) fixed an issue where the test would still pass even if an exception occurred in one of the threads. However, that fix only handles exceptions. It does not handle other throwables, such as AssertionError. This pull request extends the functionality implemented in PR #7203 to handle all throwables instead of just exceptions.

Manual Testing of Errors

Exceptions

I added a RuntimeException to BackupAndRestoreThread.run() to test exception handling:

public void run() {
  try {
    # Added this to force an exception
    throw new RuntimeException("Do we see this?");
  } catch (Throwable t) {
    LOG.error(
      "An error occurred in thread {} when performing a backup and restore with table {}: ",
      Thread.currentThread().getName(), this.table.getNameAsString(), t);
    this.throwable = t;
  }
}

Error message (note the Caused by...):

java.lang.AssertionError: An error occurred in a backup and restore thread

	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestMulti(IntegrationTestBackupRestore.java:238)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:207)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.RuntimeException: Do we see this?
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore$BackupAndRestoreThread.run(IntegrationTestBackupRestore.java:127)
	at java.base/java.lang.Thread.run(Thread.java:840)

AssertionErrors

I forced an AssertionError by modifying the assertEquals() in restoreVerifyTable():

Assert.assertEquals(-1, util.countRows(hTable));

Error message (note the Caused by...):

java.lang.AssertionError: An error occurred in a backup and restore thread

	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestMulti(IntegrationTestBackupRestore.java:238)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:207)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.AssertionError: expected:<-1> but was:<10000>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.restoreVerifyTable(IntegrationTestBackupRestore.java:346)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestSingle(IntegrationTestBackupRestore.java:319)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore$BackupAndRestoreThread.run(IntegrationTestBackupRestore.java:127)
	at java.base/java.lang.Thread.run(Thread.java:840)

…ng IntegrationTestBackupRestore to fail (apache#7243) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>

Apache-HBase · 2025-08-26T21:58:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-3 Compile Tests _
+1 💚	mvninstall	4m 38s		branch-3 passed
+1 💚	compile	0m 33s		branch-3 passed
+1 💚	javadoc	0m 18s		branch-3 passed
+1 💚	shadedjars	7m 27s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 57s		the patch passed
+1 💚	compile	0m 24s		the patch passed
+1 💚	javac	0m 24s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed
+1 💚	shadedjars	6m 40s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 48s		hbase-it in the patch passed.
		27m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7249/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7249
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 9a82e81c721e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-3 / `15aa20c`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7249/1/testReport/
Max. process+thread count	640 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7249/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-08-26T22:07:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-3 Compile Tests _
+1 💚	mvninstall	4m 40s		branch-3 passed
+1 💚	compile	0m 47s		branch-3 passed
+1 💚	checkstyle	0m 16s		branch-3 passed
+1 💚	spotbugs	0m 32s		branch-3 passed
+1 💚	spotless	1m 0s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 45s		the patch passed
+1 💚	compile	0m 37s		the patch passed
+1 💚	javac	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	hadoopcheck	14m 5s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		36m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7249/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7249
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux d343dd541894 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-3 / `15aa20c`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7249/1/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causi…

15aa20c

…ng IntegrationTestBackupRestore to fail (apache#7243) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>

taklwu approved these changes Aug 26, 2025

View reviewed changes

taklwu merged commit d2fa025 into apache:branch-3 Aug 26, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7249

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7249

Uh oh!

kgeisz commented Aug 26, 2025

Uh oh!

Apache-HBase commented Aug 26, 2025

Uh oh!

Apache-HBase commented Aug 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7249

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7249

Uh oh!

Conversation

kgeisz commented Aug 26, 2025

Summary

Manual Testing of Errors

Exceptions

AssertionErrors

Uh oh!

Apache-HBase commented Aug 26, 2025

Uh oh!

Apache-HBase commented Aug 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants