[FLAKY TEST] :bookkeeper-server:test failed with SIGSEGV on CI

[dlg99]

BUG REPORT

Test failed

org.apache.bookkeeper.bookie.CheckpointOnNewLedgersTest

  Test testCheckpoint PASSED
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f6668000840, pid=8491, tid=0x00007f66661e0700
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-linux64) (8.0_322-b06) (build 1.8.0_322-b06)
# Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x00007f6668000840
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/hs_err_pid8491.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007efc84000840, pid=8605, tid=0x00007efc81fee700
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-linux64) (8.0_322-b06) (build 1.8.0_322-b06)
# Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x00007efc84000840
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/hs_err_pid8605.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Process 'Gradle Test Executor 20' finished with non-zero exit value 134
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at https://docs.gradle.org/7.3.3/userguide/java_testing.html#sec:test_execution
org.gradle.process.internal.ExecException: Process 'Gradle Test Executor 20' finished with non-zero exit value 134
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at https://docs.gradle.org/7.3.3/userguide/java_testing.html#sec:test_execution
	at org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.stop(ForkingTestClassProcessor.java:183)
	at org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.endBatch(RestartEveryNTestClassProcessor.java:77)
	at org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.processTestClass(RestartEveryNTestClassProcessor.java:55)
	at sun.reflect.GeneratedMethodAccessor427.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:30)
	at org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)
	at org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)
	at org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)
	at org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)
	at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
	at java.lang.Thread.run(Thread.java:750)

org.apache.bookkeeper.bookie.CookieTest
...
FAILURE: Build failed with an exception.

* What went wrong:
> Task :bookkeeper-server:test FAILED
Execution failed for task ':bookkeeper-server:test'.

Expected behavior

repeatedly passing test

[dlg99]

hard to say without looking at .../bookkeeper-server/hs_err_pid8605.log but there is a chance this one is the same as #3040

@nicoloboschi is it possible to pick up logs like hs_err_pid.*\.log (if exists) from any directory in case of the test failure on CI and package with other logs for review? Unfortunately RocksDB segfaults end up creating these in working dir which happened to be bookkeeper-server/

# An error report file with more information is saved as:
# /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/hs_err_pid8605.log

[nicoloboschi]

maybe you need something like this: https://github.com/apache/pulsar/blob/60ef5e983e5e8956bd0b602b5741bd6255c6258a/.github/workflows/ci-unit-broker-client-api.yaml#L86-L99

alternatively, quick and dirty,

bookkeeper/.github/workflows/backward-compat-tests.yml

Line 54 in 1eae5e4

run: ./gradlew :tests:backward-compat:current-server-old-clients:test ${GRADLE_ARGS} || (tail -n +1 tests/backward-compat/current-server-old-clients/build/reports/tests/test/classes/* && tail -n +1 tests/backward-compat/current-server-old-clients/build/container-logs/**/* && exit 1)

[shoothzj]

Now we switch to maven, I think we can closed it.