Flaky test: TracerRecoversAfterOfflineTableIT #1811
Description
Test name(s)
- org.apache.accumulo.test.functional.FateStarvationIT.run
Describe the failure observed
I observed this test time out twice in two subsequent Jenkins runs on ASF's Jenkins. In both times, it timed out after the test timeout of 2 minutes, and the test was looping, waiting to observe traces in the trace table:
2020-11-25T08:16:47,039 [functional.ConfigurableMacBase] INFO : Trace table is online, should be able to find trace
2020-11-25T08:16:47,979 [functional.ConfigurableMacBase] INFO : Trace output:Did not find any traces!
2020-11-25T08:16:47,979 [functional.ConfigurableMacBase] INFO : Ignoring trace output as traceCount not greater than zero: 0
... message repeats for about a minute before test dies ...
2020-11-25T08:17:42,337 [functional.ConfigurableMacBase] INFO : Trace output:Did not find any traces!
2020-11-25T08:17:42,337 [functional.ConfigurableMacBase] INFO : Ignoring trace output as traceCount not greater than zero: 0
Testing Environment:
- Version of this project: 2.1.0-SNAPSHOT
- First commit known to fail (or current commit): d33e07a
Executing Maven: -B -f /home/jenkins/workspace/Accumulo/main/pom.xml -V -e -ntp clean verify -Dstyle.color=always -Dfailsafe.rerunFailingTestsCount=5 -Dsurefire.forkCount=2C
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/asfpackages/maven/apache-maven-3.6.3
Java version: 11.0.6, vendor: Oracle Corporation, runtime: /usr/local/asfpackages/java/jdk-11.0.6
Default locale: en_US, platform encoding: ISO-8859-1
OS name: "linux", version: "4.15.0-74-generic", arch: "amd64", family: "unix"
What have you tried already?
Re-running test seems to sometimes work. Test doesn't always fail.
Additional context
This might be a resource-constrained environment failure. It may also be that the SpanReceiver "misses" the traces we are expecting to create, and they will never appear, no matter how long we wait, due to the order of things coming back online or due to some other issue with tracing. It may be useful to test on a laptop, with similar resource constraints, and examine mini-tests logs to see what is going on server-side when the test fails.