Saw periodic failure of SummaryIT due to missing RFile

[ctubbsii]

The following is the error.

2020-03-08T02:55:14,387 [thrift.ProcessFunction] ERROR: Internal error processing startGetSummariesFromFiles
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist
  at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getSummaries(TabletServer.java:2032) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.startSummaryOperation(TabletServer.java:2046) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.startGetSummariesFromFiles(TabletServer.java:2123) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$8(TraceUtil.java:237) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at com.sun.proxy.$Proxy33.startGetSummariesFromFiles(Unknown Source) ~[?:?]
  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startGetSummariesFromFiles.getResult(TabletClientService.java:3607) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startGetSummariesFromFiles.getResult(TabletClientService.java:3586) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [libthrift-0.12.0.jar:0.12.0]
  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.12.0.jar:0.12.0]
  at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) [accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) [libthrift-0.12.0.jar:0.12.0]
  at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:114) [accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at org.apache.thrift.server.Invocation.run(Invocation.java:18) [libthrift-0.12.0.jar:0.12.0]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:37) [accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist
  at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
  at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getSummaries(TabletServer.java:2026) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
  ... 20 more
Caused by: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist

Evidence from the logs show the file was previously compacted.

2020-03-08T02:55:14,228 [tablet.files] DEBUG: Compacted 1;700000;600000 created file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/C000001z.rf from [file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001n.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001s.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001i.rf]

[jzgithub1]

@ctubbsii , maybe we have another race condition here. The file F000001x.rf is compacted into C000001z.rf. Then F000001x.rf gets deleted (since it had be compacted and wasn't needed anymore - I need to check if that is the way it works) but then the startGetSummariesFromFiles function looks for F000001x.rf after it had been deleted because it is in an outdated list of RFiles. We are seeing a pattern of race conditions in Jenkins being an issue like #1488 that we worked on at the Hackday. Needless to say that I can't replicate this error in the SummaryIT.testManyFiles test. I will work with a race condition hypothesis and try to come up with some solution to this unless you want me to back off of this ticket since I see that @keith-turner self assigned it to himself.

[ctubbsii]

@jzgithub1 The outdated list of RFiles is my working hypothesis also. I do not know if the test is easily reproducible. Since @keith-turner self-assigned it, I assume he's interested in working on it, but you'd have to check with him. Also, it is likely he knows more about this code anyway, since he implemented much of it.

[ctubbsii]

I haven't seen this in quite some time. I'm closing this.