OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange timeout failures

[alecgrieser]

I've seen at least once during a PRB run that the test OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange has failed. The error message isn't particularly helpful, with it essentially saying that there's some kind of timeout. Here's the stack trace:

com.apple.foundationdb.record.RecordCoreException
	at com.apple.foundationdb.record.provider.foundationdb.FDBExceptions.wrapException(FDBExceptions.java:200)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.lambda$new$0(FDBDatabase.java:167)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1037)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl.asyncToSync(FDBDatabaseRunnerImpl.java:340)
	at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.asyncToSync(OnlineIndexer.java:790)
	at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.buildIndex(OnlineIndexer.java:714)
	at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.buildSrcIndex(OnlineIndexerIndexFromIndexTest.java:71)
	at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange(OnlineIndexerIndexFromIndexTest.java:595)
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1028)
	... 95 more

Unclear if this is really the test or if the CI environment can get overwhelmed, though

[alecgrieser]

It's a little concerning that the error message appears not to have a real message associated with it ¯\_(ツ)_/¯

[scgray]

Another similar issue from https://fdb-record-layer-ci.foundationdb.org/job/prb-proto3/2758/

19:17:23 OnlineIndexerIndexFromIndexTest > testIndexFromIndexSrcVersionModifiedWithFallback() FAILED
19:17:23     com.apple.foundationdb.record.RecordCoreException
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBExceptions.wrapException(FDBExceptions.java:200)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.lambda$new$0(FDBDatabase.java:169)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1082)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl.asyncToSync(FDBDatabaseRunnerImpl.java:340)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.asyncToSync(OnlineIndexer.java:792)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.buildIndex(OnlineIndexer.java:713)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.buildSrcIndex(OnlineIndexerIndexFromIndexTest.java:72)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.testIndexFromIndexSrcVersionModifiedWithFallback(OnlineIndexerIndexFromIndexTest.java:754)
19:17:23 
19:17:23         Caused by:
19:17:23         java.util.concurrent.TimeoutException
19:17:23             at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
19:17:23             at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
19:17:23             at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1073)
19:17:23             ... 5 more
19:17:23