[CI] FollowerFailOverIT testAddNewReplicasOnFollower failed

[droberts195]

FollowerFailOverIT.testAddNewReplicasOnFollower failed in 7.0 in
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+internalClusterTest/530/console

The repro command is:

./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=19484C0EDDEDE80D \
  -Dtests.class=org.elasticsearch.xpack.ccr.FollowerFailOverIT \
  -Dtests.method="testAddNewReplicasOnFollower" \
  -Dtests.security.manager=true \
  -Dtests.locale=ca-ES \
  -Dtests.timezone=Asia/Ashgabat \
  -Dcompiler.java=11 \
  -Druntime.java=8

This didn't reproduce locally for me.

There are 20 assertion errors in the log. The first one is:

10:17:12   2> java.lang.AssertionError: [follower-index] IndexNotFoundException[no such index [follower-index]]

Then:

10:17:12   2> java.lang.AssertionError: java.lang.RuntimeException: Cluster is already closed

Then the one that causes the test failure:

10:17:12 ERROR   43.1s J7 | FollowerFailOverIT.testAddNewReplicasOnFollower <<< FAILURES!
10:17:12    > Throwable #1: java.lang.AssertionError: timed out waiting for green state

[elasticmachine]

Pinging @elastic/es-distributed

[droberts195]

I muted the test for 7.0 in 23716b7

[dnhatn]

Log: add_new_replica.zip

[dnhatn]

The optimization using seq_no tripped the assertion assertDocDoesNotExist during peer-recovery.

  2> java.lang.AssertionError: doc [doc][63] exists [1] times in index
  2> 	at __randomizedtesting.SeedInfo.seed([19484C0EDDEDE80D]:0)
  2> 	at 
  2> 	at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1042)
  2> 	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:871)
  2> 	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:784)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:758)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1323)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1310)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:358)
  2> 	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:191)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:333)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:521)

I believe this relates to #38879.

[dnhatn]

This was fixed by #38879. I unmuted this test on 7.0. Thanks @droberts195 for reporting this issue.