[CI] CCRIT::testAutoFollowing failure

[hendrikmuhs]

found in: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/343/consoleFull

Reproduces with:

./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=EDFC74BFA921E27B -Dtests.class=org.elasticsearch.client.CCRIT -Dtests.method="testAutoFollowing" -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=Australia/Victoria -Dcompiler.java=11 -Druntime.java=8

relevant logs:

  1> [2018-11-27T20:07:44,174][INFO ][o.e.c.CCRIT              ] [testAutoFollowing] before test
  1> [2018-11-27T20:07:44,359][INFO ][o.e.c.CCRIT              ] [testAutoFollowing] initializing REST clients against [http://[::1]:43809]
  1> [2018-11-27T20:07:46,727][INFO ][o.e.c.CCRIT              ] [testAutoFollowing] after test
  2> REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=EDFC74BFA921E27B -Dtests.class=org.elasticsearch.client.CCRIT -Dtests.method="testAutoFollowing" -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=Australia/Victoria -Dcompiler.java=11 -Druntime.java=8
  2> REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=EDFC74BFA921E27B -Dtests.class=org.elasticsearch.client.CCRIT -Dtests.method="testAutoFollowing" -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=Australia/Victoria -Dcompiler.java=11 -Druntime.java=8
ERROR   2.61s | CCRIT.testAutoFollowing <<< FAILURES!
   > Throwable #1: ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception, reason=no shard follow tasks for [copy-logs-20200101]]]
   >    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
   >    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1703)
   >    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1679)
   >    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1444)
   >    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1416)
   >    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1383)
   >    at org.elasticsearch.client.CcrClient.pauseFollow(CcrClient.java:106)
   >    at org.elasticsearch.client.CCRIT.testAutoFollowing(CCRIT.java:156)
   >    at java.lang.Thread.run(Thread.java:748)
   >    Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43809], URI [/copy-logs-20200101/_ccr/pause_follow], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no shard follow tasks for [copy-logs-20200101]"}],"type":"illegal_argument_exception","reason":"no shard follow tasks for [copy-logs-20200101]"},"status":400}
   >            at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:687)
   >            at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
   >            at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1431)
   >            ... 41 more
   >    Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43809], URI [/copy-logs-20200101/_ccr/pause_follow], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no shard follow tasks for [copy-logs-20200101]"}],"type":"illegal_argument_exception","reason":"no shard follow tasks for [copy-logs-20200101]"},"status":400}
   >            at org.elasticsearch.client.RestClient$1.completed(RestClient.java:307)
   >            at org.elasticsearch.client.RestClient$1.completed(RestClient.java:292)
   >            at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   >            at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   >            at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   >            at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   >            at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   >            at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   >            at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   >            at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   >            at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   >            at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   >            at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   >            at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   >            at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   >            at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   >            ... 1 moreThrowable #2: org.apache.http.ConnectionClosedException: Connection closed
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:700)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
   >    at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:362)
   >    at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:193)
   >    at java.lang.Thread.run(Thread.java:748)
   > Caused by: org.apache.http.ConnectionClosedException: Connection closed
   >    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344)
   >    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   >    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   >    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   >    ... 1 more
  2> NOTE: leaving temporary files on disk at: /home/hendrik/work/git-elastic-prod/elasticsearch/client/rest-high-level/build/testrun/integTestRunner/J0/temp/org.elasticsearch.client.CCRIT_EDFC74BFA921E27B-001
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=513, maxMBSortInHeap=5.825471116586035, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@99b1b2), locale=es-AR, timezone=Australia/Victoria
  2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=4,threads=1,free=438407176,total=514850816
  2> NOTE: All tests run in this JVM: [CCRIT]
Completed [1/1] in 4.26s, 1 test, 1 error <<< FAILURES!

and

[2018-11-27T10:07:46,227][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] fatal error in thread [elasticsearch[node-0][masterService#updateTask][T#1]], exiting
java.lang.AssertionError: null
        at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$recordLeaderIndexAsFollowFunction$8(AutoFollowCoordinator.java:398) ~[?:?]
        at java.util.HashMap.compute(HashMap.java:1228) ~[?:?]
        at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$recordLeaderIndexAsFollowFunction$9(AutoFollowCoordinator.java:397) ~[?:?]
        at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2$1.execute(AutoFollowCoordinator.java:196) ~[?:?]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:272) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:202) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:137) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

I've muted this test. This hlrc test uncovered this failing assertion, but it has nothing to do with ccr hlrc support. Something inside ccr is causing this assertion to trip.