ReindexIT#testDeleteByQuery and ReindexIT#testUpdateByQuery failures

[DaveCTurner]

Build scan:

https://gradle-enterprise.elastic.co/s/jbzekcqwjf7xc
https://gradle-enterprise.elastic.co/s/kbwxhs4vp764k
https://gradle-enterprise.elastic.co/s/cxcldr6dfkzba
https://gradle-enterprise.elastic.co/s/wwzzzkafjmkb4

Repro line:

e.g. REPRODUCE WITH: ./gradlew ':client:rest-high-level:asyncIntegTestRunner' --tests "org.elasticsearch.client.ReindexIT.testDeleteByQuery" -Dtests.seed=E520D301C9568B27 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=ru -Dtests.timezone=EST5EDT -Druntime.java=8

Reproduces locally?:

No. There's nothing particularly helpful in the node logs either AFAICT.

Applicable branches:

Only seen on 7.8 and 7.9.

Failure history:

Four failures over the last 90 days. Maybe duplicates #46301?

Failure excerpt:

org.elasticsearch.client.ReindexIT > testDeleteByQuery FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: a collection with size <1> |  
  | but: collection size was <0> |  
  | at __randomizedtesting.SeedInfo.seed([E520D301C9568B27:EBEFD62A9099F4B1]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.client.ReindexIT.testDeleteByQuery(ReindexIT.java:421)

The failure indicates that the rethrottle request did not return the expected task. However we know it's running before the rethrottle request since we call findTaskToRethrottle, and we also know it's running afterwards since subsequent tests report There are still tasks running after this test that might break subsequent tests [indices:data/write/delete/byquery].

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Reindex)

[henningandersen]

I think the reason this happens is other background delete and update by query jobs. findTaskToRethrottle could catch a wrong task and try to rethrottle that instead, which fits the symptoms (including that the task continues to run).

[tlrx]

The test failed again today with a similar failure:

org.elasticsearch.client.ReindexIT > testDeleteByQuery FAILED
    java.lang.AssertionError: 
    Expected: a collection with size <1>
         but: collection size was <0>
        at __randomizedtesting.SeedInfo.seed([EA8533146648813C:E44A363F3F87FEAA]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.client.ReindexIT.testDeleteByQuery(ReindexIT.java:266)

Build scan:

• https://gradle-enterprise.elastic.co/s/nt7iuq4iqugjk

Build stats history revels more failure over the past two months:

• https://gradle-enterprise.elastic.co/s/hx5pj3g5wf4l2
• https://gradle-enterprise.elastic.co/s/jlu6xw3t26ymw
• https://gradle-enterprise.elastic.co/s/3obmxoidozfqy
• https://gradle-enterprise.elastic.co/s/vy2rbrton7hyy

I'm reopening this issue for investigation and I'll mute the test.