[CI] XPackRestIT p0=indices.freeze/10_basic/Basic L74 Failure

[davidkyle]

java.lang.AssertionError: Failure at [indices.freeze/10_basic:74]: expected [2xx] status code but api [search] returned [503 Service Unavailable] [{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"stack_trace":"Failed to execute phase [query], all shards failed\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:
...

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+multijob-windows-compatibility/os=windows-2019/53/console
https://gradle-enterprise.elastic.co/s/t66es3e2pbe6g

Does not reproduce:

./gradlew ':x-pack:plugin:integTestRunner' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT" \
  -Dtests.method="test {p0=indices.freeze/10_basic/Basic}" \
  -Dtests.seed=143749813EC5AF86 \
  -Dtests.security.manager=true \
  -Dtests.locale=mt-MT \
  -Dtests.timezone=Africa/Ouagadougou \
  -Dcompiler.java=13 \
  -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[mark-vieira]

Couple more of these:

https://gradle-enterprise.elastic.co/s/gbbfvkr5jo5x4/tests/fcijb24gqtmnc-dyizuu7rv265i
https://gradle-enterprise.elastic.co/s/kzkf35kl7jwxk/tests/fcijb24gqtmnc-dyizuu7rv265i

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[dnhatn]

Relates to #51708?

[jpountz]

@dnhatn It looks like you already spent some time looking into it, I wonder whether you have ideas where the problem so that we could give a single area label to this failure?

[henningandersen]

One more failure on 7.6 here: https://gradle-enterprise.elastic.co/s/w2qeb5quewqde/tests/fcijb24gqtmnc-dyizuu7rv265i
But it also fails on master according to build-stats. Looks like we had an increasing frequency of this failing around the beginning of February:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-60d,mode:quick,to:now))&_a=(columns:!(_source),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'indices.freeze%20AND%20NOT(build.branch:pull-request-*)'),sort:!(process.time-start,desc))

[henningandersen]

The problem here is that the ilm-history index is created in the middle of the test (not explicitly by the test but implicitly by ILM, due to some earlier tests triggering ilm).

[2020-02-18T01:29:15,055] [ilm-history-1-000001] creating index, cause [api], templates [ilm-history], shards [1]/[0], mappings [_doc]

If the test is unfortunate enough to run the search using _all while the primary is still initializing, I believe it will fail like this. In the above test run, following happened in the previous test:

[2020-02-18T01:29:13,056] ... [.ml-notifications-000001] creating index, cause [auto(bulk api)], templates [.ml-notifications-000001], shards [1]/[1], mappings [_doc]

but the ilm-history index was not created in that test. The bulk processor used in ILMHistoryStore has a flush interval of 5 seconds. Going backwards I found:

[2020-02-18T01:29:09,771] ... [ilm-history-1-000001] creating index, cause [api], templates [ilm-history], shards [1]/[0], mappings [_doc]

Looks like it is maybe even the history entry for updating the history's ilm status that causes this.

Will open a PR to disable ILM history during these tests.