EnrichSecurityIT fails due to .monitoring name deprecation

[pgomulka]

CI Link

https://gradle-enterprise.elastic.co/s/oadbbzpxkn6tg/tests/:x-pack:plugin:enrich:qa:rest-with-security:javaRestTest/org.elasticsearch.xpack.enrich.EnrichSecurityIT/testBasicFlowDouble?top-execution=1

Repro line

https://gradle-enterprise.elastic.co/s/oadbbzpxkn6tg

Does it reproduce?

No

Applicable branches

master

Failure history

No response

Failure excerpt

org.elasticsearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:41171], URI [/my-index/_doc/1?pipeline=my_pipeline], status line [HTTP/1.1 201 Created]
Warnings: [index name [.monitoring-es-7-2022.02.01] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices] |  
{"_index":"my-index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

failed tests
EnrichSecurityIT » testBasicFlowDouble
EnrichSecurityIT » testDeleteExistingPipeline
EnrichSecurityIT » testBasicFlowDate
EnrichSecurityIT » testImmutablePolicy
EnrichSecurityIT » testBasicFlowKeyword
EnrichSecurityIT » testDeleteIsCaseSensitive

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

This one is odd. As mentioned above it looks like the root failure is:

org.elasticsearch.xpack.enrich.EnrichSecurityIT > testBasicFlowDouble FAILED
    org.elasticsearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:41171], URI [/my-index/_doc/1?pipeline=my_pipeline], status line [HTTP/1.1 201 Created]
    Warnings: [index name [.monitoring-es-7-2022.02.01] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices]
    {"_index":"my-index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
        at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:342)
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:312)
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:287)
        at app//org.elasticsearch.test.enrich.CommonEnrichRestTestCase.setupGenericLifecycleTest(CommonEnrichRestTestCase.java:122)
        at app//org.elasticsearch.test.enrich.CommonEnrichRestTestCase.testBasicFlowDouble(CommonEnrichRestTestCase.java:163)

But that request is creating an index named my-index, not a monitoring index. I don't know why the response would have information about a monitoring index. Race condition?

[masseyke]

OK it looks like we have two problems here (neither one directly related to this test) --

1. We are incorrectly logging that the .monitoring-es-* indices are not system indices
2. The HeaderWarningAppender is incorrectly putting deprecation logger headers into responses that have nothing to do with the deprecation even.

Here is @jbaiera 's theory on that second one:

ok so, I might be completely wrong on this, may need to check with the distributed team, but looking at the
code when auto create action is executed it submits a task to the master service. That task gets batched up
with any other auto create actions. The tasks are all executed together in the same thread context (a system
context created by the master). Once each task is completed, the cluster state is figured out, and it goes to
publish the new state, the thread context is restored for each listener that is being notified (the original
context). Since it is running in a different context (the system context), it copies over any response headers
from the previous thread context (in this case, the deprecation warning). In the event that the monitoring
index auto create is being run, any other auto create operations that are in the same batch as itself will get
the deprecation warning added to them when they restore their original context.

@elastic/es-distributed might need to take it from here if he's right.

[joegallo]

#84993 and #84885 are actually cases of this, too (that may already be known) -- see also #84885 (comment) and #85469 which addressed some issues with the test class generally, but did not alter anything about the reality of the warning we're getting from the monitoring index.

[ywangd]

Failed again on master intake https://gradle-enterprise.elastic.co/s/5i6a5y6v4eyki

[fcofdez]

Another instance of this https://gradle-enterprise.elastic.co/s/fmqslrb7uvg2e. This time in 7.17