MlMappingsUpgradeIT testMappingsUpgrade

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/5msxbxbgw6bbi/tests/:x-pack:qa:rolling-upgrade:v7.10.0%23upgradedClusterTest/org.elasticsearch.upgrades.MlMappingsUpgradeIT/testMappingsUpgrade#1

Repro line:
./gradlew ':x-pack:qa:rolling-upgrade:v7.10.0#upgradedClusterTest' --tests "org.elasticsearch.upgrades.MlMappingsUpgradeIT.testMappingsUpgrade" -Dtests.seed=D2B94DCD19B71635 -Dtests.security.manager=true -Dtests.locale=et-EE -Dtests.timezone=Africa/Addis_Ababa -Druntime.java=11

Reproduces locally?:
NO

Applicable branches:
master

Failure history:
Since #61834

Failure excerpt:
java.lang.AssertionError: Error the template mappings [.ml-config] and index mappings [.ml-config] are not the same
  Keys in the index missing from the template mapping: [properties.deleting.type]

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[ywangd]

This fails for 7.x as well:

• https://gradle-enterprise.elastic.co/s/fh6spnoncer7e
• https://gradle-enterprise.elastic.co/s/uaqe3aqbtkav2
  And a bunch of others for different bwc versions

They can be reproduced locally with

./gradlew ':x-pack:qa:rolling-upgrade:v7.6.2#upgradedClusterTest' --tests "org.elasticsearch.upgrades.MlMappingsUpgradeIT.testMappingsUpgrade" -Dtests.seed=F2CA8D7ABE77E726 -Dtests.security.manager=true -Dtests.locale=be-BY -Dtests.timezone=America/Maceio -Druntime.java=8

I noticed the test on master is already muted with #61909. I am backporting it to 7.x (#62136)

[hendrikmuhs]

The test is now failing for .ml-notifications-000001 in 7.x, see e.g. https://gradle-enterprise.elastic.co/s/aniecxdcpnd4g

In agreement with the other devs, I did not mute the whole test (we would loose important coverage), but only the part for .ml-notifications-000001.

The issue now only applies to 7.x, because only there upgrade from 7.any to 7.x is possible.

Excerpt:

11:59:48 org.elasticsearch.upgrades.MlMappingsUpgradeIT > testMappingsUpgrade FAILED
11:59:48     java.lang.AssertionError: Error the template mappings [.ml-notifications-000001] and index mappings [.ml-notifications-000001] are not the same
11:59:48     Keys in the template missing from the index mapping: [dynamic, properties.message.fields.raw.type]
11:59:48     Keys in the index missing from the template mapping: [properties.job_id.fields.keyword.ignore_above, properties.job_id.fields.keyword.type, properties.job_type.fields.keyword.ignore_above, properties.job_type.fields.keyword.type, properties.level.fields.keyword.ignore_above, properties.level.fields.keyword.type, properties.message.fields.keyword.ignore_above, properties.message.fields.keyword.type, properties.node_name.fields.keyword.ignore_above, properties.node_name.fields.keyword.type]
11:59:48     Values for key [properties.job_id.type] are different
11:59:48         template value [keyword] String
11:59:48         index value [text] String
11:59:48     Values for key [properties.job_type.type] are different
11:59:48         template value [keyword] String
11:59:48         index value [text] String
11:59:48     Values for key [properties.level.type] are different
11:59:48         template value [keyword] String
11:59:48         index value [text] String
11:59:48     Values for key [properties.node_name.type] are different
11:59:48         template value [keyword] String
11:59:48         index value [text] String
11:59:48     Values for key [properties.timestamp.type] are different
11:59:48         template value [date] String
11:59:48         index value [long] String

[davidkyle]

I've temporarily unmuted to above failure with some extra debug logging. #62342