Avoid race condition in ILMHistorySotre

[probakowski]

This change modifies ILMHistoryStore to always apply correct settings and mappings,
even if template is deleted and not yet recreated. This ensures that ILM history index
is correctly managed by ILM and also fixes flaky history tests that were prone to
triggering this race.

This commit also refactors and simplifies ILM history tests.

Closes #50353 and #52853

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[probakowski]

@elasticmachine run elasticsearch-ci/bwc

[probakowski]

@elasticmachine update branch

[andreidan]

Thanks for fixing this @probakowski! LGTM

[dakrone]

This change looks good to be overall, but I think these tests will still end up failing due to the race condition between index creation via _bulk requests and manually being checked in the history store.

I think in order to fully fix them we need to add the settings to make the history store as non-async as possible when indexing, so we can prevent the alias/concrete-index issues that were causing failures in the past.

We can definitely merge this (thanks for doing this!), but I think we may still end up with some failures down the road.

[probakowski]

Thanks @andreidan and @dakrone, I'll merge it as soon as build passes.
I think right now window for race is pretty small. Only possibility I see is that something deletes both history index and template between check in ILMHistoryStore#ensureHistoryIndex and actual indexing AND template is not recreated in time. Although not impossible it would be pretty hard to hit.
For what it's worth I've run 3000 back-to-back tests without single failure.

[probakowski]

@elasticmachine run elasticsearch-ci/bwc and run elasticsearch-ci/default-distro

[probakowski]

@elasticmachine update branch