Run more search Rest tests in a CCS setup

[cbuescher]

Currently we only test a small subset of cross cluster search in rest tests in the 'multi-cluster-search' qa module. In order to increase test coverage for basic CCS setups, this change adds a new qa modula that uses a subset of existing search rest tests tests and runs them in a CCS scenario.

Relates to #84481

[elasticmachine]

Pinging @elastic/clients-team (Team:Clients)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

Some of the ccs compatible endpoints (like e.g. search_template) have their yaml rest tests under "modules" or x-pack (terms_enum), I don't yet if/how we can run them from here, so this should probably be a follow up task.

[cbuescher]

I added the "indices.resolve_index" api test to the covered subset of rest tests, also fixed a small issue that led to some test assertions being ignored where they shouldn't have, which led to a few overlooked places in "field_caps" where the testrunner still needed to rewrite the expected matches. Locally 751 tests are passing now, with only the blacklisted ones skipped atm.

[javanna]

I left a couple of questions, this looks right to me though, thanks for working on this!

[cbuescher]

@javanna I pushed an update adressing your comments. I can also add the "_search_shard" endpoint to the APIs we route to the local search cluster instead of the remote holding the data if that API is supposed to support CCS and we want to include it in the testing here.

[javanna]

@cbuescher I do not think that the search shards API supports cross-cluster calls, does it?

[cbuescher]

No, I also think it doesn't. So nothing to do there then.

[javanna]

LGTM

[javanna]

looks like the reasons why these are in the blacklist are good, so maybe the TODO can be removed?

[cbuescher]

@javanna thanks for the heads up. I think I can include "_terms_enum" and "async_search" yaml tests from xpack before merging, might need another few commits on this PR though.

[cbuescher]

@javanna just fyi I added the "asych_search" tests from x-pack to be included in the suites we are running here. Adding "terms_enum" in a similar way proved to be difficult since the basic tests under "x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/terms_enum/10_basic.yml" require a security setup which we don't have here.
But then again there are rest tests covering "terms_enum" and CCS including security under "x-pack/qa/multi-cluster-search-security/src/test/resources/rest-api-spec/test/multi_cluster/120_terms_enum.yml" so I think we are covered here.