Add mixed mode BWC testing support for test-specific plugins

[ChrisHegarty]

This issues been filed to examine how best to support the inference-service-test plugin in ES|QL mixed version testing.

The ES|QL CSV and REST tests run with a variety of modes (see x-pack/plugin/esql/qa/server):

1. single-node
2. multi-node
3. mixed-cluster
4. multi-clusters

No.1 and no.2 run the same ES version on all nodes. No.3 and no.4 run with mixed ES versions, either within the same cluster or across clusters.

The issue occurs only with no.3 and no.4 where the BWC infrastructure is not able to install the interference service test-specific plugin into the older nodes. When we try it, we get the following error:

> Task :x-pack:plugin:esql:qa:server:mixed-cluster:v8.17.0#javaRestTest FAILED

MixedClusterEsqlSpecIT > classMethod FAILED
    java.lang.RuntimeException: An error occurred orchestrating test cluster.
        at __randomizedtesting.SeedInfo.seed([27B9DA09AA60FCA0]:0)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:270)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:275)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:78)
        at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:46)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1570)

        Caused by:
        java.lang.IllegalArgumentException: Can't run executable: `/Users/chegar/git/elasticsearch-lucene_snapshot/x-pack/plugin/esql/qa/server/mixed-cluster/build/testrun/v8.17.0_javaRestTest/temp/test-cluster8788876539987453686/test-cluster-2/distro/bin/elasticsearch-plugin` does not exist.
            at org.elasticsearch.test.cluster.util.ProcessUtils.exec(ProcessUtils.java:54)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.runToolScript(AbstractLocalClusterFactory.java:784)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.installPlugins(AbstractLocalClusterFactory.java:633)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:159)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:78)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
            at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
            at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
            at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
            at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
            at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
            at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
            at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
            at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:785)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:78)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:276)
            at java.base/java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
            at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1491)
            at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2073)
            at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2035)
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

@ChrisHegarty do you have a branch/open PR I can pull to repro the error locally?

[ChrisHegarty]

@ChrisHegarty do you have a branch/open PR I can pull to repro the error locally?

This commit enables multi-versioning testing in ES|QL, so reproduces the issue:
ChrisHegarty@82a6381

Then just run, e.g. ./gradlew :x-pack:plugin:esql:qa:server:mixed-cluster:v8.17.0#javaRestTest, to get output

...
MixedClusterEsqlSpecIT > classMethod FAILED
    java.lang.RuntimeException: An error occurred orchestrating test cluster.
        at __randomizedtesting.SeedInfo.seed([CF3C42D5E03B6ED4]:0)
...

Run with ./gradlew ... --info to see the logs, e.g.

...
[2024-10-25T04:20:10,409][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Installing plugins [inference-service-test] into node 'test-cluster-2
    [2024-10-25T04:20:10,410][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-2]] Installing plugins [inference-service-test] into node 'test-cluster-0
    [2024-10-25T04:20:10,452][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Installing plugins [inference-service-test] into node 'test-cluster-1
    [2024-10-25T04:20:10,452][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-3]] Installing plugins [inference-service-test] into node 'test-cluster-3
    [2024-10-25T04:20:11,790][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:11,790][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:11,848][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:11,849][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:13,828][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installed inference-service-test
    [2024-10-25T04:20:13,828][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Please restart Elasticsearch to activate any plugins installed
    [2024-10-25T04:20:13,841][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installed inference-service-test
    [2024-10-25T04:20:13,841][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Please restart Elasticsearch to activate any plugins installed
    [2024-10-25T04:20:15,427][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:15,427][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:15,428][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:15,428][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:15,443][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Failed installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
    [2024-10-25T04:20:15,443][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Failed installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
...

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)