Making sure that watcher history indices are green before querying th…

…em (#95642)

My theory is that the failure at #95077 was caused by #94133. It looks
like the test is failing because the watcher history index exists, but
not all shards have been allocated. Previously the bulk processor used
to load watcher history was blocking. So if the watch completed you were
guaranteed that the watcher history shards had been allocated because it
didn't return until the watcher history documents had been written. The
new bulk processor works asynchrnously, and doesn't block watcher. This
change waits until the watcher history indices are green (all shards
allocated) before querying them.

Closes #95077

x-pack/plugin/watcher/src/internalClusterTest/java/org/elasticsearch/xpack/watcher/test/integration/ExecutionVarsIntegrationTests.java

@@ -11,6 +11,7 @@
 import org.elasticsearch.plugins.Plugin;
 import org.elasticsearch.protocol.xpack.watcher.PutWatchResponse;
 import org.elasticsearch.xcontent.ObjectPath;
+import org.elasticsearch.xpack.core.watcher.history.HistoryStoreField;
 import org.elasticsearch.xpack.core.watcher.support.xcontent.XContentSource;
 import org.elasticsearch.xpack.core.watcher.transport.actions.execute.ExecuteWatchRequestBuilder;
 import org.elasticsearch.xpack.core.watcher.transport.actions.execute.ExecuteWatchResponse;
@@ -110,7 +111,6 @@ protected Map<String, Function<Map<String, Object>, Object>> pluginScripts() {
         }
     }
 
-    @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/95077")
     public void testVars() throws Exception {
         PutWatchResponse putWatchResponse = new PutWatchRequestBuilder(client()).setId(watchId)
             .setSource(
@@ -136,6 +136,7 @@ public void testVars() throws Exception {
         flush();
         assertBusy(() -> {
             refresh();
+            ensureGreen(HistoryStoreField.DATA_STREAM);
             SearchResponse searchResponse = searchWatchRecords(builder -> {
                 // defaults to match all;
             });