[CI] WatcherConcreteIndexTests testCanUseAnyConcreteIndexName failure

[davidkyle]

watch was triggered on [0] schedulers, expected [1]
Expected: a value equal to or greater than <1L>
     but: <0L> was less than <1L>

at __randomizedtesting.SeedInfo.seed([17F8D0FFA17B00BE:302289683F89D2CB]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase$TimeWarp.trigger(AbstractWatcherIntegrationTestCase.java:600)
at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase$TimeWarp.trigger(AbstractWatcherIntegrationTestCase.java:585)
at org.elasticsearch.xpack.watcher.WatcherConcreteIndexTests.testCanUseAnyConcreteIndexName(WatcherConcreteIndexTests.java:47)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/760/console
https://gradle-enterprise.elastic.co/s/eaphkxcjw4fea

Does not reproduce:

./gradlew ':x-pack:plugin:watcher:test' --tests "org.elasticsearch.xpack.watcher.WatcherConcreteIndexTests.testCanUseAnyConcreteIndexName" \
  -Dtests.seed=17F8D0FFA17B00BE \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=es-PY \
  -Dtests.timezone=EAT \
  -Dcompiler.java=13

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Watcher)

[martijnvg]

This failed because the watch didn't trigger, which was caused by the fact that the scheduler was paused:

[2020-01-06T16:08:15,403][INFO ][o.e.x.w.t.ScheduleTriggerEngineMock] [testCanUseAnyConcreteIndexName] not executing watch [mywatch] on this scheduler because it is paused

The reason the scheduler was paused:

[2020-01-06T16:08:15,349][INFO ][o.e.x.w.WatcherService   ] [node_s1] paused watch execution, reason [no local watcher shards found], cancelled [0] queued tasks

[martijnvg]

This test replaces the watch index after watcher got started. This triggers watches being reloaded and while this happens the trigger engine is paused, which disallows watches from being triggered. At this time there are no watches in the .watches index and I think this is just unlucky timing.

Reloading of watches happens in the background and the watch state can be started when that happens. For normal schedule trigger engines this is not an issue, because watches that are meant to be triggered are triggered when the engine triggers the next time. However for the mock scheduled trigger engine this is different, because watches are triggered programatically and there is no retry in this.

I think just adding timeWarp().trigger("mywatch"); inside in assertBusy(...) is the right fix here. If it fails because the mock schedule trigger engine is paused then the test will try again. In the mean time the the watches can be reloaded, which then resumes the mock scheduled trigger engine.