[CI] SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.test {yaml=watcher/usage/10_basic/Test watcher usage stats output}

[pugnascotia]

Build scan:

https://gradle-enterprise.elastic.co/s/tvd22xejovpkq

Repro line:

./gradlew ':x-pack:plugin:watcher:qa:with-security:yamlRestTest' --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.test {yaml=watcher/usage/10_basic/Test watcher usage stats output}" -Dtests.seed=8712769CB44639EF -Dtests.security.manager=true -Dtests.locale=fr-LU -Dtests.timezone=Asia/Krasnoyarsk -Druntime.java=8 -Dtests.fips.enabled=true

Reproduces locally?:

No

Applicable branches:

7.x

Failure history:

Failure excerpt:

org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT > test {yaml=watcher/usage/10_basic/Test watcher usage stats output} FAILED
    java.lang.AssertionError: Failure at [watcher/usage/10_basic:48]: field [watcher.count.active] is not greater than [$watch_count_active]
    Expected: a value greater than <1>
         but: <1> was equal to <1>
        at __randomizedtesting.SeedInfo.seed([8712769CB44639EF:F4649461ABA5417]:0)
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:414)
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:391)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[nik9000]

Looks like I've found another: https://gradle-enterprise.elastic.co/s/tbwnizq3cwe36/tests/:x-pack:plugin:watcher:qa:with-security:yamlRestTest/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT/test%20%7Byaml=watcher%2Fusage%2F10_basic%2FTest%20watcher%20usage%20stats%20output%7D#1

[jakelandis]

Test was never muted, and no comments for over 6m and no failures in the past 28 days. Closing. Feel free to reopen if it shows up again.

[abdonpijpelink]

Happened again: https://gradle-enterprise.elastic.co/s/wbcyvqryeyzdg/tests/:x-pack:plugin:watcher:qa:with-security:yamlRestTest/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT/test%20%7Byaml=watcher%2Fusage%2F10_basic%2FTest%20watcher%20usage%20stats%20output%7D?expanded-stacktrace=WyIwIiwiMC0xIl0&top-execution=1

@jakelandis Should I reopen this issue?

[cbuescher]

Another one today that looks like this. Doesn't reproduce locally unfortunately. On 8.6 branch:

https://gradle-enterprise.elastic.co/s/h7ukuyd3kkkpy

java.lang.AssertionError: Failure at [watcher/usage/10_basic:48]: field [watcher.count.active] is not greater than [$watch_count_active]
Expected: a value greater than <1>
     but: <1> was equal to <1>
	at __randomizedtesting.SeedInfo.seed([21674259D2BA5B17:A9337D837C4636EF]:0)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:520)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:480)

[craigtaverner]

Failed again today, so twice in the last week: https://gradle-enterprise.elastic.co/s/zedmienwnfpgu/tests/:x-pack:plugin:watcher:qa:with-security:yamlRestTest/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT/test%20%7Byaml=watcher%2Fusage%2F10_basic%2FTest%20watcher%20usage%20stats%20output%7D?top-execution=1

[salvatore-campagna]

Another failure: https://gradle-enterprise.elastic.co/s/jof7gycumcqnw/tests/:x-pack:plugin:watcher:qa:with-security:yamlRestTest/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT/test%20%7Byaml=watcher%2Fusage%2F10_basic%2FTest%20watcher%20usage%20stats%20output%7D

I will mute it.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

I closed #104495 in favor of this one because it's the same yaml test, but don't miss that there's some valuable investigation on that ticket, or that we have seen this failure via more than just one yaml test runner, specifically:

 ./gradlew ':x-pack:plugin:watcher:qa:rest:yamlRestTestV7CompatTest' --tests "org.elasticsearch.smoketest.WatcherYamlRestIT.test {p0=watcher/usage/10_basic/Test watcher usage stats output}"

 ./gradlew ':x-pack:plugin:watcher:qa:with-security:yamlRestTest' --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.test {yaml=watcher/usage/10_basic/Test watcher usage stats output}"

I'm assuming we'd see it off this one occasionally, too:

./gradlew ':x-pack:plugin:watcher:qa:rest:yamlRestTest' --tests "org.elasticsearch.smoketest.WatcherYamlRestIT.test {p0=watcher/usage/10_basic/Test watcher usage stats output}"

[joegallo]

Same story on closing #93039 in favor of this one, too.

[joegallo]

I merged #106141 to make it a little easier to understand why this test fails when it fails.

[joegallo]

#106192 has some of the fallout from having merged #106141, which I've since reverted.

[joegallo]

AH HA! I've been looking at the fallout from having merged #106141, and here's something. I suspect it's a case of failing on the shutting down side of things. And here's why I think that:

The failures come in pairs! There's a test that's failing on the shutdown side and leaving a bad state in memory, and then there's a test that fails (with ten retries) on the setup side. But they fail in pairs.

[masseyke]

Closing this as a duplicate of #112189 since that is the one listed in muted_tests.yml (and so it will get reopened by test automation if I close it).