Use waitForMessage in tests to reduce flakes

[timja]

Hi we're experiencing flakey tests trying to integrate the next bom line,

example:

Expected: a string containing "broken condition"
     but: was "Started
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] waitUntil
[Pipeline] {
[Pipeline] semaphore
[Pipeline] }
Will try again after 0.25 sec
[Pipeline] {
[Pipeline] semaphore
[Pipeline] }
[Pipeline] // waitUntil
[Pipeline] End of Pipeline

@jglick suggested replacing assertLogContains with waitForMessage
jenkinsci/bom#110 (comment)

[jglick]

jenkinsci/workflow-support-plugin#105 (comment)

[jglick]

(possibly now obsolete)

[jglick]

You also need to handle

java.lang.AssertionError: 

Expected: a string containing "caught error"
     but: was "Started
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] End of Pipeline
"
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.jvnet.hudson.test.JenkinsRule.assertLogContains(JenkinsRule.java:1402)
	at org.jenkinsci.plugins.workflow.steps.CatchErrorStepTest.assertCatchError(CatchErrorStepTest.java:241)
	at org.jenkinsci.plugins.workflow.steps.CatchErrorStepTest.optionalMessage(CatchErrorStepTest.java:120)

[bitwiseman]

This will timeout if the message is not found right? So, super slow if it fails. Just want to make sure I understand.

[timja]

It’ll throw when the log stops getting updated https://github.com/jenkinsci/jenkins-test-harness/blob/9d94c7aa2bd13ac13661e531f25291d33c1d4fe0/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L1479

[jglick]

super slow if it fails

At most 3m.

[jglick]

Integrating jenkinsci/jenkins-test-harness#210 would also suffice.

[jglick]

or now, simply parent 4.1+