[JEP-210] Integration testing #252

jglick · 2018-10-03T17:39:51Z

My own version of #251. Picks up jenkinsci/workflow-job-plugin#27, #253, #254, and jenkinsci/structs-plugin#41.

…rshalling

jglick · 2018-10-03T17:45:13Z

Actually from a quick glance it looks like StepNodeTest failures are legitimate; looking.

A FlowNode was being sent to GraphListener.Synchronous without ArgumentsAction available.

I got a couple failures locally but am unclear how to interpret them. Probably meaningless unless they are known to pass against released updates. org.junit.ComparisonFailure: expected:<[sleep]> but was:<[// parallel]> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.jenkinsci.plugins.workflow.cps.FlowDurabilityTest.assertIncludesNodes(FlowDurabilityTest.java:664) at org.jenkinsci.plugins.workflow.cps.FlowDurabilityTest$25.evaluate(FlowDurabilityTest.java:887) java.lang.AssertionError: expected:<SUCCESS> but was:<FAILURE> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.jenkinsci.plugins.workflow.cps.FlowDurabilityTest$30.evaluate(FlowDurabilityTest.java:1000)

jglick · 2018-10-03T19:06:22Z

Regarding PersistenceProblemsTest, I am indeed getting some failures locally which might reflect regressions, or might be flaky test issues etc.:

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.junit.Assert.assertFalse(Assert.java:74)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.assertCompletedCleanly(PersistenceProblemsTest.java:77)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:115)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:121)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.lambda$completedNoNodesPersisted$5(PersistenceProblemsTest.java:189)
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.junit.Assert.assertFalse(Assert.java:74)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.assertCompletedCleanly(PersistenceProblemsTest.java:77)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:115)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:121)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.lambda$completedButWrongDoneStatus$7(PersistenceProblemsTest.java:210)
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.junit.Assert.assertFalse(Assert.java:74)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.assertCompletedCleanly(PersistenceProblemsTest.java:77)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:115)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.runBasicBuild(PersistenceProblemsTest.java:121)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.lambda$completedFinalFlowNodeNotPersisted$3(PersistenceProblemsTest.java:164)

I am not sure how seriously to take these since I get a different test failure when I simply use the same Jenkins core baseline and the most recent released deps of the affected plugins:

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.junit.Assert.assertFalse(Assert.java:74)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.assertCompletedCleanly(PersistenceProblemsTest.java:59)
	at org.jenkinsci.plugins.workflow.cps.PersistenceProblemsTest.lambda$inProgressButProgramLoadFailure$19(PersistenceProblemsTest.java:321)

In other words, we have a situation where PCT is not getting regularly run here, and at least some failures are not related to JEP-210.

…se in test output.

If there is something you know ought to happen soon, but you are not exactly sure when, simply wait for it and let the test time out after 3m if it in fact never happens.

…Test.oneOrMoreArgsStepDocs.

svanoort · 2018-10-03T21:57:32Z

@jglick PersistenceProblemsTests should be taken very seriously indeed. For me, if I use workflow-job 2.25 and everything else the same, these tests pass. And unfortunately they still fail with my AsynchronousExecution fix applied (so far), which suggests something deeper is amiss.

jglick · 2018-10-03T22:25:41Z

if I use workflow-job 2.25 and everything else the same, these tests pass

Not for me, at least not quite. See #252 (comment).

they still fail with my AsynchronousExecution fix applied

Passes here after 15f2e7d. Just a bit of flakiness in the tests AFAICT, nothing to do with the upstream PRs except for slight changes in timing.

svanoort

It looks reasonable to me, afaict -- would suggest cutting this into a portion that does not depend on the rewrite (so it's independently releasable) and then merging & releasing for workflow-cps.

dwnusbaum

The fixes here look reasonable to me.

svanoort · 2018-10-04T15:13:10Z

(Note only reason I'm not approving is because it needs to be converted to something backported for merge there)

jglick · 2018-10-04T22:26:23Z

At this point it may as well be merged since it is already picking up some parts of JEP-210.

jglick · 2018-10-04T22:42:54Z

(and it is only a test-scoped dep anyway)

svanoort and others added 5 commits October 2, 2018 17:37

Start validating log rewrite

d3449aa

Fix for some weird IntelliJ dependency resolution issue with jboss ma…

8dbbd91

…rshalling

Update deps

6abdf07

Pull in latest wf-job incremental

108b0e6

Unused import.

9d5965e

jglick requested a review from svanoort October 3, 2018 17:39

jglick added the DO NOT MERGE label Oct 3, 2018

Useless test.

75d0629

jglick added 6 commits October 3, 2018 14:06

Adapting test to changes in output format.

50d298f

Fixing StepNodeTest.

e8953b5

A FlowNode was being sent to GraphListener.Synchronous without ArgumentsAction available.

No longer need to test against an older LTS.

d12db85

Bumping workflow-api dep just in case.

90fa829

Trying to work around CI failures.

0e414a0

jglick mentioned this pull request Oct 3, 2018

[Testing Only] Start validating log rewrite #251

Closed

jglick added 8 commits October 3, 2018 15:24

Trying another tact.

528d973

Different syntax.

8219dac

Guessing that the Windows m2repo was bad too.

a418b3a

Picking up fix of JENKINS-47779, which otherwise produces tons of noi…

7ec8acb

…se in test output.

PersistenceProblemsTest was simply flaky.

15f2e7d

If there is something you know ought to happen soon, but you are not exactly sure when, simply wait for it and let the test time out after 3m if it in fact never happens.

7ec8acb necessitated a POM cleanup too.

6df4322

After jenkinsci/jenkins#882 we must explicitly grant CANCEL.

3acab7e

Pick up jenkinsci/structs-plugin#41 to solve a failure in Snippetizer…

a59116e

…Test.oneOrMoreArgsStepDocs.

jglick changed the title ~~JEP-210 integration testing~~ [JEP-210] Integration testing and minor follow-up fixes Oct 3, 2018

jglick mentioned this pull request Oct 3, 2018

[JENKINS-52189] Ensure GraphListeners are notified of the FlowStartNode too #232

Merged

jglick requested review from abayer, rsandell and dwnusbaum October 3, 2018 20:41

jglick removed the DO NOT MERGE label Oct 3, 2018

jglick added 2 commits October 3, 2018 16:42

Cache hack is done.

3ebb400

Noting all uses of incremental deps.

aa9e5bb

svanoort reviewed Oct 4, 2018

View reviewed changes

dwnusbaum approved these changes Oct 4, 2018

View reviewed changes

This was referenced Oct 4, 2018

Test fixes #253

Merged

Making sure ArgumentsAction is attached in time for GraphListener.Synchronous notification #254

Merged

jglick added 2 commits October 4, 2018 13:57

Merge branch 'master' into logs-JENKINS-38381

a3632d2

Merge branch 'DSL+ArgumentsAction' into logs-JENKINS-38381

4086969

jglick added the DO NOT MERGE label Oct 4, 2018

jglick changed the title ~~[JEP-210] Integration testing and minor follow-up fixes~~ [JEP-210] Integration testing Oct 4, 2018

Merge branch 'master' into logs-JENKINS-38381

739902e

jglick removed the DO NOT MERGE label Oct 4, 2018

jglick merged commit c71db05 into jenkinsci:master Oct 4, 2018

jglick deleted the logs-JENKINS-38381 branch October 4, 2018 23:11

jglick mentioned this pull request Oct 5, 2018

[JENKINS-53917] DescribableModel again treating ChoiceParameterDefinition.choices as an object #255

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JEP-210] Integration testing #252

[JEP-210] Integration testing #252

jglick commented Oct 3, 2018 •

edited

Loading

jglick commented Oct 3, 2018

jglick commented Oct 3, 2018

svanoort commented Oct 3, 2018

jglick commented Oct 3, 2018

svanoort left a comment

dwnusbaum left a comment

svanoort commented Oct 4, 2018

jglick commented Oct 4, 2018

jglick commented Oct 4, 2018

[JEP-210] Integration testing #252

[JEP-210] Integration testing #252

Conversation

jglick commented Oct 3, 2018 • edited Loading

jglick commented Oct 3, 2018

jglick commented Oct 3, 2018

svanoort commented Oct 3, 2018

jglick commented Oct 3, 2018

svanoort left a comment

Choose a reason for hiding this comment

dwnusbaum left a comment

Choose a reason for hiding this comment

svanoort commented Oct 4, 2018

jglick commented Oct 4, 2018

jglick commented Oct 4, 2018

jglick commented Oct 3, 2018 •

edited

Loading