Add pipeline-model-definition plugin #179

chriskilding · 2020-01-27T15:03:18Z

Add pipeline-model-definition plugin to the BOM. Add a declarative pipeline test to sample-plugin.

jglick · 2020-01-28T00:24:40Z

bom-latest/pom.xml

+            </dependency>
+            <dependency>
+                <groupId>org.jenkins-ci.plugins</groupId>
+                <artifactId>docker-workflow</artifactId>


I would like to do JENKINS-58732 BTW.

jenkinsci/pipeline-model-definition-plugin#373 specifically

That's ok - and indeed would let us knock a few transitive dependencies off what is quite a big change here. We can revisit this when that PR is merged

Also the split would reduce the number of tests run in pipeline-model-definition, which is currently gigantic, even slower than workflow-cps, thus becoming the bottleneck for the bom parallel run.

I noticed that jenkinsci/pipeline-model-definition-plugin#373 was merged earlier today - let me know when the next version is released, and I'll re-work this PR.

Actually it is going to take longer than that, for purposes of this PR. See rollout plan in jenkinsci/docker-workflow-plugin#202. I do not think you need to block this PR on it.

I was hoping to avoid wrangling the icon-shim-plugin's inconsistent tags, which I believe docker-workflow was pulling in, by waiting. Is that unblocked at this point, or is is it still present?

Afraid I know nothing about this icon-shim issue.

sample-plugin/src/test/java/io/jenkins/tools/bom/sample/DeclarativePipelineTest.java

chriskilding · 2020-01-28T11:02:06Z

While we're at it, any idea why the PCT step fails to check out the icon-shim plugin from source? I looked in its Git repo and there certainly seems to be a version 1.0.3.

error: pathspec 'icon-shim-1.0.3' did not match any file(s) known to git

jglick · 2020-01-28T14:22:36Z

any idea why the PCT step fails to check out the icon-shim plugin from source?

Not offhand. Something nonstandard with the metadata or Git layout perhaps.

chriskilding · 2020-01-29T10:25:01Z

Looks like the icon-shim plugin is only being pulled in by docker-commons, so if you do the other PR first then this problem will also go away. (Well until we decide to put icon-shim in the POM for other reasons.)

+- org.jenkins-ci.plugins:docker-workflow:jar
|  \- org.jenkins-ci.plugins:docker-commons:jar
|     +- org.jenkins-ci.plugins:authentication-tokens:jar
|     \- org.jenkins-ci.plugins.icon-shim:icon-shim:jar

chriskilding · 2020-01-29T11:04:48Z

But if we were interested in fixing it, the answer is as you said, non-standard Git layout.

Most plugins either have a tag naming convention of [plugin]-[version] or [version]. But the icon-shim plugin is non-standard, so the PCT checkout won't work:

$ git tag
icon-shim-1.0.0
icon-shim-pom-1.0.1
icon-shim-pom-1.0.2
icon-shim-pom-1.0.3
icon-shim-pom-1.0.4
...

jetersen · 2020-01-29T12:59:50Z

This is how we fixed in configuration-as-code parent pom: https://github.com/jenkinsci/configuration-as-code-plugin/blob/b68e82e7eb859519975f29bc5e3b5ca77a4ef2ce/pom.xml#L27

jglick · 2020-03-06T18:35:52Z

the icon-shim plugin is non-standard, so the PCT checkout won't work

Supposed to have been taken care of by jenkinsci/plugin-compat-tester#181 without any tricks.

chriskilding · 2020-03-10T12:59:10Z

After some more dependency twiddling it appears docker-commons imports some more troublesome dependencies - notably authentication-tokens which fails PCT on its InjectedTest:

[ERROR] InjectedTest.initializationError(InjectedTest)
[ERROR]   Run 1: InjectedTest.suite:15 » NoClassDefFound hudson/slaves/CommandLauncher

That plugin hasn't been updated since last year and still uses parent 3.42 / Jenkins 2.60.3, so bringing it up to date may not be trivial. Is it worth it? Would it be easier to wait for the Docker dependencies to be removed from pipeline-model-definition.

jglick · 2020-03-10T21:52:24Z

If there is an error in authentication-tokens it should be simple to solve. Just updating the parent POM will generally suffice to pull in jenkinsci/jenkins-test-harness#79.

jglick · 2020-03-10T21:53:05Z

wait for the Docker dependencies to be removed from pipeline-model-definition

Or jenkinsci/pipeline-model-definition-plugin#383 might suffice.

chriskilding · 2020-03-11T11:35:09Z

Have opened jenkinsci/authentication-tokens-plugin#7 for this

chriskilding · 2020-05-06T20:19:41Z

It appears that even when the docker workflow dep is marked optional, it's still dragged in to the analysis with its own dependencies. I've filed jenkinsci/pipeline-model-definition-plugin#391 to do the honours

jglick · 2020-06-23T18:59:38Z

jenkinsci/authentication-tokens-plugin#7 is released as 1.4.

jetersen · 2020-06-24T09:50:25Z

Currently has some conflicts

chriskilding · 2020-06-24T09:51:07Z

also seems to have lost the CI trigger. I tried unmarking as draft to see if that would make a difference.

jetersen · 2020-06-24T10:16:32Z

it won't trigger CI with conflicts

chriskilding · 2020-06-24T13:47:35Z

Looks like a lot more tests are able to pass now.

The next blocker is java.lang.NoClassDefFoundError: hudson/slaves/CommandLauncher in the pipeline-stage-step PCT. The curious thing is, command-launcher-plugin (which contains that class) is available on the classpath, at least as it appears in my local IDE checkout.

timja · 2020-06-24T14:57:45Z

Weird it seems to be included:

https://github.com/jenkinsci/command-launcher-plugin/blob/master/src/main/java/hudson/slaves/CommandLauncher.java

https://github.com/chriskilding/bom/blob/add-pipeline-model-definition-plugin/bom-latest/pom.xml#L136

Local IDE checkout doesn't really represent what's actually run, best to run the PCT locally with the local-test script or my grabbing the command out that's used in the Jenkins job.

Basically PCT does some trickery...

bom-latest/pom.xml

jglick

Getting close!

bom-latest/pom.xml

sample-plugin/pom.xml

bom-latest/pom.xml

jglick · 2020-06-24T22:41:25Z

sample-plugin/src/test/java/io/jenkins/tools/bom/sample/DeclarativePipelineTest.java

+                "  stages {",
+                "    stage('Example') {",
+                "      steps {",
+                "        echo 'Hello world'",


May as well

Suggested change

" echo 'Hello world'",

" example(x: 'some value')",

and for that matter just move this to ExampleStepTest.

chriskilding · 2020-06-25T10:22:58Z

More things working now.

The basics test in org.jenkinsci.plugins.workflow.support.steps.StageStepTest still fails, which I think is the next bottleneck in the graph.

chriskilding · 2020-06-25T11:04:39Z

FWIW the pipeline that the test constructs is unusual in that the stages don't wrap their contents, and the build steps (chiefly echo) are not wrapped in step {} either:

stage(name: 'A', concurrency: 2);
echo('in A');
semaphore('B');
stage(name: 'B', concurrency: 1);
echo('in B');
semaphore('X');
echo('done')

I believe I saw a warning recently that this kind of stage usage might be deprecated, though can't recall where exactly I saw this. And Jesse has just pushed the first pipeline-stage-step release in 2/3 years. Is the test no longer using valid pipeline DSL?

#179 (comment) Unfortunately TEST_FAILURES only asserts that surefire:test did not succeed, but does not verify that it even ran to begin with: https://github.com/jenkinsci/plugin-compat-tester/blob/c502ba1cc6ebf39b9525e914fc537c38918ed535/plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java#L322-L323

…odel-definition-plugin

jglick · 2020-07-20T16:15:30Z

I would again propose #179 (comment)

chriskilding · 2020-07-20T16:25:42Z

I've incrementally removed dependencies as we've gone along, so we're looking at something near the minimal set already:

pipeline-model-definition (and its multimodule artifacts)

Plus three that the sample project requires to be present in order to build - if omitted we get the "Plugin dependency on [plugin] is not from dependencyManagement" error:

jackson2-api
workflow-multibranch
pipeline-stage-step

jglick · 2020-07-20T17:03:13Z

jackson2-api, workflow-multibranch, and pipeline-stage-step could all be split off into their own PRs. (Cherry-picking b489a73, assuming it works as intended.)

jglick · 2020-07-20T17:04:04Z

Time permitting, we should also be looking to add some other plugins mentioned in jenkinsci/workflow-aggregator-plugin#36.

jglick · 2020-07-20T19:01:51Z

I am afraid there seem to be a number of plugins already in the BOM which did not pass PCT, something that needs to be fixed before adding new plugins. Will file a new PR to track this.

jenkinsci#179 (comment) Unfortunately TEST_FAILURES only asserts that surefire:test did not succeed, but does not verify that it even ran to begin with: https://github.com/jenkinsci/plugin-compat-tester/blob/c502ba1cc6ebf39b9525e914fc537c38918ed535/plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java#L322-L323 (cherry picked from commit b489a73)

…ill leave no test files behind; really better for PCT to set -Dmaven.test.failure.ignore (cherry picked from commit 7c271c3)

chriskilding · 2020-07-21T09:41:13Z

workflow-multibranch-plugin proposed in #280

chriskilding · 2020-07-21T10:33:09Z

pipeline-stage-step proposed in #281

chriskilding · 2020-07-21T11:08:05Z

jackson2-api proposed in #282

jetersen · 2020-09-30T11:41:59Z

Are we ready to move forward with this one?

chriskilding · 2020-09-30T11:50:55Z

gotta rebase and tidy up, but yep

# Conflicts: # bom-latest/pom.xml

jetersen · 2020-09-30T12:47:47Z

We are squash merging so no need for rebase 😁

chriskilding · 2020-09-30T13:42:03Z

Hmm, got the same failure on test_submitters() in InputStepTest in pipeline-input-step-plugin (403 login forbidden) as I did in the workflow-multibranch PR. It passed after a rebuild in that PR, so suspect there might be some flakiness there.

chriskilding · 2020-09-30T18:15:44Z

Ok definitely a couple of flaky tests in there, selfTestLibraries in SCMRetrieverTest timed out and test_submitters in InputStepTest couldn't log in.

jetersen

Changes look good to me however the flaky tests are worrisome 😓

jglick · 2020-09-30T19:32:34Z

a couple of flaky tests in there

Both of those sound familiar from other PRs. Will have to dig into them at some point. Likely unrelated to this PR.

chriskilding mentioned this pull request Jan 27, 2020

Support: Depend on the Jenkins tools BOM jenkinsci/aws-secrets-manager-credentials-provider-plugin#23

Merged

4 tasks

jglick reviewed Jan 28, 2020

View reviewed changes

sample-plugin/src/test/java/io/jenkins/tools/bom/sample/DeclarativePipelineTest.java Outdated Show resolved Hide resolved

jglick mentioned this pull request Jan 28, 2020

Bump docker-commons from 1.14 to 1.16 jenkinsci/kubernetes-plugin#670

Closed

jglick mentioned this pull request Apr 9, 2020

Introduce bom and bump minimum required version jenkinsci/workflow-cps-global-lib-plugin#95

Merged

chriskilding marked this pull request as ready for review June 24, 2020 09:48

chriskilding added 2 commits June 24, 2020 11:21

Add pipeline-model-definition plugin

a8eb438

Bump pipeline-model-definition to 1.7.0, remove docker deps

1c7f014

jglick requested changes Jun 24, 2020

View reviewed changes

bom-latest/pom.xml Outdated Show resolved Hide resolved

jglick reviewed Jun 24, 2020

View reviewed changes

fixes

02bb058

jglick added 2 commits July 20, 2020 12:12

Merge branch 'master' of github.com:jenkinsci/bom into add-pipeline-m…

bcbe59a

…odel-definition-plugin

jglick mentioned this pull request Jul 20, 2020

Update Pipeline: Aggregator to Jenkins 2.176.4 + Use Plugin BOM to define plugin versions jenkinsci/workflow-aggregator-plugin#36

Closed

b489a73 was improperly marking plugins failed due to known test failures

91fdadd

jglick mentioned this pull request Jul 20, 2020

Detect overall PCT failures #277

Merged

Still not right; if Surefire starts to run but then the VM crashes, w…

97a6938

…ill leave no test files behind; really better for PCT to set -Dmaven.test.failure.ignore (cherry picked from commit 7c271c3)

Merge branch 'master' into add-pipeline-model-definition-plugin

0efb311

# Conflicts: # bom-latest/pom.xml

chriskilding added 3 commits September 30, 2020 15:23

Use ExampleStep in declarative pipeline test, reinstate failfast

862b01e

Bump pipeline-model-definition to 1.7.2

32499c4

Rebuild

bd36b6b

chriskilding marked this pull request as ready for review September 30, 2020 18:12

jetersen approved these changes Sep 30, 2020

View reviewed changes

jglick approved these changes Sep 30, 2020

View reviewed changes

jglick added the enhancement New feature or request label Sep 30, 2020

jglick merged commit 47dc496 into jenkinsci:master Sep 30, 2020

Add pipeline-model-definition plugin #179

Add pipeline-model-definition plugin #179

Conversation

chriskilding commented Jan 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriskilding commented Jan 28, 2020

jglick commented Jan 28, 2020

chriskilding commented Jan 29, 2020 • edited

chriskilding commented Jan 29, 2020

jetersen commented Jan 29, 2020

jglick commented Mar 6, 2020

chriskilding commented Mar 10, 2020

jglick commented Mar 10, 2020

jglick commented Mar 10, 2020

chriskilding commented Mar 11, 2020

chriskilding commented May 6, 2020

jglick commented Jun 23, 2020

jetersen commented Jun 24, 2020

chriskilding commented Jun 24, 2020

jetersen commented Jun 24, 2020

chriskilding commented Jun 24, 2020

timja commented Jun 24, 2020

jglick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriskilding commented Jun 25, 2020

chriskilding commented Jun 25, 2020

jglick commented Jul 20, 2020

chriskilding commented Jul 20, 2020 • edited

jglick commented Jul 20, 2020

jglick commented Jul 20, 2020

jglick commented Jul 20, 2020

chriskilding commented Jul 21, 2020

chriskilding commented Jul 21, 2020

chriskilding commented Jul 21, 2020

jetersen commented Sep 30, 2020

chriskilding commented Sep 30, 2020 • edited

jetersen commented Sep 30, 2020 • edited

chriskilding commented Sep 30, 2020

chriskilding commented Sep 30, 2020

jetersen left a comment

Choose a reason for hiding this comment

jglick commented Sep 30, 2020

chriskilding commented Jan 27, 2020 •

edited

chriskilding commented Jan 29, 2020 •

edited

chriskilding commented Jul 20, 2020 •

edited

chriskilding commented Sep 30, 2020 •

edited

jetersen commented Sep 30, 2020 •

edited