[FIXED JENKINS-41334] Adding parallel stages. #152

abayer · 2017-04-13T17:38:49Z

JENKINS issue(s):
- JENKINS-41334
Description:
- This implements the JENKINS-41334 RFC, adding support for one level of nested parallel stages. The backend implementation does not actually use the stage step at this point, just parallel branches, but the user experience with configuration, etc is the same.
Documentation changes:
- Will be needed, obviously.
Users/aliases to notify:
- @reviewbybees
- @rsandell
- @michaelneale
- @i386
- @kzantow
- @HRMPW

ghost · 2017-04-13T17:39:21Z

This pull request originates from a CloudBees employee. At CloudBees, we require that all pull requests be reviewed by other CloudBees employees before we seek to have the change accepted. If you want to learn more about our process please see this explanation.

rsandell · 2017-04-18T09:53:44Z

...on/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy

@@ -80,35 +80,10 @@ public class ModelInterpreter implements Serializable {
                                    for (int i = 0; i < root.stages.getStages().size(); i++) {
                                        Stage thisStage = root.stages.getStages().get(i)
                                        try {
+                                            // NOTE that this will eventually be moved into evaluateStage, once BO can
+                                            // handle visualizing stage-inside-parallel.


By "this" I assume you mean script.stage?

rsandell · 2017-04-18T10:02:26Z

...odel-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java

-                } else {
-                    result.append(',');
+        if (branches.isEmpty() && parallelStages != null) {
+            result.append("parallelStages {\n");


This does not follow the RFC.
And revenge of the camelCase :) Why not go with parallel as the RFC states? Confusion? There are plenty of synonyms or other contextually applickable things to parallel to choose from if you don't want to use that word; threads, simultaneous, concurrent. But I personally think that reusing the word stages is best. It self explains in a way that it is sub stages to this stage although that could add to the meaning as it is not apparent they would run in parallel so it would change the implementation a bit since probably an attribute should then be added to declare them to be executed in parallel.

And in that last case, why not allow all stages to potentially be run in parallel, even the outer ones :)

Compromise, basically. I believe that parallel is too overloaded for sure, the synonyms for parallel are completely new terms for Pipeline and so I'd prefer to avoid them (both to prevent the need for learning new terminology and, more importantly, to keep those terms free for possible later use!), and just stages is too overloaded as well.

With this, stages ends up meaning "execute these stages linearly" while parallel means, as in Scripted, "execute these arbitrary blocks in parallel", and the new parallelStages means, fairly explicitly, "execute these stages concurrently". While I don't have any further use for those distinctions now, I can envision some possible use cases in the future.

rsandell · 2017-04-18T10:13:14Z

...on/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy

+                    Utils.markStageSkippedForFailure(thisStage.name)
+                } else if (skipUnstable(root.options)) {
+                    Utils.logToTaskListener("Stage '${thisStage.name}' skipped due to earlier stage(s) marking the build as unstable")
+                    Utils.markStageSkippedForUnstable(thisStage.name)


Will this work for nested stages since there is no actual stage to mark?

Good question! Needs a test.

rsandell · 2017-04-18T10:19:20Z

...on/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy

+            }
+
+            if (firstError != null) {
+                throw firstError


This looks a bit iffy whether nested stages will get their post run or not. Some tests should be added that checks for exceptions in various parts of a nested stage tree.

Yyyup, this is buggy. Working on it.

So the nested stages were doing post fine, but the parent stage was not. So...juggling things around. Hooray.

bitwiseman

parallelStages seems reasonable to me. We seem to be shooting for a bit more verbose but clear names, like customWorkspace.

But it is probably worth getting buy off from more people (and maybe updating the RFC?) and addressing questions like: can parallelStages be used instead of stages at the top of pipeline? Design-wise, I'd expect the answer to be yes.

The first thing someone is going to do when they see this:

pipeline {
    stages {   
         stage("foo") {
             paralllelStages {
                 stage("first") {
                     parallelStages {
                         stage("first-and-one") {

They'll want to do either these two things:

pipeline {
    parallelStages {  // this
         stage("foo") {
           agent { label 'someLabel' } // and this 
             stages { // and this
                 stage("first") {
                     parallelStages {
                         stage("first-and-one") {

And none of those three is supported in this implementation, right?

I'd almost rather push this out and do this in a way that is consistent with the stage behavior elsewhere.

abayer · 2017-04-21T14:38:22Z

RFC already was updated. =)

Lemme think on the top-level idea. Not sure about that.

bitwiseman · 2017-04-21T16:45:42Z

@abayer - making all stage calls have the same syntax and I'd describe it less as "top-level" more more "stages and parallelStages should be syntactically interchangeable - anywhere I can put one, I should be able to put the other". Otherwise, it will confuse and annoy people.

HRMPW · 2017-04-21T16:56:41Z

@bitwiseman I think it will get more confusing trying to nest stages and parallelStages. If they are the same you can nest parallelStages inside stages then it follows that you can nest stages inside stages and parallelStages inside parallelStages.

The alternative could be to make them both top-level only:

pipeline {
  agent any
  stages {
    stage ("A") { }
  }
  parallelStages {
    stage("P1") {}
    stage("P2") {} 
 }
 stages { 
    stage("B") {}
}

The big disadvantage I see with treating stages and parallelStages as equivalents is that you lose the wrapping stage currently proposed around parallelStages. This lets you set environment, agent, when, and post for the entire parallelStages block. You completely lose that if you by promoting parallelStages.

abayer · 2017-04-21T17:11:02Z

Oh, we're definitely not supporting multiple stages at the top level in any form. I'm still leaning hard towards just allowing parallelStages within a stage and having that syntax disconnect since it is meaningful - stages inside parallelStages themselves can't contain parallelStages, so there is a difference. Plus, Blue Ocean is still going to want the actual parallel "stages" to be nested within a parent stage, so we'd have to create a synthetic stage for top-level parallel stages.

HRMPW · 2017-04-21T17:18:27Z

I agree that we should leave this inside a stage but to @bitwiseman 's point I do think calling it parallelStages does imply that it is the same as stages. Alternatives:

parallel - might be overloaded but it is simple
furcate - too obscure?
split
children

bitwiseman · 2017-04-21T18:29:38Z

stages inside parallelStages themselves can't contain parallelStages

There's a test that contains exactly that. ... Oh, I see, it's an error case. Sigh.

@HRMPW
I'm with Andrew on having a single top-level entry. I'd like to allow parallelStages or stages, but that may have been a misunderstanding on my part.

@abayer
What Blue Ocean wants is not sufficient justification (IMHO) for how we design the declarative syntax. 😄

I do feel strongly that we don't want to have stage have different syntax depending on where it shows up. That syntax disconnect would be worse than having to wait for this feature.

abayer · 2017-04-21T18:42:27Z

@bitwiseman stage has to have slightly different syntax unless we instead go with an entirely different term for an individual parallel "stage". The difference is just that an individual parallel "stage" can't itself contain further parallel "stages".

I'm not yet persuaded that parallel-at-top-level is a good idea.

bitwiseman · 2017-04-21T18:56:13Z

@abayer @HRMPW @rsandell - Seems like this is something that would benefit from a meeting.

abayer · 2017-04-21T19:01:15Z

Wait, @bitwiseman, re-reading your original comment, I think you may have misunderstood - you can't do

pipeline {
    stages {   
         stage("foo") {
             parallelStages {
                 stage("first") {
                     parallelStages {
                         stage("first-and-one") {

just

pipeline {
    stages {   
         stage("foo") {
             parallelStages {
                 stage("first") {

i.e., a stage within parallelStages cannot itself contain parallelStages. We only support one level of nesting. A stage with parallelStages also can't have steps, agent, or tools, since it serves no purpose other than being a "container".

bitwiseman · 2017-04-21T20:12:54Z

@abayer - Here's my core problem with the stage syntax disconnect - if you say "all stage syntax and behavior is the same", then you don't need to test as thoroughly. Because if it works in stage one place it works everywhere.

If stage is going to have different behavior here than elsewhere, I'd like to see basically all the tests you have for stages-stage duplicated here for parallelStages-stage, with the ones that specifically aren't supported marked as such.

abayer · 2017-04-21T20:14:50Z

@bitwiseman I disagree. The only thing that needs to be tested is that you can't use parallelStages in an already nested stage - the execution paths are identical, the only difference is at validation time.

abayer · 2017-04-21T20:16:50Z

Well, ok, technically they're not identical behind the scenes in that in stages->stage, it's actually invoking a closure inside a stage step, while in parallelStages->stage, it's invoking a closure inside a parallel step, but the content of that closure (i.e., everything about stage in Declarative other than "actually running the stage step") is identical.

bitwiseman · 2017-04-21T20:22:10Z

@abayer - Ah. I see. From a test perspective, I would never trust what you just said. 😃

How about let's compromise and add a test or two showing parallelStages->stage using other elements of stage than just steps. Show it with an agent, an environment, etc. Think of the test as functional documentation. They test framework you have works really well in this respect.

abayer · 2017-04-21T20:27:50Z

@bitwiseman Happy to do so!

abayer · 2017-04-21T21:18:18Z

@bitwiseman And environment is thoroughly busted, so good call on the additional testing needed. =)

abayer · 2017-04-21T21:42:10Z

@rsandell @bitwiseman And now we have testing of nested stages with agent, post, environment, tools, and when. And they all work, which some of them (post for parent stages, environment and when for nested stages) did not previously. Now if you'll excuse me, I'm going to go collapse for a while. =)

bitwiseman · 2017-04-21T22:12:01Z

@abayer - Excellent! Thanks!

rsandell

🐝

rsandell · 2017-05-03T10:45:57Z

...on/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy

+                            if (thisStage.parallelStages != null) {
+                                def parallelStages = [:]
+                                for (int i = 0; i < thisStage.parallelStages.stages.size(); i++) {
+                                    Stage parallelStage = thisStage.parallelStages.getStages().get(i)


Can you please stick to one syntax type, either parallelStages.stages or parallelStages.getStages(). I would prefer parallelStages.stages[i] as well instead of parallelStages.stages.get(i)

abayer · 2017-05-03T16:26:19Z

Rebased on master, didn't have a chance to verify tests passed, so will loop back in tomorrow.

michaelneale · 2017-05-03T22:04:16Z

oh thats a shame, went with parallelStages vs just "parallel" (I guess it wasn't possible to repurpose the existing parallel function?) I thought the RFC originally said just "parallel" - the verbosity is not needed as it is clear there are stages below it - is it a technical constraint that it was named parallelStages ? so 🐜 that it isn't confirming to the RFC as I see it.

i386 · 2017-05-03T23:11:32Z

Will have to agree with @michaelneale here - this does not line up with the RFC so 🐜 from me too.

rsandell · 2017-05-04T10:07:53Z

@michaelneale @i386 I believe it is technically possible to call it just parallel and I suggested it as well, but @abayer doesn't like it, you need to convince him ;)

abayer · 2017-05-04T12:54:55Z

@i386 @michaelneale We discussed this offline a few weeks back? I'm still open to alternative names, but vehemently opposed to overloading parallel for comprehension reasons, as we discussed.

…beta-2

This reverts commit 9e729c4.

…beta-3

abayer · 2017-07-25T18:35:43Z

@i386 @michaelneale FYI, I'm releasing 1.1.9 from master right now and then will be merging this to master, with the intention of not doing another release from master until Blue Ocean is ready for this. It's just been too damned long sitting unmerged. =)

Conflicts: pipeline-model-api/pom.xml pipeline-model-definition/pom.xml pipeline-model-extensions/pom.xml pipeline-model-json-shaded/pom.xml pipeline-stage-tags-metadata/pom.xml pom.xml

rtyler · 2017-07-25T19:06:29Z

abayer · 2017-07-25T19:13:15Z

@rtyler Not released yet! I just decided that it made no sense to sit unmerged any longer. I still owe a doc update PR too.

bitwiseman · 2017-07-25T19:36:59Z

@abayer - Still, every step toward this being merged is a win.

wagnst · 2017-07-25T19:40:27Z

stumbled upon this one yesterday as well, would love to see it being released 😄

samgomena · 2017-09-28T19:54:49Z

I'm not sure if this is still open for discussion, but I currently have this pipeline setup:

However, I need a pipeline that looks like this:

(Note: this pipeline was made possible by the use of Photoshop)

Is it possible to achieve this architecture with this update?

kzantow · 2017-09-28T20:44:37Z

@samgomena no, this only changes parallel to accept stages, however I believe what you're asking for is planned, just not necessarily a path forward yet.

morganchristiansson · 2017-09-29T19:55:36Z

I've noticed that the stages are reported in different order sometimes which disruptes the pipeline stage view history.

If stages within parallel were always reported in the order they are defined it would work better.

This works great btw! Using it for multiple pipelines with great success!

michaelneale · 2017-09-29T20:03:39Z

Morgan do you mind opening that as a new JIrA as I have seen that too.

quasarea · 2017-10-02T07:55:10Z

@kuhnroyal I have created an issue about not rendering properly skipped stages https://issues.jenkins-ci.org/browse/JENKINS-47219

abayer added this to the 1.2 milestone Apr 13, 2017

rsandell requested changes Apr 18, 2017

View reviewed changes

abayer requested a review from bitwiseman April 19, 2017 14:50

bitwiseman requested changes Apr 20, 2017

View reviewed changes

bitwiseman approved these changes Apr 21, 2017

View reviewed changes

rsandell approved these changes May 3, 2017

View reviewed changes

abayer force-pushed the jenkins-41334-mk2 branch from 7daff9d to 934d7b3 Compare May 3, 2017 15:34

abayer added 16 commits July 24, 2017 18:28

Whoops, bad copy pasta

00a2d3d

Fix an unclear indentation

5ff80e3

Test and fix post logic for parent stages

6a436a8

Fix nested stage environments and hopefully when

d3a7d8d

Test nested stage when as well.

8e3b376

Add nested stage tools test as well

16c75c2

Fix parameter name

e1b0ad9

Newer workflow-cps means we need that sandboxing.

bfb0028

Switch back to "parallel" for parallel stages syntax

4580b53

[maven-release-plugin] prepare for next development iteration

3431861

[maven-release-plugin] prepare release pipeline-model-definition-1.2-…

0551c50

…beta-2

[maven-release-plugin] prepare for next development iteration

37ed2ee

[JENKINS-44456] Set STAGE_NAME on parallel stages.

d898261

Revert "[JENKINS-44456] Set STAGE_NAME on parallel stages."

b65e432

This reverts commit 9e729c4.

[maven-release-plugin] prepare release pipeline-model-definition-1.2-…

f9ca30c

…beta-3

[maven-release-plugin] prepare for next development iteration

7707b42

abayer force-pushed the jenkins-41334-mk2 branch from d123b76 to 7707b42 Compare July 24, 2017 22:30

Merge branch 'master' into jenkins-41334-mk2

9b046f0

Conflicts: pipeline-model-api/pom.xml pipeline-model-definition/pom.xml pipeline-model-extensions/pom.xml pipeline-model-json-shaded/pom.xml pipeline-stage-tags-metadata/pom.xml pom.xml

abayer merged commit 103bec1 into jenkinsci:master Jul 25, 2017

abayer mentioned this pull request Jul 28, 2017

[JENKINS-41334] Add parallel stages documentation jenkins-infra/jenkins.io#1046

Merged

[FIXED JENKINS-41334] Adding parallel stages. #152

[FIXED JENKINS-41334] Adding parallel stages. #152

Conversation

abayer commented Apr 13, 2017

ghost commented Apr 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitwiseman left a comment

Choose a reason for hiding this comment

abayer commented Apr 21, 2017

bitwiseman commented Apr 21, 2017

HRMPW commented Apr 21, 2017 • edited

abayer commented Apr 21, 2017

HRMPW commented Apr 21, 2017

bitwiseman commented Apr 21, 2017

abayer commented Apr 21, 2017

bitwiseman commented Apr 21, 2017

abayer commented Apr 21, 2017 • edited

bitwiseman commented Apr 21, 2017

abayer commented Apr 21, 2017

abayer commented Apr 21, 2017

bitwiseman commented Apr 21, 2017 • edited

abayer commented Apr 21, 2017

abayer commented Apr 21, 2017

abayer commented Apr 21, 2017

bitwiseman commented Apr 21, 2017

rsandell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abayer commented May 3, 2017

michaelneale commented May 3, 2017 • edited

i386 commented May 3, 2017

rsandell commented May 4, 2017

abayer commented May 4, 2017

abayer commented Jul 25, 2017

rtyler commented Jul 25, 2017

abayer commented Jul 25, 2017

bitwiseman commented Jul 25, 2017

wagnst commented Jul 25, 2017 • edited

samgomena commented Sep 28, 2017

kzantow commented Sep 28, 2017

morganchristiansson commented Sep 29, 2017 • edited

michaelneale commented Sep 29, 2017

quasarea commented Oct 2, 2017

HRMPW commented Apr 21, 2017 •

edited

abayer commented Apr 21, 2017 •

edited

bitwiseman commented Apr 21, 2017 •

edited

michaelneale commented May 3, 2017 •

edited

wagnst commented Jul 25, 2017 •

edited

morganchristiansson commented Sep 29, 2017 •

edited