RFC: JENKINS-41334: Parallel stage execution

[abayer]

• JENKINS issue(s):
  • JENKINS-41334
• Description:
  • RFC for parallel stages in Declarative. [JENKINS-41334] Very very preliminary stage parallelism #98 is pertinent.
• Documentation changes:
  • n/a - this is just an RFC.
• Users/aliases to notify:
  • @rsandell
  • @HRMPW
  • @kzantow
  • @michaelneale
  • @i386
  • @bitwiseman

[svanoort]

This could be fixed though -- in all honesty, probably simpler than full nested stages support. Would require some work with pipeline-graph-analysis and extensions of the TagsAction for the status coding.

[abayer]

Well, not simpler from the Declarative perspective, given that we'd still have to have different code for executing nested stages-but-actually-just-parallel-branches and executing non-nested actual-stages.

[abayer]

And worth mentioning that I'm putting a healthy emphasis on keeping a common code path for all stage executions - if we have to differentiate at runtime between a parallel stage and a non-parallel stage, that's just going to create problems down the road.

[abayer]

@vivek - Also, I don't know how BO does status visualization for parallel branches currently. If we had equivalent metadata (i.e., the TagsAction stuff) on a parallel branch, would that be visualized in Blue Ocean as things are now?

[vivek]

@abayer I am using pipeline-graph-analysis to compute status of parallel branch. To be specific https://github.com/jenkinsci/pipeline-graph-analysis-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/pipelinegraphanalysis/StatusAndTiming.java#L152. So maybe some work is needed there @svanoort?

[abayer]

At this point, we're just talking about the metadata from https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-stage-tags-metadata/src/main/java/org/jenkinsci/plugins/pipeline/StageStatus.java. That is to say - nothing new.

[svanoort]

@abayer I was unaware you'd added that. Congratulations! You have now broken status handling, exactly as I described, and ensured that declarative is no longer truly compatible with pipeline because it uses a completely different status-coding mechanism that is declarative-specific. 😄

[abayer]

99% sure you 👍 ed it at the time... =)

[svanoort]

@abayer Pretty sure I didn't +1 anything like this in its current form ;-) -- I would have given this a hard 🐛 and "do not do this, not ever" if I'd seen it.

[svanoort]

@abayer So, here's the scoop, after looking closer at the history and context: using these tags is okay as long as you are also generating appropriate ErrorActions, etc from running the pipeline.

I think unless we missed something critical at the time that the old way of using these was to assist on top of that mechanism by adding some additional context.

What it sounds like is that there's a possibility of replacing the original flow structure and action information with the Metadata, for functional purposes. And that is very incompatible.

The flow needs to either execute correctly, or we need to make an upstream change in workflow-api to support use of status tags on nodes/blocks as a superset of just the current mechanisms.

[svanoort]

Pinging @i386 is this ever coming? My understanding is there was some discussion of never supporting this.

[svanoort]

Specifically the use of parallels inside stages, I mean.

[vivek]

Parallel inside stage is supported. So just one level deep, nested stages/parallels within parallel branches within a stage are not supported.

[abayer]

So parallel(foo: { stage('first nested') { ... } }, bar: { stage('second nested') { ... } } is not supported, correct?

[i386]

@abayer that is not supported today.

[svanoort]

@i386 I'm not suggesting you should support it necessarily here, so you can relax -- but I think Andrew's proposal would require it 100%.

[abayer]

Yeah, I'm the one saying we need it. =)

[michaelneale]

Hrm... I can't remember those mockups... do you happen to remember what medium we stored them in @abayer ?

[michaelneale]

The way I see this is a great improvement to parallel, but without radically changing how a stage graph may behave - fair to say?

[abayer]

Yup!

[abayer]

cc @vivek as well, now that I think about it.

[abayer]

Worth mentioning that we can definitely opt to block multiple levels of nested stages if we so choose. I'm starting with the default assumption that we allow arbitrary levels of nesting, but there's no technical reason keeping us from limiting it to one level. Now, limiting to 2 or 3 or any other >1 level of nesting, that'd be annoying, but would also be goofy, so...

[i386]

Just want to be clear that I strongly believe that the language only supports levels of nesting that the visualization can handle. I personally have to triage many tickets relating to combinatorial stage nesting and users are frequently perplexed to why the script dialect supports N stage/parallel nesting but none of the visualizations do. Evidence for this can be found ad infinitum in JENKINS-38442.

[abayer]

Hence my comment above in re: limiting the nesting level supported by Declarative. =)

[ykoehler]

I have the following stages A, B, C & D. B depends on A, C can be done in parallel to both A & B and finally D depends on A, B & C. How would this above support this?

 A -> B -> D
 C ---------^

I think the embedded approach limits the possibility because sometimes the visual aspect (embed) doesn't apply well to advanced pattern such as the above..

[HRMPW]

@ykoehler what is C doing in this case? @i386 and I just talked about this use-case earlier today but would really like more information on the problem it solves. There are various ways we could do something like that.

One of which is captured here: https://issues.jenkins-ci.org/browse/JENKINS-26052

[rsandell]

Given that I hear and see the need for some type of matrix support _(e.g. defining one set of steps that will be executed in parallel with just a difference in environment and agent)_I think we need to prepare for that coming quite soon, and we should at least think about how it would be impacted by this so that the matrix support doesn't get squished in with some half measures because we designed this part the wrong way.
Yes that could be done with yet another directive that is not parallel but it might confuse the visualization part.

[joshsleeper]

@ykoehler @HRMPW
I could be off mark here, but what about varying length build chains for final build dependencies?
Something like the following...

---

Let's say that my project build requires libs B and C, and in turn lib B requires lib A, and before any of that we get code from SCM.
So the dependency graph looks something like this:

SCM --> A --> B --> Project
   \------ C ------/

Furthermore, for the sake of argument let's give those libs estimated build times:

• Lib A builds in ~5 minutes
• Lib B builds in ~10 minutes
• Lib C builds in ~20 minutes

In the current declarative pipeline syntax the most efficient thing we can do is make an educated guess about which stages will parallelize the best, resulting in something like this.

stage ('SCM') {
    steps {...}
}

stage ('Build A) {
    steps {...}
}

stage ('Build B and C') {
    steps {
        parallel (
            'Build B': {...},
            'Build C': {...}
        )
    }
}

stage ('Project Build') {
    steps {...}
}

This results in a pipeline flow graph that looks like this:

              B
SCM --> A --> + --> Project
              C

Which at least parallelizes the longest running pieces of the process, but it still has C arbitrarily waiting for A to finish.

The desired pipeline flow graph would look exactly like the first dependency graph, where A --> B runs 100% parallel to C.
The suggested RFC syntax is somewhat vague on how that logic might be written (or I'm not just not grasping it), but it could potentially read something like this:

stage ('SCM') {
    steps {...}
}

stage ('Build Libs') {
    parallel {
        stages {
            stage ('Build A) {
                steps {...}
            }

            stage ('Build B') {
                steps {...}
            }
        }

        stage ('Build C') {
            steps {...}
        }
    }
}

stage ('Project Build') {
    steps {...}
}

Or, taking a more pseudo-declarative-pipeline stab at it myself where parallel is an explicit type of stage and contains X number of parallel paths (which I think is a bit simpler to read, or at least to grok), it could read something like this:

stage ('SCM') {
    steps {...}
}

parallel ('Build Libs') {
    path {
        stage ('Build A) {
            steps {...}
        }

        stage ('Build B') {
            steps {...}
        }
    }

    path {
        stage ('Build C') {
            steps {...}
        }
    }
}

stage ('Project Build') {
    steps {...}
}

---

What I and many coworkers want is a [reasonably] arbitrary number of stages in each parallel path, with proper post{} control flow at both the path level and the stage level.

This type of logic is particularly useful for:

• Building unrelated build dependencies in parallel, as per my example.
• Platform deployment testing, because deploying and testing Platform A shouldn't be dependent on how fast or slow Platform B is.

Aaaand that was a novel.
Hope it at least got my point across!

[michaelneale]

@joshsleeper good "novel" - I enjoyed reading it.
Some comments:

In the case of:

SCM --> A --> B --> Project
   \------ C ------/

You know the dependencies, so you lay them out clearly. YOu can effectively do this now but without stages by using steps. This still lets you run A->B but with C concurrently, just there won't be a discrete circle going green for A or B, but A&B together:

It would look like this:

stage ('SCM') {
    steps {...}
}

stage ('Build B and C') {
    steps {
        parallel (
            'Build A&B': {
                .. steps for A ...
                .. steps for B ...
             },
            'Build C': {...}
        )
    }
}

stage ('Project Build') {
    steps {...}
}

There is no functional difference to the functionaltiy, it is just that build A&B are not separated out into their own grouping (stage) yet. But it will work. You just see one circle for A&B, but it doesn't stop C from taking its own time.

Your proposal for using nested stages looks logically correct to, IF/When it can be visuall supported (in the stage graph, and visual editor - takes time. Know SVG and reactjs and html5?). This PR is more about the first change - ie getting stage into parallel so it looks consistent and then opens the way to the type of thing you suggest, which I like the look of.

As for your final points:

• Building unrelated build dependencies in parallel, as per my example.
• Platform deployment testing, because deploying and testing Platform A shouldn't be dependent on how fast or slow Platform B is.

There was an attempt in the past to do a more "declarative" dependency chain which could calculate the graph for you - but it was viewed as more confusing for many to use. But who knows, it could happen. But your suggestion of path - that is functionally the same as your earlier suggestion right? I don't think that is needed as the outer parallel can easily work that out from the nested stages in it.

For the final point: There is planned support specifically for matrix - which is what most people want here. Ie specifying some dimensions and data - and letting the pipeline how to work it out. It also changes the visualisation to be more "matrix like" as you don't want some explotion of parallel lines here anyway.

I have now added to your novel

[joshsleeper]

@michaelneale Good points all around, thanks for the feedback!

1. You're 110% correct, I could totally just combine A and B into a parallel group. My examples didn't clearly express that I already knew I could though, so totally fair point.

The problem, from my standpoint, is quickly determining where something failed.
I gave a simple example just to support the discussion between @HRMPW and @ykoehler, but in reality that's not the best example for that type of separation.

That said, extend that logic and say that one parallel path has 10 things and another has 2.
In a case like that, combining all 10 builds into a single "stage" from the pipeline's view does make it much harder to quickly determine exactly where the failure really was, which is what I was trying to get at there.

Another example was something my co-worker brought up after switching to declarative pipeline syntax for his jobs. He was doing a build -> deploy -> test process for a number of different platforms, and was irked when he found out that to get real parallelism his pipeline ended up looking like this:

       +-(p1)-+
       |      |
       +-(p2)-+
(SCM)--|      |--(Results)
       +-(p3)-+
       |      |
       +-(p4)-+

When what he wanted to see was this:

       +-(p1_build)--(p1_deploy)--(p1_test)-+
       |                                    |
       +-(p2_build)--(p2_deploy)--(p2_test)-+
(SCM)--|                                    |--(Results)
       +-(p3_build)--(p3_deploy)--(p3_test)-+
       |                                    |
       +-(p4_build)--(p4_deploy)--(p4_test)-+

Not saying that has to be in the first revision here, but I am saying that that type of visualization is desired for parallelism and I see a lot of value there.

2. Hey, glad I ballparked right on the nesting! haha.
   I re-wrote that like 5 times trying to make sure my logic made sense.
   Unfortunately I'm minimally knowledgeable in any of those things, so I'm afraid I won't be too much help with the actual UI coding side of things. Sorry!

3. Correct again, my parallel() {...} and path{...} example was functionally equivalent to my first nested example, and was purely to clarify the logical flow of things. That's why I called it "pseudo-declarative-pipeline", because I didn't want people to think I was formally suggesting that exact syntax. Not that I mind it, now that I give it a second glance...

4. I also agree, that general matrix support likely covers a lot of the bases here. I just wanted to provide another side to things, in that I firmly believe imbalanced parallel paths are totally viable and should be able to be visualized as such, at least in the long run.

At what point does this become an omnibus?

[michaelneale]

@joshsleeper yeah I think we are on same page. That last graph of yours is clearly a type of matrix, so that is good. And yes, I can totally see why you need that fast feedback of where in the graph things have failed.

https://issues.jenkins-ci.org/browse/JENKINS-38442 is for nested stages etc - from a front end perspective.

[rsandell]

Just throwing a completely different "half baked/half stupid" idea out there;
Introducing the notion of execution groups

stage("Prepare") {
  steps {
    sh "echo 'Do something'"
  }
}
stage("Build A") {
  group "A"
  steps {
    dir ('A') {
      sh "mvn clean compile"
    }
  }
}
stage("Build B") {
  group "B"
  steps {
    dir ('B') {
      sh "mvn clean compile"
    }
  }
}
stage("Test") {
  group ["A", "B"]
  steps {
    dir (group.name) {
      sh "mvn test"
    }
  }
}
stage("Extra B") {
  group "B"
  steps {
    dir ('B') {
      sh "..."
    }
  }
}
stage("Done") {
  steps {
    sh "echo 'Do something more'"
  }
}

Stage A followed by Test would run in parallel with Stage B followed by Extra B followed by Test

Any non "grouped" stage means waiting for the previous declared "grouped" stages to finish.

[michaelneale]

@rsandell that looks pretty confusing (and visualising it, kidn of breaks the idea of having stages at all). I am not sure what that does that others can't funcitonally do, unless we want to give up the idea of stages and just have a graph of nodes (which could be better done with dependencies anyway).

[abayer]

@vivek @i386 @michaelneale I've updated the RFC with the decision to simply use parallel branches without actual nested StageStep executions, meaning no changes needed for Blue Ocean visualization at all. Obviously this will still need changes for the editor, but that's all.

[HRMPW]

What happens when Blue Ocean implements support for nested stages inside parallel? They will have to special case these stages which will be hidden from visualization for now.

[abayer]

If Blue Ocean's eventual implementation for visualizing something like

parallel(a: { stage('a') { echo 'Hi, this is a' } },
             b: { stage('b') { echo 'Hi, this is b' } })

ends up being different than how it visualizes

parallel(a: { echo 'Hi, this is a' },
             b: { echo 'Hi, this is b' })

then we'll want to change Declarative's runtime for parallel "stages" accordingly. But that's a bridge we can cross if/when that happens.

[joshsleeper]

@abayer To be clear, this would enable support for things like agent and post on a per-parallel branch basis, correct?

[michaelneale]

@abayer I don't quite follow - what is the latest proposal for this? how will parallel look different to now?

@HRMPW I expect it will show nested at some point - maybe special casing will be required (not the end of the world).

[abayer]

@joshsleeper Yes, exactly. =)

@michaelneale So this means no change at all from the proposed Declarative syntax. It's just a change to how we execute that syntax behind the scenes, using something like:

parallel(a: { echo 'Hi, this is a' },
             b: { echo 'Hi, this is b' })

...which Blue Ocean already visualizes.

[HRMPW]

@michaelneale that is why I was asking about future. Right now BO would ignore stage inside parallel based on @abayer 's changes. In the future, to support stage inside parallel you will have to find a way to dig out the previously ignored stage

[joshsleeper]

@abayer Mmk, great.

To further clarify, this also does not facilitate displaying multiple parallel stages, correct?

That is, I still can't get a visualization of something like this (where paren sets denote a visualized "stage"):

       +-(p1_build)--(p1_deploy)--(p1_test)-+
(SCM)--|                                    |--(Results)
       +-(p2_build)--(p2_deploy)--(p2_test)-+

But I could get the same functional parallelism by doing this instead:

       +-(p1_build -> p1_deploy -> p1_test)-+
(SCM)--|                                    |--(Results)
       +-(p2_build -> p2_deploy -> p2_test)-+

[abayer]

@joshsleeper So this is for executing a list of stages in parallel with each other - i.e., one parallel call - and no arbitrary nesting - i.e., you could only do parallel stages inside a top-level stage, not inside a stage that itself is already part of a list of parallel stages.

What you're talking about is what I've been calling "matrix equivalence" - that's most definitely a longer-term goal, both for the Declarative syntax/editor and Blue Ocean visualization, but there is a lot of work (both design and implementation) to get there, so I don't yet have a timeframe for when we will hopefully be able to do that. =)

[michaelneale]

@abayer ok - so the syntax changes to accommodate per branch agent config (for example) but really it is the same thing in terms of what the result are (sorry I am way out of the loop here).

[abayer]

@michaelneale Basically, yes. There's some work needed on the Declarative runtime side to make sure that the supplementary metadata we add to stages now (like I said, to mark a stage as "skipped due to REASON" and the like) can be added to parallel branches as well. As far as I can tell from quick looks at pipeline-graph-analysis and blueocean-pipeline-api-impl, once we've got the metadata on the branches too, the same status visualization for the parallel branches as for standalone stages will just work.

[svanoort]

My input has been requested into the implementation here, so after taking another look:

This feels like it's more likely to make declarative more confusing rather than more "lovable" because of the added use confusion and code complexity that will result. A couple points:

• Conditional execution -- what about interaction b/w branches? Trying to compare to previous runs will be a big problem. Parallel branches may/may not exist, may never have run, may have run but without that stage, etc.
• Pre/post but what about failFast vs. non-failfast on the parallel?
• Inheritance of configuration from the enclosing stage vs. the stage inside the parallel? This has the potential to get rather confusing fast.
• We're adding another dimension to complexity here, and from the number of bugs and the amount of extra code it's taken to handle some of the parallel cases in the flowgraph I would approach this very cautiously.

My strong recommendation would be to force a different syntax for multi-component parallel branches that creates a simpler structure for when you need to run through a matrix of options. Maybe something like a for-each-combination which uses the same stages for each branch but allows you to apply a different input variable to each. This ensures consistent structure and removes some of the variables, but still lets you accommodate the base use cases.

But at this point, I honestly don't understand Declarative -- it has clearly become its own little world so who knows? I only know the challenges that came with throwing the world wide-open in normal pipeline.

[svanoort]

Do we need inheritance here? Weigh this carefully -- it greatly increases the potential for user confusion and bugs. Would it make more sense ensure all branches follow a common set of parameters (or possibly take an input for each branch to use in the given property)?

[svanoort]

What's the use case for this?

[abayer]

What I'm thinking at the moment is to allow environment to be specified on the parent stage (since that's something I can see applying, with overrides, across all the child stages). But for the others, yeah, inheritance actually seems like a bad idea (beyond the always there default inheritance to the top-level). In which case, we'd probably reject agent and tools fields from a parent stage entirely.

[svanoort]

Perhaps. I think we may want to run take a closer look specifically at the options allowed and when inheritance does/does not make sense, and where it might make sense to use something other than inheritance, potentially (leaving that one open here for now).

[abayer]

Fair enough - I'd say environment should always inherit, but I can see an argument for not even having the top-level agent and tools be inherited by parallel stages - or perhaps inheriting the configuration for agent as a default config for the parallel stages but not by default running on the same node block as the top-level...will ponder.

[svanoort]

This is smart.

[svanoort]

Smart, good common sense.

[svanoort]

Recommend something like "foreach" or "branches" to create a distinction between this implementation and the different vanilla pipeline step.

[michaelneale]

-1 on names like branch or foreach - would want to pick something more declarative (and branch creates confusion already..)

[svanoort]

👎 on this. If you want to declare a branch, call it a branch, not a stage. Pre/post can be done, but I do not think we could or should effectively support using metadata with branches, and especially conditional execution at this point. This is a hack and a potentially dangerous one. There's a right way to do this, and that's to start looking at a richer execution & status model for pipeline -- but that needs to be provided from a lower level API and then bubble up to the Declarative which takes advantage.

[abayer]

I'm happy to wait for such functionality to be introduced, assuming we can get the ball rolling on that.

[michaelneale]

But there are already nested stages (they aren't very useful, but they are there). FWIW a lot of people use the term "stage" when talking about parallel branches of execution (doesn't make it right, but there is some intuition in there going on I guess).

[abayer]

@svanoort Thanks for the feedback! In re a couple things you brought up:

• Conditional execution - I hadn't considered that very thoroughly, so good catch. =) I think we'd still want when on the parent stage (so as not to require copy-pasting the same when config to all the child stages), but not matching the parent when condition should probably result in the child stages being executed-but-skipped as if they'd had a non-matching when condition themselves.
• failFast - Not planning to support it here at all. Seems to me like a reasonable opinionated decision for Declarative to make. i.e., "This is how parallel stages work in Declarative, which may use parallel on the backend, but that doesn't mean you get to control all the knobs and switches on parallel's configuration".
• Inheritance of configuration - as I mentioned inline, yeah, you're right on this. Needs to be simpler and harsher in rejecting things.

[svanoort]

A couple quick replies before I dive back into other stuff to wrap up today:

I think we'd still want when on the parent stage

Think carefully about the implications of this for comparing one parallel vs. another -- Vivek and I spent the better part of 2 hours talking through how to match pipelines with different parallel & stage structures & names for purposes of gathering averages & and metrics. The conclusion was a set of heuristics but knowing it would always have problems.

My gut is that "when" should only apply to the overall parallel block, never specific branches (to simplify), making a parallel "all or none." If someone wants more, that's a perfect case for vanilla pipeline.

failFast

That's actually a very useful part of parallel IMO, but I can't think of a sensible way to manage it along with pre/post either. Agree it's probably best to say "if you want this, you need full pipeline."

[abayer]

@svanoort So you mean only allowing when being specified on the parent stage, not on the parallel child stages?

[abayer]

I'm going to mark this as approved - we can revise it later if needed, but I think we've got a general consensus.

[abayer]

Though I am moving the target release to 1.3 - 1.2 may well happen too soon for Blue Ocean to be ready for this.