Allow parallel execution of tasks #156

Open
patriksvensson opened this Issue Jan 16, 2015 · 5 comments

Projects

None yet

5 participants

@patriksvensson
Member

Image the following task graph which start at 1 and ends at 5:

      1
     /|\
    / | \
   2  3  4
    \ | /
     \|/
      5

Normally, we would do a topographic sort that gives us the order 1 -> 2 -> 3 -> 4 -> 5, but since this example graph is diamond shaped, we could actually run task 2,3 and 4 in parallel, by treating them as a single task which would result in the execution order 1 -> (2 | 3 | 4) -> 5.

Since most tasks in a Cake script probably will be I/O bound, this isn't always what you would want, but theoretically this might shave some seconds of a big build script if applied to the right kind of tasks. We could make this functionality opt-in to prevent it from actually being slower.

@patriksvensson patriksvensson changed the title from Allow parallel execution of tasks. to Allow parallel execution of tasks Jan 16, 2015
@RichiCoder1
Contributor

I'd piggy back on this and also (re?)mention the idea of async tasks.

@lucian-podereu lucian-podereu pushed a commit to lucian-podereu/cake that referenced this issue Jan 11, 2016
@lukaishere lukaishere Allow parallel execution of tasks #156 3121baf
@aabenoja

Took a stab at this with this spike. Not ideal, but at least it cut down our 8-minute build time down to five.

@jnm2
Contributor
jnm2 commented Jan 15, 2017 edited

A big issue with parallelism is viewing the log output. I came up with a solution that buffers output so that the log looks exactly like the tasks are executed sequentially, but they are in fact simultaneous. As soon as the first action is finished, all the second action's buffered output is dumped immediately and you get to watch the second action continue live- or, if it's also finished, you see the third. Etc.

This can be used in your build scripts without modifying Cake.

https://gist.github.com/jnm2/a8a39b67a584ad555360102407049ae2

Task("Example")
    .Does(() => RunSimultaneously(
        () =>
        {
             StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("github.com")));
             Information("FINISHED 1");
        },
        () =>
        {
             RunSimultaneously(() =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("192.123.45.67")));
                 Information("FINISHED 2");
             },
             () =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("twitter.com")));
                 Information("FINISHED 3");
             });
        }
    ));

I'd love for some of this to be baked into Cake's dependency graph analysis without even needing to manually specify RunSimultaneously. That would be fantastic!

@pitermarx

might i suggest this for declaring parallel taks. Feel free to ignore if you think it doesn't make sense:

Task("Default")
.IsDependentOn("Build.Net", "Build.Js");
@aabenoja

I feel like it should be during the dependency graph analysis that it should determine how to best spin off and schedule tasks on their own threads. I feel that introducing any special syntax is unnecessary, as the same task definition should would regardless of parallelization.

@jnm2 As soon as I dropped my parallelization module into my project my build logs were almost useless. I think I'll take a cue from your gist and incorporate the output capturing into a buffered console writer. Still pretty satisfied with the gains, though my dependency graph analysis algorithm could definitely use some work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment