New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow parallel execution of tasks #156

Open
patriksvensson opened this Issue Jan 16, 2015 · 27 comments

Comments

Projects
None yet
9 participants
@patriksvensson
Member

patriksvensson commented Jan 16, 2015

Image the following task graph which start at 1 and ends at 5:

      1
     /|\
    / | \
   2  3  4
    \ | /
     \|/
      5

Normally, we would do a topographic sort that gives us the order 1 -> 2 -> 3 -> 4 -> 5, but since this example graph is diamond shaped, we could actually run task 2,3 and 4 in parallel, by treating them as a single task which would result in the execution order 1 -> (2 | 3 | 4) -> 5.

Since most tasks in a Cake script probably will be I/O bound, this isn't always what you would want, but theoretically this might shave some seconds of a big build script if applied to the right kind of tasks. We could make this functionality opt-in to prevent it from actually being slower.

@patriksvensson patriksvensson changed the title from Allow parallel execution of tasks. to Allow parallel execution of tasks Jan 16, 2015

@RichiCoder1

This comment has been minimized.

Show comment
Hide comment
@RichiCoder1

RichiCoder1 Jan 16, 2015

Contributor

I'd piggy back on this and also (re?)mention the idea of async tasks.

Contributor

RichiCoder1 commented Jan 16, 2015

I'd piggy back on this and also (re?)mention the idea of async tasks.

lucian-podereu pushed a commit to lucian-podereu/cake that referenced this issue Jan 11, 2016

@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Jan 12, 2017

Took a stab at this with this spike. Not ideal, but at least it cut down our 8-minute build time down to five.

aabenoja commented Jan 12, 2017

Took a stab at this with this spike. Not ideal, but at least it cut down our 8-minute build time down to five.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Jan 15, 2017

Contributor

A big issue with parallelism is viewing the log output. I came up with a solution that buffers output so that the log looks exactly like the tasks are executed sequentially, but they are in fact simultaneous. As soon as the first action is finished, all the second action's buffered output is dumped immediately and you get to watch the second action continue live- or, if it's also finished, you see the third. Etc.

This can be used in your build scripts without modifying Cake.

https://gist.github.com/jnm2/a8a39b67a584ad555360102407049ae2

Task("Example")
    .Does(() => RunSimultaneously(
        () =>
        {
             StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("github.com")));
             Information("FINISHED 1");
        },
        () =>
        {
             RunSimultaneously(() =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("192.123.45.67")));
                 Information("FINISHED 2");
             },
             () =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("twitter.com")));
                 Information("FINISHED 3");
             });
        }
    ));

I'd love for some of this to be baked into Cake's dependency graph analysis without even needing to manually specify RunSimultaneously. That would be fantastic!

Contributor

jnm2 commented Jan 15, 2017

A big issue with parallelism is viewing the log output. I came up with a solution that buffers output so that the log looks exactly like the tasks are executed sequentially, but they are in fact simultaneous. As soon as the first action is finished, all the second action's buffered output is dumped immediately and you get to watch the second action continue live- or, if it's also finished, you see the third. Etc.

This can be used in your build scripts without modifying Cake.

https://gist.github.com/jnm2/a8a39b67a584ad555360102407049ae2

Task("Example")
    .Does(() => RunSimultaneously(
        () =>
        {
             StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("github.com")));
             Information("FINISHED 1");
        },
        () =>
        {
             RunSimultaneously(() =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("192.123.45.67")));
                 Information("FINISHED 2");
             },
             () =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("twitter.com")));
                 Information("FINISHED 3");
             });
        }
    ));

I'd love for some of this to be baked into Cake's dependency graph analysis without even needing to manually specify RunSimultaneously. That would be fantastic!

@pitermarx

This comment has been minimized.

Show comment
Hide comment
@pitermarx

pitermarx Jan 16, 2017

might i suggest this for declaring parallel taks. Feel free to ignore if you think it doesn't make sense:

Task("Default")
.IsDependentOn("Build.Net", "Build.Js");

pitermarx commented Jan 16, 2017

might i suggest this for declaring parallel taks. Feel free to ignore if you think it doesn't make sense:

Task("Default")
.IsDependentOn("Build.Net", "Build.Js");
@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Jan 17, 2017

I feel like it should be during the dependency graph analysis that it should determine how to best spin off and schedule tasks on their own threads. I feel that introducing any special syntax is unnecessary, as the same task definition should would regardless of parallelization.

@jnm2 As soon as I dropped my parallelization module into my project my build logs were almost useless. I think I'll take a cue from your gist and incorporate the output capturing into a buffered console writer. Still pretty satisfied with the gains, though my dependency graph analysis algorithm could definitely use some work.

aabenoja commented Jan 17, 2017

I feel like it should be during the dependency graph analysis that it should determine how to best spin off and schedule tasks on their own threads. I feel that introducing any special syntax is unnecessary, as the same task definition should would regardless of parallelization.

@jnm2 As soon as I dropped my parallelization module into my project my build logs were almost useless. I think I'll take a cue from your gist and incorporate the output capturing into a buffered console writer. Still pretty satisfied with the gains, though my dependency graph analysis algorithm could definitely use some work.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Feb 7, 2017

Contributor

Okay so how about this use case:

I want to run unit tests and integration tests before building the installer, but both take a while. So what I'd like to do run tests and build the installer at the same time, but cancel building the installer as soon as any test fails. If the installer task finishes before the tests, it should wait for the tests to complete (or fail) before staging the build installer files.

Contributor

jnm2 commented Feb 7, 2017

Okay so how about this use case:

I want to run unit tests and integration tests before building the installer, but both take a while. So what I'd like to do run tests and build the installer at the same time, but cancel building the installer as soon as any test fails. If the installer task finishes before the tests, it should wait for the tests to complete (or fail) before staging the build installer files.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Feb 7, 2017

Contributor

In pseudocode:

var runUnitTests = Task("RunUnitTests").Does(() => ...);
var integrationTestNewDatabase = Task("IntegrationTestNewDatabase").Does(() => ...);
var integrationTestUpgradeDatabase = Task("IntegrationTestUpgradeDatabase").Does(() => ...);

var buildInstaller = Task("BuildInstaller").Does(parallelDependencies => 
{
    BuildInstallerStep1();

    if (parallelDependencies.HaveFailure) return; // If any fails, let the build end
    BuildInstallerStep2();

    if (parallelDependencies.HaveFailure) return;
    BuildInstallerStep3();

    if (!parallelDependencies.WaitForSuccess()) return; // Waits for all to succeed or any to fail
    StageFiles();
});

Task("TestAndBuildInstaller")
    .DependsOn(runUnitTests)
    .DependsOn(integrationTestNewDatabase)
    .DependsOn(integrationTestUpgradeDatabase)
    .DependsOn(buildInstaller, parallelDependences: new[] {
        runUnitTests,
        integrationTestNewDatabase,
        integrationTestUpgradeDatabase
    });
Contributor

jnm2 commented Feb 7, 2017

In pseudocode:

var runUnitTests = Task("RunUnitTests").Does(() => ...);
var integrationTestNewDatabase = Task("IntegrationTestNewDatabase").Does(() => ...);
var integrationTestUpgradeDatabase = Task("IntegrationTestUpgradeDatabase").Does(() => ...);

var buildInstaller = Task("BuildInstaller").Does(parallelDependencies => 
{
    BuildInstallerStep1();

    if (parallelDependencies.HaveFailure) return; // If any fails, let the build end
    BuildInstallerStep2();

    if (parallelDependencies.HaveFailure) return;
    BuildInstallerStep3();

    if (!parallelDependencies.WaitForSuccess()) return; // Waits for all to succeed or any to fail
    StageFiles();
});

Task("TestAndBuildInstaller")
    .DependsOn(runUnitTests)
    .DependsOn(integrationTestNewDatabase)
    .DependsOn(integrationTestUpgradeDatabase)
    .DependsOn(buildInstaller, parallelDependences: new[] {
        runUnitTests,
        integrationTestNewDatabase,
        integrationTestUpgradeDatabase
    });
@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Feb 7, 2017

Task("stage-installer")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer");

Task("ci")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer")
  .IsDependentOn("stage-installer");

We have a similar setup on the project I dropped my module in. unit-tests, integration-tests and build-installer have their own dependencies setup. In the end, these three tasks will all run in parallel and the staging tasks just waits until these are done. If any of these tasks fail the exception bubbles up and the whole build fails; stage-installer gets skipped completely.

aabenoja commented Feb 7, 2017

Task("stage-installer")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer");

Task("ci")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer")
  .IsDependentOn("stage-installer");

We have a similar setup on the project I dropped my module in. unit-tests, integration-tests and build-installer have their own dependencies setup. In the end, these three tasks will all run in parallel and the staging tasks just waits until these are done. If any of these tasks fail the exception bubbles up and the whole build fails; stage-installer gets skipped completely.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Feb 8, 2017

Contributor

@aabenoja Much cleaner! However, no early exit for "build-installer" which is one of the longest parts for me. They are intrinsically different: I want "build-installer" to exit as soon as any of the tests fail, but I want all test runs to complete even if any of them fail. Could we do this?

It's a little weird getting my head around the fact that Tasks would execute in two stages: 1. IsDependentOn tasks are executed in parallel (if possible) 2. Does(() => ...) runs afterwards, not in parallel with the dependencies. So if you want to force something to not be parallel, you have to create an additional task and separate the non-parallelizable part into a Does section. Seems solid enough though.

Contributor

jnm2 commented Feb 8, 2017

@aabenoja Much cleaner! However, no early exit for "build-installer" which is one of the longest parts for me. They are intrinsically different: I want "build-installer" to exit as soon as any of the tests fail, but I want all test runs to complete even if any of them fail. Could we do this?

It's a little weird getting my head around the fact that Tasks would execute in two stages: 1. IsDependentOn tasks are executed in parallel (if possible) 2. Does(() => ...) runs afterwards, not in parallel with the dependencies. So if you want to force something to not be parallel, you have to create an additional task and separate the non-parallelizable part into a Does section. Seems solid enough though.

@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Feb 8, 2017

Actually, build-installer would exit early if it takes significantly longer than either of those test tasks and if either of them fail. I recently fixed an issue where the error wouldn't kill the other task chains. What it doesn't do, however, is continue to run the test tasks regardless of the failure of the others. I think that is something better addressed as a configuration of the individual tasks themselves to continue executing regardless of the state of the cancellation token.

Task("unit-tests")
  .IgnoreCancellation();

Task("integration-tests")
  .IgnoreCancellation();

While maybe not ideal, but at least this way the whole build will still fail and we get the full results of these two tasks.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

aabenoja commented Feb 8, 2017

Actually, build-installer would exit early if it takes significantly longer than either of those test tasks and if either of them fail. I recently fixed an issue where the error wouldn't kill the other task chains. What it doesn't do, however, is continue to run the test tasks regardless of the failure of the others. I think that is something better addressed as a configuration of the individual tasks themselves to continue executing regardless of the state of the cancellation token.

Task("unit-tests")
  .IgnoreCancellation();

Task("integration-tests")
  .IgnoreCancellation();

While maybe not ideal, but at least this way the whole build will still fail and we get the full results of these two tasks.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Feb 8, 2017

Contributor

I'm thinking cancellation should be opt-in. Thread.Abort has issues for this time of thing and really isn't recommended. I'd be more in favor of passing a CancellationToken and having real cleanup, as in, running processes get killed.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

Contributor

jnm2 commented Feb 8, 2017

I'm thinking cancellation should be opt-in. Thread.Abort has issues for this time of thing and really isn't recommended. I'd be more in favor of passing a CancellationToken and having real cleanup, as in, running processes get killed.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Feb 8, 2017

I'd be more in favor of passing a CancellationToken and having a real cleanup

Which I am. I've been generating task chains for each dependency, sharing tasks where necessary, and passing around a single CancellationToken across all tasks.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

Task("nuget-restore").Does(() => {});

Task("compile")
  .IsDependentOn("nuget-restore")
  .Does(() => {});

Task("unit-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("integration-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("tests")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests");

To go more in-depth on what I've done, build.ps1 -t unit-tests will execute "nuget-restore" -> "compile" -> "unit-tests" all sequentially. Same with build.ps1 -t integration-tests. The tests task will run restore and compile sequentially (as expected) and then run unit-tests and integration-tests in parallel.

The whole idea is to execute in parallel where it makes sense. It's possible to throw .IsDependentOn("nuget-restore") onto the integration-tests task. As stated before each task is wrapped in its own TPL Task and shared where it makes sense. Even though integration-tests would await on both tasks to complete that is no different than before.

aabenoja commented Feb 8, 2017

I'd be more in favor of passing a CancellationToken and having a real cleanup

Which I am. I've been generating task chains for each dependency, sharing tasks where necessary, and passing around a single CancellationToken across all tasks.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

Task("nuget-restore").Does(() => {});

Task("compile")
  .IsDependentOn("nuget-restore")
  .Does(() => {});

Task("unit-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("integration-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("tests")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests");

To go more in-depth on what I've done, build.ps1 -t unit-tests will execute "nuget-restore" -> "compile" -> "unit-tests" all sequentially. Same with build.ps1 -t integration-tests. The tests task will run restore and compile sequentially (as expected) and then run unit-tests and integration-tests in parallel.

The whole idea is to execute in parallel where it makes sense. It's possible to throw .IsDependentOn("nuget-restore") onto the integration-tests task. As stated before each task is wrapped in its own TPL Task and shared where it makes sense. Even though integration-tests would await on both tasks to complete that is no different than before.

@aabenoja

This comment has been minimized.

Show comment
Hide comment
@aabenoja

aabenoja Feb 8, 2017

Another example is a mvc web app. Before we create a deployable zip to drop into iis we want to ensure javascript is compiled, javascript tests are passing, our dlls are generated, and xunit tests (both unit and integration) are run. We shouldn't care what order these things and their dependencies occur, just that they happen, no different than when setting up the task without parallelization in mind. If anything, it makes more sense to me to use a cli flag run all tasks sequentially or in parallel. Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

aabenoja commented Feb 8, 2017

Another example is a mvc web app. Before we create a deployable zip to drop into iis we want to ensure javascript is compiled, javascript tests are passing, our dlls are generated, and xunit tests (both unit and integration) are run. We shouldn't care what order these things and their dependencies occur, just that they happen, no different than when setting up the task without parallelization in mind. If anything, it makes more sense to me to use a cli flag run all tasks sequentially or in parallel. Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Feb 8, 2017

Contributor

Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

That's certainly an opinionated way for Cake to go, but it might be a good thing.

Contributor

jnm2 commented Feb 8, 2017

Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

That's certainly an opinionated way for Cake to go, but it might be a good thing.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

Counterexample:

Task("Clean")
    .Does(() => { });

// Priority #1: You want to be able to restore without cleaning
Task("Restore")
    .Does(() => { });

// Priority #2: You want to be able to build without cleaning or restoring
Task("Build")
    .Does(() => { });

// Priority #3: But never pack without cleaning
// Problem: how do you keep all three dependencies from attempting to run at the same time while preserving priorities 1 and 2?
Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .Does(() => { });
Contributor

jnm2 commented Apr 28, 2017

Counterexample:

Task("Clean")
    .Does(() => { });

// Priority #1: You want to be able to restore without cleaning
Task("Restore")
    .Does(() => { });

// Priority #2: You want to be able to build without cleaning or restoring
Task("Build")
    .Does(() => { });

// Priority #3: But never pack without cleaning
// Problem: how do you keep all three dependencies from attempting to run at the same time while preserving priorities 1 and 2?
Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .Does(() => { });
@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

If the only way to prevent things from running in parallel is to make them dependent on each other, there's no way to preserve priorities 1 and 2 which are very legit priorities.

This boilerplate can't help you:

Task("CleanAndRestore")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore");

Neither can this:

Task("Clean2")
    .IsDependentOn("Clean")

Task("Restore2")
    .IsDependentOn("Clean2")
    .IsDependentOn("Restore");
 // There's still nothing to indicate that "Restore" should wait for "Clean2" to finish

Task("Build2")
    .IsDependentOn("Restore2")
    .IsDependentOn("Build")
 // There's still nothing to indicate that "Build" should wait for "Restore2" to finish

In fact, the only way around this problem is to entirely duplicate the definition of the Clean task, the Restore task and the Build task, including the .Does(() => ...), but with the added interdependencies to get the required nonparallel behavior.

On top of this insurmountable problem, we have the problem that making everything parallel by default is by nature a breaking change.

This leads me to think we need a better API to opt in to parallelization with as little boilerplate as possible. Ideally without creating the need for boilerplate tasks.

Contributor

jnm2 commented Apr 28, 2017

If the only way to prevent things from running in parallel is to make them dependent on each other, there's no way to preserve priorities 1 and 2 which are very legit priorities.

This boilerplate can't help you:

Task("CleanAndRestore")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore");

Neither can this:

Task("Clean2")
    .IsDependentOn("Clean")

Task("Restore2")
    .IsDependentOn("Clean2")
    .IsDependentOn("Restore");
 // There's still nothing to indicate that "Restore" should wait for "Clean2" to finish

Task("Build2")
    .IsDependentOn("Restore2")
    .IsDependentOn("Build")
 // There's still nothing to indicate that "Build" should wait for "Restore2" to finish

In fact, the only way around this problem is to entirely duplicate the definition of the Clean task, the Restore task and the Build task, including the .Does(() => ...), but with the added interdependencies to get the required nonparallel behavior.

On top of this insurmountable problem, we have the problem that making everything parallel by default is by nature a breaking change.

This leads me to think we need a better API to opt in to parallelization with as little boilerplate as possible. Ideally without creating the need for boilerplate tasks.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

So we need to be able to say:

  • Run only Restore when it is called directly, but when invoked as a dependency and Clean is also a dependency, make sure it runs only after Clean
Contributor

jnm2 commented Apr 28, 2017

So we need to be able to say:

  • Run only Restore when it is called directly, but when invoked as a dependency and Clean is also a dependency, make sure it runs only after Clean
@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

Do we want to specify a potentially different parallel relationship between dependencies every time they show up in dependent tasks? It could lead to duplication and inconsistency, but the API is minimalist and perhaps the extra control per dependent over the parallelism of the dependencies will be needed someday:

Task("Build")
    .Does(() => { });

Task("TestA")
    .IsDependentOn("Build")
    .Does(() => { });

Task("TestB")
    .IsDependentOn("Build")
    .Does(() => { });

Task("BuildInstaller")
    .IsDependentOn("Build")
    .Does(() => { });

Task("AllTests")
    .IsDependentOnParallel("TestA", "TestB");

Task("TestAndBuildInstaller")
    .IsDependentOnParallel("AllTests", "BuildInstaller");
Contributor

jnm2 commented Apr 28, 2017

Do we want to specify a potentially different parallel relationship between dependencies every time they show up in dependent tasks? It could lead to duplication and inconsistency, but the API is minimalist and perhaps the extra control per dependent over the parallelism of the dependencies will be needed someday:

Task("Build")
    .Does(() => { });

Task("TestA")
    .IsDependentOn("Build")
    .Does(() => { });

Task("TestB")
    .IsDependentOn("Build")
    .Does(() => { });

Task("BuildInstaller")
    .IsDependentOn("Build")
    .Does(() => { });

Task("AllTests")
    .IsDependentOnParallel("TestA", "TestB");

Task("TestAndBuildInstaller")
    .IsDependentOnParallel("AllTests", "BuildInstaller");
@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

So in this API, the order of the IsDependentOn calls matters. Is that a good thing? Is it a breaking change?

Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .IsDependentOnParallel("X", "Y", "Z")
    .Does(() => { });
Contributor

jnm2 commented Apr 28, 2017

So in this API, the order of the IsDependentOn calls matters. Is that a good thing? Is it a breaking change?

Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .IsDependentOnParallel("X", "Y", "Z")
    .Does(() => { });
@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

@patriksvensson I'm interested in your thoughts.

Contributor

jnm2 commented Apr 28, 2017

@patriksvensson I'm interested in your thoughts.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

Another reason not to parallelize by default is that it messes with logs, and there is a solution for that but it means redirecting the standard output and error of everything you run, losing Windows console colors.

Contributor

jnm2 commented Apr 28, 2017

Another reason not to parallelize by default is that it messes with logs, and there is a solution for that but it means redirecting the standard output and error of everything you run, losing Windows console colors.

@daveaglick

This comment has been minimized.

Show comment
Hide comment
@daveaglick

daveaglick Apr 28, 2017

Contributor

Personally, I would prefer any sort of parallelization be opt-in as you suggest. There are just too many build scripts out there to turn on something that so fundamentally changes flow without causing major headaches.

As an alternative to your API, what if we used an .AsParallel() method to indicate that a given task is free to run it's dependencies in parallel. If you have a mix of parallel and non-parallel dependent tasks, then you can create "dummy" tasks to enable the parallelism. With this API your example would look like:

Task("Build")
    .Does(() => { });

Task("TestA")
    .Does(() => { });

Task("TestB")
    .Does(() => { });

Task("BuildInstaller")
    .Does(() => { });

// New task to run tests in parallel
Task("RunTests")
    .IsDependentOn("TestA")
    .IsDependentOn("TestB")
    .AsParallel();

// This would run "Build" followed by "RunTests", which runs only the tests in parallel
Task("AllTests")
    .IsDependentOn("Build")
    .IsDependentOn("RunTests");

// New task to run tests and build installer in parallel
// Since "AllTests" is also parallel, they would be run in parallel with "BuildInstaller"
Task("ParallelTestAndBuildInstaller")
    .IsDependentOn("AllTests")
    .IsDependentOn("BuildInstaller")
    .AsParallel();

Task("TestAndBuildInstaller")
    .IsDependentOn("Build")
    .IsDependentOn("ParallelTestAndBuildInstaller");

It's a little more verbose, but to me it's clearer what's going on and better matches the existing API.

Contributor

daveaglick commented Apr 28, 2017

Personally, I would prefer any sort of parallelization be opt-in as you suggest. There are just too many build scripts out there to turn on something that so fundamentally changes flow without causing major headaches.

As an alternative to your API, what if we used an .AsParallel() method to indicate that a given task is free to run it's dependencies in parallel. If you have a mix of parallel and non-parallel dependent tasks, then you can create "dummy" tasks to enable the parallelism. With this API your example would look like:

Task("Build")
    .Does(() => { });

Task("TestA")
    .Does(() => { });

Task("TestB")
    .Does(() => { });

Task("BuildInstaller")
    .Does(() => { });

// New task to run tests in parallel
Task("RunTests")
    .IsDependentOn("TestA")
    .IsDependentOn("TestB")
    .AsParallel();

// This would run "Build" followed by "RunTests", which runs only the tests in parallel
Task("AllTests")
    .IsDependentOn("Build")
    .IsDependentOn("RunTests");

// New task to run tests and build installer in parallel
// Since "AllTests" is also parallel, they would be run in parallel with "BuildInstaller"
Task("ParallelTestAndBuildInstaller")
    .IsDependentOn("AllTests")
    .IsDependentOn("BuildInstaller")
    .AsParallel();

Task("TestAndBuildInstaller")
    .IsDependentOn("Build")
    .IsDependentOn("ParallelTestAndBuildInstaller");

It's a little more verbose, but to me it's clearer what's going on and better matches the existing API.

@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 28, 2017

Contributor

Cool! That's a new angle.
It's not my favorite; I really dislike boilerplate tasks, it's a bit more verbose, and the most fundamental thing from an API point of view is that it's a modifier on the current task instead of a modifier on a relationship between tasks or dependencies. This looks like it means something it doesn't:

Task("B")
    .IsDependentOn("A")
    .Does(() =>
    {
        // ...
    })
    .AsParallel();

.AsParallelDependencies() would solve that.

Contributor

jnm2 commented Apr 28, 2017

Cool! That's a new angle.
It's not my favorite; I really dislike boilerplate tasks, it's a bit more verbose, and the most fundamental thing from an API point of view is that it's a modifier on the current task instead of a modifier on a relationship between tasks or dependencies. This looks like it means something it doesn't:

Task("B")
    .IsDependentOn("A")
    .Does(() =>
    {
        // ...
    })
    .AsParallel();

.AsParallelDependencies() would solve that.

@ckolumbus

This comment has been minimized.

Show comment
Hide comment
@ckolumbus

ckolumbus Nov 17, 2017

Any progress on this topic? I'd like to have such a feature to speed up large project builds.

ckolumbus commented Nov 17, 2017

Any progress on this topic? I'd like to have such a feature to speed up large project builds.

@andymac4182

This comment has been minimized.

Show comment
Hide comment
@andymac4182

andymac4182 Apr 27, 2018

I have used https://cakebuild.net/docs/fundamentals/asynchronous-tasks and RunTarget to get this working now. It isn't pretty but does allow multiple chains of tasks to be ran. If there was a nice way for logs that would be even better.

Task("__SplitHere")
    .Does(async () => {
        var dotNetTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__DotNetTasks"));
        var clientTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__ClientTasks"));
        await System.Threading.Tasks.Task.WhenAll(dotNetTasks, clientTasks);
    });

andymac4182 commented Apr 27, 2018

I have used https://cakebuild.net/docs/fundamentals/asynchronous-tasks and RunTarget to get this working now. It isn't pretty but does allow multiple chains of tasks to be ran. If there was a nice way for logs that would be even better.

Task("__SplitHere")
    .Does(async () => {
        var dotNetTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__DotNetTasks"));
        var clientTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__ClientTasks"));
        await System.Threading.Tasks.Task.WhenAll(dotNetTasks, clientTasks);
    });
@jnm2

This comment has been minimized.

Show comment
Hide comment
@jnm2

jnm2 Apr 27, 2018

Contributor

@andymac4182 Here's an interim solution for the log issue that might interest you: #156 (comment)

Contributor

jnm2 commented Apr 27, 2018

@andymac4182 Here's an interim solution for the log issue that might interest you: #156 (comment)

@mabead

This comment has been minimized.

Show comment
Hide comment
@mabead

mabead Sep 11, 2018

I pretty much liked the suggestion of @andymac4182 to use RunTarget. I tried it out and It works fine with the exception that each call to RunTarget starts a new Cake invocation from scratch. It therefore re-executes the Setup code. Also, it don't get a clean summary of my script at the end like the following:

Task                          Duration
--------------------------------------------------
Setup                         00:00:00.0177014
Restore                       00:00:06.5128068
Build                         00:00:19.4424271
PackageLambda                 00:00:08.3597148
DockerBuild                   00:00:55.6567670
GenerateClient                00:00:36.4401349
UnitTest                      00:01:55.2787180
Publish                       Skipped
Tag                           00:00:00.0032837
--------------------------------------------------
Total:                        00:04:01.7147065

Everything invoked through RunTarget is not listed in the summary. Is there another function that I could use to execute a target without restarting cake from the beginning and still getting a decent task summary?

mabead commented Sep 11, 2018

I pretty much liked the suggestion of @andymac4182 to use RunTarget. I tried it out and It works fine with the exception that each call to RunTarget starts a new Cake invocation from scratch. It therefore re-executes the Setup code. Also, it don't get a clean summary of my script at the end like the following:

Task                          Duration
--------------------------------------------------
Setup                         00:00:00.0177014
Restore                       00:00:06.5128068
Build                         00:00:19.4424271
PackageLambda                 00:00:08.3597148
DockerBuild                   00:00:55.6567670
GenerateClient                00:00:36.4401349
UnitTest                      00:01:55.2787180
Publish                       Skipped
Tag                           00:00:00.0032837
--------------------------------------------------
Total:                        00:04:01.7147065

Everything invoked through RunTarget is not listed in the summary. Is there another function that I could use to execute a target without restarting cake from the beginning and still getting a decent task summary?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment