Add support for Parallel Tasks (using PowerShell 3.0 Workflows as back-end) #97

Closed
maximpashuk opened this Issue Jan 10, 2014 · 14 comments

Comments

Projects
None yet
9 participants

Psake for now based on Powershell 2.0 and does not support parallel tasks at all.

Parallel tasks is very desired feature, making psake really first-class build system.
The only feature in PowerShell that support parallelism is workflows.
Workflows available since PowerShell 3.0.

So there only way I see to support parallel tasks is to migrating to PowerShell 3.0 and using workflows as back-end abstraction for parallel tasks.

Owner

damianh commented Mar 31, 2014

What would the output of those parallel tasks looks like in the console / stdout? Pretty certain this would mess up tools that process stdout (i.e. TeamCity) and the build log. From http://blogs.technet.com/b/heyscriptingguy/archive/2012/12/26/powershell-workflows-the-basics.aspx

In which order do you think the data will be returned?

You can’t tell! You can run this workflow a number of times and the data may be returned in a different order each time you run it!

If you are running workflow activities in parallel there are no guarantees as to the order in which data will be returned. You cannot assume that one piece of data will be returned before another."

Yeah I'm against this from a build script perspective... @whut , thoughts?

scryb3 commented Jun 11, 2014

That seems like an easy to address objection, just use a message queue to pass logging back to the master/controller and prefix any log events with the child process name+ID or something...

With the "depends" logic it's fairly easy to multi-thread if it makes sense, still not sure when it would make sense though, use case needs more refining.

Contributor

dmarlow commented Dec 19, 2014

I could use parallel task. Say I need to run grunt/gulp and upload files, those could be done in parallel to build/deploy.

giggio commented Jan 13, 2015

I like this idea a lot.

giggio commented Jan 13, 2015

Output could be:

Task A: whatever output
Task B: whatever output

Could really use this can think of many instances that we can use it

@damianh damianh added the UpForGrabs label Jan 13, 2015

Owner

damianh commented Jan 13, 2015

Well does someone want to try implementing it so? 😉

vors commented Feb 8, 2015

Workflows can be overkill.
Maybe can be achieved with https://github.com/RamblingCookieMonster/Invoke-Parallel

Contributor

sweeneyrobb commented Jan 30, 2016

are there any thoughts on how one would build and/or compose parallel tasks? Task Default -parallelDepends t1, t2, t3?

Contributor

sweeneyrobb commented Jan 31, 2016

i just had a thought about this. without proper testing, i don't believe that the $psake global and anything bolted onto it (e.g. $psake.context) is safe for concurrency. One might need to copy the variable and pass into an invocation of Invoke-Task.

Maybe the easiest thing to do is to created nested calls to Invoke-psake inside of a Start-Job. i added a PR #152 to show what an example might look like at a Task level.

Owner

gep13 commented Feb 27, 2016

@sweeneyrobb this seems like a sensible approach to me, and fits in with the existing infrastructure.

Contributor

sweeneyrobb commented Feb 27, 2016

@gep13 thanks! I'll put together another PR. Thanks for the response.

@gep13 thanks, I think your solution is well enough/
Start-Job\Wait-Job\Receive-Job combination allows to run tasks as parallel.

I close my issue, I think it is already resolved.

Owner

gep13 commented Mar 16, 2016

@maximpashuk glad to hear that you were able to make progress and thanks for following up by closing the issue.

@gep13 gep13 removed the UpForGrabs label Mar 16, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment