RFC0023-Native-Support-for-Concurrent-Programming-in-PowerShell

[BrucePay]

Feedback for https://github.com/PowerShell/PowerShell-RFC/blob/master/1-Draft/RFC0023-Native-Support-for-Concurrent-Programming-in-PowerShell.md

[SteveL-MSFT]

Can we also have an Input stream for PSTasks?

[DerpMcDerp]

1. The markdown formatting of the cmdlets is really screwed up
2. I think the -Arguments parameter is a typo that should have been -ArgumentList
3. -Timeout should have a -TimeoutMSec alias
4. The documentation says the task is identified by an integer id but the C# code shows string TaskID
5. The properties on PSTaskInfo look like they shouldn't have public setters

Also, I'm skeptical of the utility of the -FunctionName parameter set when you can use { Foo @args }. There was even a prior PowerShell proposal to do something like:

function foo([cmd]$Function) {
	& $Function -Stuff
}

foo { write-host @args }
foo write-host

where [cmd] will accept [scriptblock] and [Management.Automation.CommandInfo] and will convert [string]s into [scriptblock]s (if it refers to a function) or [System.Management.Automation.CmdletInfo] (if it refers to a cmdlet).

Which is the general solution to removing these kinds of [scriptblock]/[string] parameter sets.

[PaulHigin]

Overall this looks great.

This specifies three new cmdlets. I propose two more:
Get-PSTask - Lists currently running/queued tasks
Debug-PSTask - Attaches script debugger to running task

Can Tasks be built into the PowerShell language? Something like a foreachparallel that returns TaskInfo[]

Parameter name -NoThrottle to start task immediately confusing. Maybe -RunNow?

TaskState should include "Broken" or "Error" or "NotResponding" if it cannot be stopped (because it is hung in a native call or such). It would be nice to get a script stack trace in this case (Get-PSTaskStack?).

TaskInfo should have an "AggregateData" collection that displays all data stream items in correct time order.

$PSTaskQueueLimit instead of $PSThrottleLimit?

It would be nice if Start-Task took a -SessionTemplate argument that is an IntialSessionState object so that multiple tasks can be created with identical environments.

Please add examples for how -On* actions work.

It would be nice if TaskInfo included "BeginTime" and "EndTime" properties.

[iSazonov]

If we believe PoshRSJob is a very popular the RFC is right way.
Statistics - 4000 - All downloads (and ~70 per week).

As system administrator I have always tried to avoid the use of PoshRSJob due to its complexity (more internal), overheads and need to install.
If we integrate the functionality it is possible it will be more popular.
I would prefer other ways of concurrency insertion in PowerShell language. Although "factory" from the RFC can be used in them too.

It is possible we need add TaskState = Sleep/Blocked - ex., task is waiting a native call, Start-Sleep...

Loading modules in each task context is large overhead. Need common context. Maybe for a task pool.

Task Stream operation - split/union. If "union" of output streams is Wait-PSTask what about "split" input streams? What with other streams (error, warning, information)?

[powercode]

I think there is a strong case to be made for a cmdlet that just forwards the output from all stream in the parallel runspaces to the matching streams in the originating runspace.

Such a cmdlet (or extension of ForEach-Object) will keep the programming model, and just make it an implementation detail that the runspaces are invoked in parallel.

This proposal leaves it up to the user to collect the streams from the tasks. Personally, I think that could be done better in the implementation of a cmdlet like Invoke-Parallel or Foreach-Object -parallel.

[nightroman]

@powercode

FWIW, I think this approach is implemented by SplitPipeline with one caveat. It "extends" and makes parallel not ForEach-Object but rather this less known PowerShell construct:

    Get-SomeData | . { process { "process $_" } }

which after replacing the operator . becomes

    Get-SomeData | Split-Pipeline { process { "process $_" } }

[iSazonov]

Invoke-Parallel or Foreach-Object -parallel.

I like it more than cloning PoshRSJob. But it requires enhancing "environment". We should look at the capabilities of SCOR (System Center Orchestrator 2012).

[powercode]

@iSazonov Can you expand on that?

[iSazonov]

We already have Foreach -Parallel in Workflow. It is based on Jobs. Maybe if we replace "job factory" with "runspace factory" we could get more robust, fast and lightweight solution bearing in mind that "workflow environment" already exists (although lose job persistence).
In Powershell we have no ready "environment" for Foreach -Parallel. So we need a simple and clear idea. This idea can be that used in SCOR. We can see how SCOR manage "streams" and combine "activity" in "runbooks" on the screenshots.
There we can get three levels of parallelism:

1. Run same runbook several times
2. Split work in "runbook" by some "activity" on several flows
3. Get parallel processes inside "activity"
   Something we already have for this.

[BatmanAMA]

This is sorely needed, thanks for starting it off. I have a few comments:

• I feel like the timeout argument on wait should be a timespan object to allow date math. This still allows string representations for ad-hoc commandline use, but is significantly cleaner then simple milliseconds for more complex scenarios (look at the timetolive argument on add-dnsserverresourcerecorda for an example)

• I would like to be able to use this via pipeline like I do with Invoke-Parallel currently. I can certainly just wrap it in foreach-object, but that feels much less elegant.

[jhoneill]

I have put my attempt at solving this "Start-Parallel" on the PowerShell gallery. This uses the run space factory to run the same block of script multiple times with different parameters and return the results from all the runspaces.
I'm unclear what the exact meaning of "The execution environment for a throttled task will be equivalent to runspace pool runspace. " is, but tasks appear to have their own run-spaces with the isolation which that provides and the advantage over what I've been able to implement of having read access to variables in the calling session with $using: . start-parallel is the equivalent of the example
1..10 | foreach {Start-PSTask {param($x) "x is $x"} -Arguments $_} | wait-pstask
i.e. it won't start many instances of different tasks before waiting for completion. If multiple tasks are run something in the returned output would have to then identify how the returned object is to be processed.

[fMichaleczek]

Hi,

I worked on the same features, you will find a POC powershell class and some examples.

https://gist.github.com/fmichaleczek/35d65770186972c9d45a255566543b76

This PS class use the TaskFactory.FromAsync method (and some delegate and reflection to use it).

https://msdn.microsoft.com/en-US/library/system.threading.tasks.taskfactory.fromasync(v=vs.110).aspx

Would it make sense to have Await-PSTask instead of wait? Such that a PSTask is worth awaiting.

[SteveL-MSFT]

@kasper3 await isn't an approved verb though...

[markekraus]

@BrucePay Looking at the PSTaskInfo, I'm curious how the stream List<PSObject> will work in concert with the underlying PSDataCollection to prevent a thread lock or race conditions with a user adding or removing items to/from the stream lists. Or how that would consume the PSDataCollection on-demand and deal with user additions/removals. I understand the pain point of providing a consumable collection to users, I'm just curious how these lists would be implemented in a way that wont lead to confusion. for a savvy user modifying the stream lists.

Also, I'd like to possibly consider some way of simplifying a producer-consumer pattern for tasks. Often, I'm having to run multiple producer threads that feed into multiple consumer threads. I'm doing this with a fake pipeline that uses a BlockingCollection<PSObject>. The way I do this is to wrap the provided scriptblock and then do something like $Queue.GetConsumingEnumerable() | & $ScriptBlock. I then take the output streams into another BlockingCollection<PSObject> that is then consumed by either the parent thread or used as the input to another runspace.

The goal is to have a kind of PowerShell pipeline for runspaces where PSObjects flow between runspaces and/or the parent. Example uses are having producers generate lists of paths and consumers performing ACL operations on the paths, or producers listing mailboxes and consumers generating lists of inbox rules.

Maybe that doesn't need to be included initially, but it's something I'd like considered.