v2.3

ParallelKit 2.3

After updating, start a new PowerShell window, so nothing from the older copy is left in your session.

New

• Faster with short programs and short jobs. The dispatch and runspace loops now wake as soon as a job finishes, instead of sleeping between checks. One slot running a very short program went from about 106 ms per program to about 19 ms, and a 20 ms runspace job from about 102 ms to 23 ms. (#31)
• A time limit per job in Invoke-PkProcessDispatch. -TimeoutSeconds sets it for every job, and a TimeoutSeconds key on a job overrides it (0 means no limit). A job that runs over has its program and everything it started killed, and goes through -JobFinalize with $job['TimedOut'] = $true. Invoke-PkProcessBatch also takes a TimeoutSeconds key per command now. (#32)
• Arguments takes a list, one argument per element, quoted for you: Arguments = @($in, $out) works with spaces in the paths. This works for every function that takes Arguments. (#34)
• Each engine accepts the other's names for its limits. -MaxParallel/-MaxConcurrent and -DrainTimeoutSeconds/-AbortTimeoutSeconds work on every engine. (#35)

Changes you might notice

• Callbacks that run on every loop pass run more often with fast jobs: -PollCallback, -AbortPollCallback and -GroupReleaseCheck. (#31)
• A retry queued from -JobFinalize can start within milliseconds of the failure, instead of about 125 ms later. (#31)
• Dispatchers sharing one SemaphoreSlim gain less speed than one with its own limit, because they pause to take turns. (#31)
• A job field named TimeoutSeconds now sets the kit's time limit, and a value that isn't a number fails that job's start. TimedOut, TimeLimit and StartFailureReported are now reserved job keys. (#32)
• Repeated start failures in a batch print once. If the same program can't start for many commands, the first failure prints in full and a count follows at the end, so a message handler that counted one line per command sees fewer. Each command still gets its StartError. (#33)
• A list in Arguments is no longer joined as-is. A list of command-line pieces that are already quoted (@('"C:\My Files\a.png"', 'out.jxl')) now passes the quote characters on to the program. Pass such a command line as one string instead. $null elements are left out, by ConvertTo-PkArgumentString too. (#34)
• Arguments holding & | < > ^ ( ) , ; are quoted, by ConvertTo-PkArgumentString and so by a list in Arguments. A .bat or .cmd program now gets such an argument as one argument instead of cmd.exe acting on it (a&ver used to run ver). Programs that follow the usual Windows rules see no change. A list passed to cmd.exe /c now has those arguments quoted, a program that reads its raw command line sees the quotes, and a trailing backslash in such an argument reaches a batch file doubled. (#36)
• Unknown arguments are now an error on the seven public functions that used to ignore them, so a typo like Initialize-ParallelKit -MaxParallel 8 no longer passes silently. (#35)
• Invoke-PkRunspaceJobs -Abort ... is now an error. It used to resolve to -AbortWaitMessage, but the new -AbortTimeoutSeconds alias makes it ambiguous. Write -AbortWaitMessage in full. (#35)

Fixed

• Invoke-PkProcessBatch -TimeoutSeconds below 1 meant no limit. It was rounded to a whole number, so 0.4 became 0 and nothing was ever timed out. It now takes fractions. (#32)
• $OFS could change a command line. A list in Arguments was joined with $OFS, so a script that set it got different arguments. (#34)
• A hashtable in Arguments reached the program as the text System.Collections.Hashtable. It's now an error before anything starts: Arguments must be one string or a list of arguments. So is a list inside the list, which used to become one argument joined with $OFS. (#34, #37)
• A file object in Arguments became just its name on Windows PowerShell 5.1. A Get-ChildItem result reached the program as a.png instead of its full path, so the program looked for it in the wrong folder. File and folder objects are now passed as their full path on 5.1 and 7, and to ConvertTo-PkArgumentString. (#37)
• Invoke-PkProcessBatch -MaxConcurrent 0 failed with an error naming Invoke-PkProcessDispatch. It now names the batch. (#35)

v2.2

ParallelKit 2.2

After updating, start a new PowerShell window, so nothing from the older copy is left in your session.

Changes you might notice

• The kit now loads as an in-memory module. You still copy one file and dot-source it, and nothing is installed. Get-Module now lists ParallelKit, and internal helpers such as Invoke-PkProcessDispatchCore can no longer be called or show up in tab completion. Only the functions in the README are public. (#27)
• Preference variables set at script level no longer reach the kit's own code. Your own callbacks still see them. (#27)

Fixed

• Callbacks no longer collide with the kit's variables. A callback written without .GetNewClosure() could pick up the kit's own $results, $state, $job or $TimeoutSeconds instead of yours and silently give wrong results. A plain { ... } now sees your variables, on the main thread (#27) and inside a worker runspace (#28). You still need a hashtable for state a callback changes.
• Paths like \tools and C:tools now follow your current location. They used to follow the PowerShell process's drive and folder, so the program failed to start or could run in the wrong place. (#29)
• A non-terminating error in a worker that finishes while an abort waits for it now prints Runspace error: ..., as it does at any other time, so a partial result doesn't look fine. (#29)
• -SharedFunctionNames can share a module's private functions when the script that loaded the kit calls into that module. (#27)

v2.1

ParallelKit 2.1

After updating, start a new PowerShell window: a session that already loaded 2.0 refuses to load 2.1 and tells you so.

Changes you might notice

• Programs start in your current location. Before, they ran in the PowerShell process's working folder, which cd and Set-Location never change. So after cd D:\work and .\script.ps1, programs started wherever the console had opened. A relative WorkingDirectory, a FilePath like .\tools\tool.exe, and paths on ~ or a New-PSDrive drive are now read the way PowerShell reads them. One case gets worse: a .bat or .cmd program run while your current location is a network share (\\server\share) falls back to C:\Windows, so give those a local WorkingDirectory. (#22)
• Each runspace job starts clean. Every job starts in your current location, in a fresh scope. A folder change or a variable from an earlier job, $ErrorActionPreference included, no longer carries over to the next job on the same runspace. Inside a worker, $MyInvocation now describes a scriptblock call. (#23)
• Abort messages say what actually happened. Ctrl+Break and Request-PkAbort are no longer announced as Ctrl+C, and Get-PkAbortTriggerLabel returns "Ctrl+Break" or "Request-PkAbort" for them. Force-stop messages give the real reason instead of always "did not finish within 60s". (#20)
• CrashedCount also counts workers that threw while an abort waited for them, and the result of Invoke-PkRunspaceJobs has a new Crashed property. (#24)

Fixed

• Pressing Ctrl+C again now skips the wait. It used to be held back until the 60-second drain ended, and then it stopped the script, losing a batch's result. Now the second press stops running jobs straight away and your script carries on; a third press stops the script. (#20)
• Closing the window after Ctrl+C now runs -JobAbortCleanup before Windows ends the process. (#20)
• A -ResultHandler written without .GetNewClosure() that did $results.Add(...) no longer loops forever. (#19)
• Output you drop in -OnComplete ($result.Stdout = $null) is now freed as the batch goes, instead of being held until it returns. (#25)

New

• -JobCrashedHandler { param($err, $job) } gets the job that crashed. One-parameter handlers work as before, and a handler that throws is now reported instead of swallowed. (#24)
• The README now covers crashes: what counts as a worker crash, how to return an error object from a worker, and why not to call exit in one. (#24)
• The test suite now sends real Ctrl+C, Ctrl+Break and window-close events, on every change, on 5.1 and 7. (#21)

v2.0

ParallelKit 2.0

Major changes

• Results are objects, not hashtables. Everything the kit returns (batch results, Invoke-PkProcessWithTimeout, the dispatcher, Invoke-PkRunspaceJobs, worker-safety findings) is now a [pscustomobject], so $batch.Results | Format-Table shows proper columns. $r.ExitCode works as before; $r['ExitCode'], .ContainsKey() and adding fields to a result don't. Your own command and job hashtables, and what Start-PkProcess returns, are still hashtables.
• The worker-safety check is on by default. Invoke-PkRunspaceJobs checks worker code before it starts anything and refuses code that breaks the worker rules. Pass -CheckWorkerSafety:$false to turn it off.
• Scriptblocks can't be passed as job values. A scriptblock made on the main thread and run inside a worker can corrupt, crash or hang the run, so it's now refused, naming the job and key. Pass the code as text and rebuild it in the worker with [scriptblock]::Create($text).

Cmdlets are allowed in worker code

v1.0's no-cmdlets rule was a misdiagnosis. The real cause of the hangs is a scriptblock running on another runspace's thread, so there are now two worker rules instead of three, and cmdlets are fine. See #5 for the details and #10 for the checker fixes that followed.

Fixed

• Scripts run from a prompt. When a script dot-sourced the kit and was run as .\script.ps1, Invoke-PkProcessBatch failed every command, and any .GetNewClosure() callback calling a kit function failed too. Running with powershell -File worked, which hid it. This dates back to v1.0. The kit's functions are now also defined globally.
• Stopping a program stops what it started. A timeout, an overrun in Invoke-PkProcessWithTimeout, or an abort's force-stop now kills the program's whole process tree, so its child processes no longer keep running and holding its output open.
• Closing the window is announced as stopping current jobs, rather than wrongly saying it waits for them.
• Constrained Language Mode (AppLocker/WDAC-locked machines) now fails at load with a message saying why, instead of a confusing type error.

New

• Invoke-PkProcessBatch -TimeoutSeconds: kill a program that runs too long. Its result has TimedOut = $true.
• -OutputEncoding (batch, Invoke-PkProcessWithTimeout, Start-PkProcess, or per command): decode a program's output as UTF-8 or another code page. Without it, output stays in the console's code page, which garbles UTF-8 output on a default console.
• ConvertTo-PkArgumentString: build a correctly quoted command line from a list of arguments, including paths with spaces, quotes and trailing backslashes.
• Stop-PkProcessTree: kill one or more programs and everything they started.
• Get-Help works for every public function: Get-Help Invoke-PkProcessBatch -Full.
• PowerShell 7 on Windows is supported alongside 5.1. The full test suite runs on both for every change.

Docs

• The file header now carries the version and licence notice, so a copy in your project says which version it is.
• The README covers which hosts work (consoles and editor terminals, not the PowerShell ISE), the FAQ answers common questions, and the defaults are listed where each option is introduced.

Upgrading from 1.0

• Replace $r['Field'] with $r.Field on anything the kit returns.
• If worker code is now refused, the message says why. Fix it, or pass -CheckWorkerSafety:$false.
• Scriptblocks in job values: pass the text and use [scriptblock]::Create in the worker.
• You can now use cmdlets in worker code, so .NET workarounds written for the old rule can go.

v1.0

First release of ParallelKit: run command-line programs, or blocks of PowerShell code, in parallel from Windows PowerShell 5.1, and stop cleanly on Ctrl+C. One file, nothing to install.

• Invoke-PkProcessBatch runs a list of external programs, a set number at a time, with their output captured.
• Invoke-PkProcessDispatch does the same for work that comes in groups, with retries and a step when each group finishes.
• Invoke-PkRunspaceJobs runs PowerShell worker code in parallel runspaces. Test-PkWorkerSafety and -CheckWorkerSafety check worker code against the kit's rules.
• Invoke-PkProcessWithTimeout, Start-PkProcess, Wait-PkProcess and Complete-PkProcess handle single programs.
• Invoke-PkRun makes Ctrl+C graceful: running work finishes, then the run stops. It also keeps the PC from idle-sleeping and turns off QuickEdit for the length of the run.
• Programs the script started are killed with it, even if the script is force-closed.

Download ParallelKit.ps1 below and dot-source it. Requires Windows PowerShell 5.1. See the README for usage.