From 193ecf722bc2d643125566d6b9a5df0114171322 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Tue, 28 Jul 2020 10:31:22 -0700 Subject: [PATCH 01/17] Adds doc --- .../write-progress-across-multiple-threads.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md new file mode 100644 index 000000000000..a065ff13372b --- /dev/null +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -0,0 +1,15 @@ +--- +title: Displaying progress across multiple threads +description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel +ms.date: 07/27/2020 +--- + +# Writing Progress across multiple threads with Foreach Parallel + +Starting in PowerShell 7.0, the ability to work in multiple threads simultaneously is possible using +the **Parallel** parameter in the `Foreach-Object` cmdlet. Monitoring the progress of these threads +can be a challenge though. Normally, you can monitor the progress of a process using +`Write-Progress`. However, since PowerShell uses a separate runspace for each thread when using +**Parallel**, reporting the progress back to the host isn't as straight forward as normal use of +`Write-Progress`. + From 4d95a529a0c012eb6221e4e90ceadddd8b3fc36c Mon Sep 17 00:00:00 2001 From: chasewilson Date: Tue, 28 Jul 2020 10:31:22 -0700 Subject: [PATCH 02/17] Adds doc --- .../write-progress-across-multiple-threads.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md new file mode 100644 index 000000000000..a065ff13372b --- /dev/null +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -0,0 +1,15 @@ +--- +title: Displaying progress across multiple threads +description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel +ms.date: 07/27/2020 +--- + +# Writing Progress across multiple threads with Foreach Parallel + +Starting in PowerShell 7.0, the ability to work in multiple threads simultaneously is possible using +the **Parallel** parameter in the `Foreach-Object` cmdlet. Monitoring the progress of these threads +can be a challenge though. Normally, you can monitor the progress of a process using +`Write-Progress`. However, since PowerShell uses a separate runspace for each thread when using +**Parallel**, reporting the progress back to the host isn't as straight forward as normal use of +`Write-Progress`. + From 27f8bb3c4dc59f4179c2faebf8e9fa7ab8b164e1 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Sun, 2 Aug 2020 18:29:01 -0700 Subject: [PATCH 03/17] Finishes article and adds it to toc --- .vscode/settings.json | 3 + .../write-progress-across-multiple-threads.md | 253 +++++++++++++++++- reference/docs-conceptual/toc.yml | 2 + 3 files changed, 252 insertions(+), 6 deletions(-) create mode 100644 .vscode/settings.json diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 000000000000..79e520aee0b5 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "powershell.codeFormatting.addWhitespaceAroundPipe": true +} \ No newline at end of file diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index a065ff13372b..18de6bf73fea 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -1,15 +1,256 @@ --- title: Displaying progress across multiple threads description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel -ms.date: 07/27/2020 +ms.date: 08/02/2020 --- # Writing Progress across multiple threads with Foreach Parallel Starting in PowerShell 7.0, the ability to work in multiple threads simultaneously is possible using -the **Parallel** parameter in the `Foreach-Object` cmdlet. Monitoring the progress of these threads -can be a challenge though. Normally, you can monitor the progress of a process using -`Write-Progress`. However, since PowerShell uses a separate runspace for each thread when using -**Parallel**, reporting the progress back to the host isn't as straight forward as normal use of -`Write-Progress`. +the **Parallel** parameter in the [Foreach-Object](/powershell/reference/7.0/Microsoft.PowerShell.Core/Foreach-Object) +cmdlet. Monitoring the progress of these threads can be a challenge though. Normally, you can +monitor the progress of a process using [Write-Progress](/powershell/reference/7.0/Microsoft.PowerShell.Utility/Write-Progress). +However, since PowerShell uses a separate runspace for each thread when using **Parallel**, +reporting the progress back to the host isn't as straight forward as normal use of `Write-Progress`. +## Using a synced hashtable to track progress + +When writing the progress from multiple threads, tracking becomes difficult because when running +parallel processes in PowerShell, each process has it's own runspace. To get around this, you can +use a [synchronized hashtable](/dotnet/api/system.collections.hashtable.synchronized). A synced +hashtable is a thread safe data structure that can be edited by multiple threads simultaneously +without throwing an error. + +### Set up + +One of the downsides to this approach is it takes a, somewhat, complex set up to ensure everything +runs without error. + +```powershell +#example workload +$dataset = @( + @{ + Id = 1 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 2 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 3 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 4 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 5 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } +) + +#Create a hashtable for progress. +#keys should be ID's of the progresses +#in this example i'l just use int 1-10 +$origin = @{} +$dataset | Foreach-Object {$origin.($_.id) = @{}} + +#create synced hashtable +$sync = [System.Collections.Hashtable]::Synchronized($origin) +``` + +This section creates three different data structures, for three different purposes. + +The `$dataSet` variable stores an array of hashtables that is used to coordinate the next steps +without the risk of being modified. When looping through an object in PowerShell, if the object +being looped through is modified, PowerShell will throw an error. So, keeping the object being +looped through, separate from the objects being modified is a must. The `Id` key in each hashtable +is the identifier for a mock process. The `Wait` key simulates the workload of each mock process to +track. + +The `$origin` variable stores a nested hashtable with each key being one of the mock process id's. +Then, it is used to hydrate the synchronized hashtable stored in the `$sync` variable. The `$sync` +variable is responsible for reporting the progress back to the parent process, PowerShell, which +displays the progress. + +### Running the processes + +This section runs the multi threaded processes and creates some of the output used to display +progress. + +```powershell +$job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { + $syncCopy = $using:sync + $process = $syncCopy.$($PSItem.Id) + + $process.Id = $PSItem.Id + $process.Activity = "$($PSItem.Id) starting" + $process.Status = "Processing" + + #fake workload that takes x amount of time to complete + start-sleep -Milliseconds ($PSItem.wait*5) + + #process. update activity + $process.Activity = "$($PSItem.id) processing" + foreach ($percent in 1..100) + { + #update process on status + $process.Status = "Handling $percent/100" + $process.PercentComplete = (($percent / 100) * 100) + + #fake workload that takes x amount of time to complete + Start-Sleep -Milliseconds $PSItem.Wait + } + + #end + $process.Completed = $true +} +``` + +The mock processes are sent to `Foreach-Object` and started as jobs. The **ThrottleLimit** is set to +**3** to highlight running multiple processes in a queue. The jobs are stored in the `$job` variable +and allows us to know when all the processes have finished later on. + +When using the `using:` statement to reference a parent scope variable in PowerShell, you can't use +expressions to make it dynamic. For example, if you tried to create the `$process` variable like +this, `$process = $using:sync.$($PSItem.id)`, you would get an error stating you can't use +expressions there. So, we create the `$syncCopy` variable to be able to reference and modify the +`$sync` variable without the risk of it failing. + +Next, we build out a hashtable to represent the progress of the process currently in the loop using +the `$process` variable by referencing the synchronized hashtable keys. The **Activity** and the +**Status** keys are used as parameter values for `Write-Progress` to display the status of a given +mock process in the next section. + +The `foreach` loop is just a way to simulate the process working and is randomized based on the +`$dataSet` **Wait** attribute to set `Start-Sleep` using milliseconds. How you calculate the +progress of your process may vary. + +### Displaying the progress of multiple processes + +Now that the mock processes are running as jobs, we can start to write the processes progress to the +PowerShell window. + +```powershell +while($job.State -eq 'Running') +{ + #foreach key that is available + $sync.Keys | Foreach-Object { + #if key is not defined, ignore + if(![string]::IsNullOrEmpty($sync.$_.keys)) + { + #create param + $param = $sync.$_ + + #execute write-progress + Write-Progress @param + } + } + + #wait to refresh to not overload gui + Start-Sleep -Seconds 0.1 +} +``` + +The `$job` variable contains the parent **job** and has a child **job** for each of the mock +processes. While any of the child jobs are still running, the parent job **State** will remain +"Running". This allows us to reliable use the `while` loop to continually update the progress of +every process until all processes are finished. + +Within the while loop, we loop through each of the keys in the `$sync` variable. Since this is +a synchronized hashtable, it is constantly updated but can still be accessed without throwing any +errors. + +There is a check to ensure that the process being reported is actually running using the +`IsNullOrEmpty()` method. If the process hasn't been started, the loop won't report on it and move +on to the next until it gets to a process that has been started. If the process is started, the +hashtable from the current key is used to splat the parameters to `Write-Progress`. + +### Full example + +```powershell +# Example workload +$dataset = @( + @{ + Id = 1 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 2 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 3 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 4 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 5 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } +) + +# Create a hashtable for process. +# Keys should be ID's of the processes +$origin = @{} +$dataset | Foreach-Object {$origin.($_.id) = @{}} + +# Create synced hashtable +$sync = [System.Collections.Hashtable]::Synchronized($origin) + +$job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { + $syncCopy = $using:sync + $process = $syncCopy.$($PSItem.Id) + + $process.Id = $PSItem.Id + $process.Activity = "$($PSItem.Id) starting" + $process.Status = "Processing" + + # Fake workload start up that takes x amount of time to complete + start-sleep -Milliseconds ($PSItem.wait*5) + + # Process. update activity + $process.Activity = "$($PSItem.id) processing" + foreach ($percent in 1..100) + { + # Update process on status + $process.Status = "Handling $percent/100" + $process.PercentComplete = (($percent / 100) * 100) + + # Fake workload that takes x amount of time to complete + Start-Sleep -Milliseconds $PSItem.Wait + } + + # Mark process as completed + $process.Completed = $true +} + +while($job.State -eq 'Running') +{ + $sync.Keys | Foreach-Object { + # If key is not defined, ignore + if(![string]::IsNullOrEmpty($sync.$_.keys)) + { + # Create parameter hashtable to splat + $param = $sync.$_ + + # Execute Write-Progress + Write-Progress @param + } + } + + # Wait to refresh to not overload gui + Start-Sleep -Seconds 0.1 +} +``` + +## Related Links + +- [about_Jobs](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Jobs) +- [about_Scopes](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Scopes) +- [about_Splatting](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Splatting) diff --git a/reference/docs-conceptual/toc.yml b/reference/docs-conceptual/toc.yml index 2bcb6ef4d187..6ff085ff97ef 100644 --- a/reference/docs-conceptual/toc.yml +++ b/reference/docs-conceptual/toc.yml @@ -73,6 +73,8 @@ href: learn/deep-dives/everything-about-null.md - name: Everything you want to know about ShouldProcess href: learn/deep-dives/everything-about-shouldprocess.md + - name: Write-Progress while multithreading + href: learn/deep-dives/write-progress-across-multiple-threads.md - name: PowerShell remoting items: - name: Just Enough Administration (JEA) From b0bb2bd5ff125d242f73cd3e93bdf8d2da7b1003 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Tue, 28 Jul 2020 10:31:22 -0700 Subject: [PATCH 04/17] Adds doc --- .../write-progress-across-multiple-threads.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md new file mode 100644 index 000000000000..a065ff13372b --- /dev/null +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -0,0 +1,15 @@ +--- +title: Displaying progress across multiple threads +description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel +ms.date: 07/27/2020 +--- + +# Writing Progress across multiple threads with Foreach Parallel + +Starting in PowerShell 7.0, the ability to work in multiple threads simultaneously is possible using +the **Parallel** parameter in the `Foreach-Object` cmdlet. Monitoring the progress of these threads +can be a challenge though. Normally, you can monitor the progress of a process using +`Write-Progress`. However, since PowerShell uses a separate runspace for each thread when using +**Parallel**, reporting the progress back to the host isn't as straight forward as normal use of +`Write-Progress`. + From 0d5a811d61fe9b01f11eded153f1e16eeb24f553 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Sun, 2 Aug 2020 18:29:01 -0700 Subject: [PATCH 05/17] Finishes article and adds it to toc --- .vscode/settings.json | 3 + .../write-progress-across-multiple-threads.md | 253 +++++++++++++++++- reference/docs-conceptual/toc.yml | 2 + 3 files changed, 252 insertions(+), 6 deletions(-) create mode 100644 .vscode/settings.json diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 000000000000..79e520aee0b5 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "powershell.codeFormatting.addWhitespaceAroundPipe": true +} \ No newline at end of file diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index a065ff13372b..18de6bf73fea 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -1,15 +1,256 @@ --- title: Displaying progress across multiple threads description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel -ms.date: 07/27/2020 +ms.date: 08/02/2020 --- # Writing Progress across multiple threads with Foreach Parallel Starting in PowerShell 7.0, the ability to work in multiple threads simultaneously is possible using -the **Parallel** parameter in the `Foreach-Object` cmdlet. Monitoring the progress of these threads -can be a challenge though. Normally, you can monitor the progress of a process using -`Write-Progress`. However, since PowerShell uses a separate runspace for each thread when using -**Parallel**, reporting the progress back to the host isn't as straight forward as normal use of -`Write-Progress`. +the **Parallel** parameter in the [Foreach-Object](/powershell/reference/7.0/Microsoft.PowerShell.Core/Foreach-Object) +cmdlet. Monitoring the progress of these threads can be a challenge though. Normally, you can +monitor the progress of a process using [Write-Progress](/powershell/reference/7.0/Microsoft.PowerShell.Utility/Write-Progress). +However, since PowerShell uses a separate runspace for each thread when using **Parallel**, +reporting the progress back to the host isn't as straight forward as normal use of `Write-Progress`. +## Using a synced hashtable to track progress + +When writing the progress from multiple threads, tracking becomes difficult because when running +parallel processes in PowerShell, each process has it's own runspace. To get around this, you can +use a [synchronized hashtable](/dotnet/api/system.collections.hashtable.synchronized). A synced +hashtable is a thread safe data structure that can be edited by multiple threads simultaneously +without throwing an error. + +### Set up + +One of the downsides to this approach is it takes a, somewhat, complex set up to ensure everything +runs without error. + +```powershell +#example workload +$dataset = @( + @{ + Id = 1 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 2 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 3 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 4 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 5 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } +) + +#Create a hashtable for progress. +#keys should be ID's of the progresses +#in this example i'l just use int 1-10 +$origin = @{} +$dataset | Foreach-Object {$origin.($_.id) = @{}} + +#create synced hashtable +$sync = [System.Collections.Hashtable]::Synchronized($origin) +``` + +This section creates three different data structures, for three different purposes. + +The `$dataSet` variable stores an array of hashtables that is used to coordinate the next steps +without the risk of being modified. When looping through an object in PowerShell, if the object +being looped through is modified, PowerShell will throw an error. So, keeping the object being +looped through, separate from the objects being modified is a must. The `Id` key in each hashtable +is the identifier for a mock process. The `Wait` key simulates the workload of each mock process to +track. + +The `$origin` variable stores a nested hashtable with each key being one of the mock process id's. +Then, it is used to hydrate the synchronized hashtable stored in the `$sync` variable. The `$sync` +variable is responsible for reporting the progress back to the parent process, PowerShell, which +displays the progress. + +### Running the processes + +This section runs the multi threaded processes and creates some of the output used to display +progress. + +```powershell +$job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { + $syncCopy = $using:sync + $process = $syncCopy.$($PSItem.Id) + + $process.Id = $PSItem.Id + $process.Activity = "$($PSItem.Id) starting" + $process.Status = "Processing" + + #fake workload that takes x amount of time to complete + start-sleep -Milliseconds ($PSItem.wait*5) + + #process. update activity + $process.Activity = "$($PSItem.id) processing" + foreach ($percent in 1..100) + { + #update process on status + $process.Status = "Handling $percent/100" + $process.PercentComplete = (($percent / 100) * 100) + + #fake workload that takes x amount of time to complete + Start-Sleep -Milliseconds $PSItem.Wait + } + + #end + $process.Completed = $true +} +``` + +The mock processes are sent to `Foreach-Object` and started as jobs. The **ThrottleLimit** is set to +**3** to highlight running multiple processes in a queue. The jobs are stored in the `$job` variable +and allows us to know when all the processes have finished later on. + +When using the `using:` statement to reference a parent scope variable in PowerShell, you can't use +expressions to make it dynamic. For example, if you tried to create the `$process` variable like +this, `$process = $using:sync.$($PSItem.id)`, you would get an error stating you can't use +expressions there. So, we create the `$syncCopy` variable to be able to reference and modify the +`$sync` variable without the risk of it failing. + +Next, we build out a hashtable to represent the progress of the process currently in the loop using +the `$process` variable by referencing the synchronized hashtable keys. The **Activity** and the +**Status** keys are used as parameter values for `Write-Progress` to display the status of a given +mock process in the next section. + +The `foreach` loop is just a way to simulate the process working and is randomized based on the +`$dataSet` **Wait** attribute to set `Start-Sleep` using milliseconds. How you calculate the +progress of your process may vary. + +### Displaying the progress of multiple processes + +Now that the mock processes are running as jobs, we can start to write the processes progress to the +PowerShell window. + +```powershell +while($job.State -eq 'Running') +{ + #foreach key that is available + $sync.Keys | Foreach-Object { + #if key is not defined, ignore + if(![string]::IsNullOrEmpty($sync.$_.keys)) + { + #create param + $param = $sync.$_ + + #execute write-progress + Write-Progress @param + } + } + + #wait to refresh to not overload gui + Start-Sleep -Seconds 0.1 +} +``` + +The `$job` variable contains the parent **job** and has a child **job** for each of the mock +processes. While any of the child jobs are still running, the parent job **State** will remain +"Running". This allows us to reliable use the `while` loop to continually update the progress of +every process until all processes are finished. + +Within the while loop, we loop through each of the keys in the `$sync` variable. Since this is +a synchronized hashtable, it is constantly updated but can still be accessed without throwing any +errors. + +There is a check to ensure that the process being reported is actually running using the +`IsNullOrEmpty()` method. If the process hasn't been started, the loop won't report on it and move +on to the next until it gets to a process that has been started. If the process is started, the +hashtable from the current key is used to splat the parameters to `Write-Progress`. + +### Full example + +```powershell +# Example workload +$dataset = @( + @{ + Id = 1 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 2 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 3 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 4 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } + @{ + Id = 5 + Wait = 3..10 | get-random | Foreach-Object {$_*100} + } +) + +# Create a hashtable for process. +# Keys should be ID's of the processes +$origin = @{} +$dataset | Foreach-Object {$origin.($_.id) = @{}} + +# Create synced hashtable +$sync = [System.Collections.Hashtable]::Synchronized($origin) + +$job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { + $syncCopy = $using:sync + $process = $syncCopy.$($PSItem.Id) + + $process.Id = $PSItem.Id + $process.Activity = "$($PSItem.Id) starting" + $process.Status = "Processing" + + # Fake workload start up that takes x amount of time to complete + start-sleep -Milliseconds ($PSItem.wait*5) + + # Process. update activity + $process.Activity = "$($PSItem.id) processing" + foreach ($percent in 1..100) + { + # Update process on status + $process.Status = "Handling $percent/100" + $process.PercentComplete = (($percent / 100) * 100) + + # Fake workload that takes x amount of time to complete + Start-Sleep -Milliseconds $PSItem.Wait + } + + # Mark process as completed + $process.Completed = $true +} + +while($job.State -eq 'Running') +{ + $sync.Keys | Foreach-Object { + # If key is not defined, ignore + if(![string]::IsNullOrEmpty($sync.$_.keys)) + { + # Create parameter hashtable to splat + $param = $sync.$_ + + # Execute Write-Progress + Write-Progress @param + } + } + + # Wait to refresh to not overload gui + Start-Sleep -Seconds 0.1 +} +``` + +## Related Links + +- [about_Jobs](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Jobs) +- [about_Scopes](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Scopes) +- [about_Splatting](/powershell/reference/7.0/Microsoft.PowerShell.Core/About/about_Splatting) diff --git a/reference/docs-conceptual/toc.yml b/reference/docs-conceptual/toc.yml index 2ac08e0da54c..9840fc4dbc26 100644 --- a/reference/docs-conceptual/toc.yml +++ b/reference/docs-conceptual/toc.yml @@ -73,6 +73,8 @@ href: learn/deep-dives/everything-about-null.md - name: Everything you want to know about ShouldProcess href: learn/deep-dives/everything-about-shouldprocess.md + - name: Write-Progress while multithreading + href: learn/deep-dives/write-progress-across-multiple-threads.md - name: PowerShell remoting items: - name: Just Enough Administration (JEA) From 2ccef840744c17548582b2f79e535bd6e64abb94 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Sun, 2 Aug 2020 18:37:31 -0700 Subject: [PATCH 06/17] Updates code notes to match full example formatting --- .../write-progress-across-multiple-threads.md | 27 +++++++++---------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index 18de6bf73fea..bfc7657f45c3 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -27,7 +27,6 @@ One of the downsides to this approach is it takes a, somewhat, complex set up to runs without error. ```powershell -#example workload $dataset = @( @{ Id = 1 @@ -51,13 +50,12 @@ $dataset = @( } ) -#Create a hashtable for progress. -#keys should be ID's of the progresses -#in this example i'l just use int 1-10 +# Create a hashtable for process. +# Keys should be ID's of the processes $origin = @{} $dataset | Foreach-Object {$origin.($_.id) = @{}} -#create synced hashtable +# Create synced hashtable $sync = [System.Collections.Hashtable]::Synchronized($origin) ``` @@ -89,22 +87,22 @@ $job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { $process.Activity = "$($PSItem.Id) starting" $process.Status = "Processing" - #fake workload that takes x amount of time to complete + # Fake workload start up that takes x amount of time to complete start-sleep -Milliseconds ($PSItem.wait*5) - #process. update activity + # Process. update activity $process.Activity = "$($PSItem.id) processing" foreach ($percent in 1..100) { - #update process on status + # Update process on status $process.Status = "Handling $percent/100" $process.PercentComplete = (($percent / 100) * 100) - #fake workload that takes x amount of time to complete + # Fake workload that takes x amount of time to complete Start-Sleep -Milliseconds $PSItem.Wait } - #end + # Mark process as completed $process.Completed = $true } ``` @@ -136,20 +134,19 @@ PowerShell window. ```powershell while($job.State -eq 'Running') { - #foreach key that is available $sync.Keys | Foreach-Object { - #if key is not defined, ignore + # If key is not defined, ignore if(![string]::IsNullOrEmpty($sync.$_.keys)) { - #create param + # Create parameter hashtable to splat $param = $sync.$_ - #execute write-progress + # Execute Write-Progress Write-Progress @param } } - #wait to refresh to not overload gui + # Wait to refresh to not overload gui Start-Sleep -Seconds 0.1 } ``` From 02b1fac25e907f5c6de751c599601c2c482c8c80 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Sun, 2 Aug 2020 18:39:40 -0700 Subject: [PATCH 07/17] minor update --- .../deep-dives/write-progress-across-multiple-threads.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index bfc7657f45c3..af663267ee0c 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -1,5 +1,5 @@ --- -title: Displaying progress across multiple threads +title: Displaying progress while multi-threading description: How to use Write-Progress across multiple threads with Foreach-Object -Parallel ms.date: 08/02/2020 --- @@ -75,7 +75,7 @@ displays the progress. ### Running the processes -This section runs the multi threaded processes and creates some of the output used to display +This section runs the multi-threaded processes and creates some of the output used to display progress. ```powershell From 8b16124e27ec16540c2f44caf981da4f3492c0d6 Mon Sep 17 00:00:00 2001 From: chasewilson Date: Mon, 3 Aug 2020 08:10:49 -0700 Subject: [PATCH 08/17] Removes .vscode settings file --- .vscode/settings.json | 3 --- 1 file changed, 3 deletions(-) delete mode 100644 .vscode/settings.json diff --git a/.vscode/settings.json b/.vscode/settings.json deleted file mode 100644 index 79e520aee0b5..000000000000 --- a/.vscode/settings.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "powershell.codeFormatting.addWhitespaceAroundPipe": true -} \ No newline at end of file From 099ed72ca9b462efc1139b5605ea5eb48ac2c5db Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:40:38 -0700 Subject: [PATCH 09/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index af663267ee0c..f09b819ad458 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -18,7 +18,7 @@ reporting the progress back to the host isn't as straight forward as normal use When writing the progress from multiple threads, tracking becomes difficult because when running parallel processes in PowerShell, each process has it's own runspace. To get around this, you can use a [synchronized hashtable](/dotnet/api/system.collections.hashtable.synchronized). A synced -hashtable is a thread safe data structure that can be edited by multiple threads simultaneously +hashtable is a thread safe data structure that can be modified by multiple threads simultaneously without throwing an error. ### Set up From 2342a4b6ac89bf0b4ed1ac28bfac9ae0258619ef Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:41:21 -0700 Subject: [PATCH 10/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../deep-dives/write-progress-across-multiple-threads.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index f09b819ad458..5bbe7d0f3347 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -62,11 +62,10 @@ $sync = [System.Collections.Hashtable]::Synchronized($origin) This section creates three different data structures, for three different purposes. The `$dataSet` variable stores an array of hashtables that is used to coordinate the next steps -without the risk of being modified. When looping through an object in PowerShell, if the object -being looped through is modified, PowerShell will throw an error. So, keeping the object being -looped through, separate from the objects being modified is a must. The `Id` key in each hashtable -is the identifier for a mock process. The `Wait` key simulates the workload of each mock process to -track. +without the risk of being modified. If a object collection is modified while iterating through the +collection, PowerShell throws an error. You must keep the object collection in the loop separate +from the objects being modified. The `Id` key in each hashtable is the identifier for a mock +process. The `Wait` key simulates the workload of each mock process being tracked. The `$origin` variable stores a nested hashtable with each key being one of the mock process id's. Then, it is used to hydrate the synchronized hashtable stored in the `$sync` variable. The `$sync` From 04f5d2ab91af1c1f65b70fbd6d76534c93b45819 Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:41:37 -0700 Subject: [PATCH 11/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index 5bbe7d0f3347..7397971539f3 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -83,7 +83,7 @@ $job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { $process = $syncCopy.$($PSItem.Id) $process.Id = $PSItem.Id - $process.Activity = "$($PSItem.Id) starting" + $process.Activity = "Id $($PSItem.Id) starting" $process.Status = "Processing" # Fake workload start up that takes x amount of time to complete From 5d795f4ed9b32582993035c584302a6155cb3fa4 Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:41:56 -0700 Subject: [PATCH 12/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index 7397971539f3..daa5f7b1a250 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -152,7 +152,7 @@ while($job.State -eq 'Running') The `$job` variable contains the parent **job** and has a child **job** for each of the mock processes. While any of the child jobs are still running, the parent job **State** will remain -"Running". This allows us to reliable use the `while` loop to continually update the progress of +"Running". This allows us to use the `while` loop to continually update the progress of every process until all processes are finished. Within the while loop, we loop through each of the keys in the `$sync` variable. Since this is From 327aac125c031da923582789122189e1b6155286 Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:42:04 -0700 Subject: [PATCH 13/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index daa5f7b1a250..9c1c1d1e5304 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -211,7 +211,7 @@ $job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { start-sleep -Milliseconds ($PSItem.wait*5) # Process. update activity - $process.Activity = "$($PSItem.id) processing" + $process.Activity = "Id $($PSItem.id) processing" foreach ($percent in 1..100) { # Update process on status From c188025e91c30c914540ea8c330981820fddde1b Mon Sep 17 00:00:00 2001 From: chasewilson Date: Mon, 3 Aug 2020 08:42:55 -0700 Subject: [PATCH 14/17] typo fix --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index 9c1c1d1e5304..4ea3338b5818 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -62,7 +62,7 @@ $sync = [System.Collections.Hashtable]::Synchronized($origin) This section creates three different data structures, for three different purposes. The `$dataSet` variable stores an array of hashtables that is used to coordinate the next steps -without the risk of being modified. If a object collection is modified while iterating through the +without the risk of being modified. If an object collection is modified while iterating through the collection, PowerShell throws an error. You must keep the object collection in the loop separate from the objects being modified. The `Id` key in each hashtable is the identifier for a mock process. The `Wait` key simulates the workload of each mock process being tracked. From 526250ebb7b6b46e6a449bea0a676fd466652684 Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:48:30 -0700 Subject: [PATCH 15/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index 4ea3338b5818..c90072fc525f 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -204,7 +204,7 @@ $job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { $process = $syncCopy.$($PSItem.Id) $process.Id = $PSItem.Id - $process.Activity = "$($PSItem.Id) starting" + $process.Activity = "Id $($PSItem.Id) starting" $process.Status = "Processing" # Fake workload start up that takes x amount of time to complete From 030604110c36f304f3b8f3d55e2f6401c27f8836 Mon Sep 17 00:00:00 2001 From: Chase Wilson <31453523+chasewilson@users.noreply.github.com> Date: Mon, 3 Aug 2020 08:48:38 -0700 Subject: [PATCH 16/17] Update reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md Co-authored-by: Sean Wheeler --- .../learn/deep-dives/write-progress-across-multiple-threads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index c90072fc525f..b9c8cae93685 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -90,7 +90,7 @@ $job = $dataset | Foreach-Object -ThrottleLimit 3 -AsJob -Parallel { start-sleep -Milliseconds ($PSItem.wait*5) # Process. update activity - $process.Activity = "$($PSItem.id) processing" + $process.Activity = "Id $($PSItem.id) processing" foreach ($percent in 1..100) { # Update process on status From 3234805ab3bf8c6c045bd13ddf76a74a822aec6a Mon Sep 17 00:00:00 2001 From: chasewilson Date: Mon, 3 Aug 2020 10:51:28 -0700 Subject: [PATCH 17/17] Update for PR --- .../deep-dives/write-progress-across-multiple-threads.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md index b9c8cae93685..814cd04105e1 100644 --- a/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md +++ b/reference/docs-conceptual/learn/deep-dives/write-progress-across-multiple-threads.md @@ -69,8 +69,8 @@ process. The `Wait` key simulates the workload of each mock process being tracke The `$origin` variable stores a nested hashtable with each key being one of the mock process id's. Then, it is used to hydrate the synchronized hashtable stored in the `$sync` variable. The `$sync` -variable is responsible for reporting the progress back to the parent process, PowerShell, which -displays the progress. +variable is responsible for reporting the progress back to the parent runspace, which displays the +progress. ### Running the processes