Set-AzStorageBlobContent throws PSArgumentOutOfRangeException on small files

[yvzn]

Description

From time to time Set-AzStorageBlobContent Cmdlet throws a PSArgumentOutOfRangeException with message 'Cannot set percent because PercentComplete cannot be greater than 100', when uploading a small file to blob storage.

It looks like the file is uploaded after all. However the thrown exception can be a blocker if -ErrorAction is set to Stop.

Unhandled exception. Unhandled exception. System.Management.Automation.PSArgumentOutOfRangeException: Cannot set percent because PercentComplete cannot be greater than 100. (Parameter 'value')
Actual value was 3625.
   at System.Management.Automation.ProgressRecord.set_PercentComplete(Int32 value)
   at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.<>c__DisplayClass61_0.<UploadBlobwithSdk>b__0(Int64 finishedBytes)
   at System.Progress`1.InvokeHandlers(Object state)
   at System.Threading.SynchronizationContext.<>c.<Post>b__8_0(ValueTuple`2 s)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback`1.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()System.Management.Automation.PSArgumentOutOfRangeException: Cannot set percent because PercentComplete cannot be greater than 100. (Parameter 'value')

Source code
https://github.com/Azure/azure-powershell/blob/main/src/Storage/Storage/Blob/Cmdlet/SetAzureStorageBlobContent.cs#L464

Issue script & Debug output

PS /> $DebugPreference='Continue'
PS /> $subscriptionName = '...'
PS /> $tenantId = '...'
PS /> $StorageAccountName = '...'
PS /> Connect-AzAccount -UseDeviceAuthentication -Tenant $tenantId -Subscription $subscriptionName
WARNING: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ... to authenticate.

Account                                             SubscriptionName             TenantId                             Environment
-------                                             ----------------             --------                             -----------
....

PS /> $AzStorageContext = New-AzStorageContext -StorageAccountName $StorageAccountName -UseConnectedAccount 
PS /> $FilePath = 'a-small-file.txt'
PS /> $TargetPath = $FilePath
PS /> $Container = '$web'
PS /> 'Hello world! (this is not the content of real file)' | Out-File $FilePath
PS /> Set-AzStorageBlobContent `
	-File $FilePath `
	-Blob $TargetPath `
	-Context $AzStorageContext `
	-Container $Container `
	-Force `
	-ConcurrentTaskCount 10 `
	-ClientTimeoutPerRequest 10 `
	-ErrorAction Stop
...some sensitive output obfuscated...
An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.
An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.
Unhandled exception. Unhandled exception. System.Management.Automation.PSArgumentOutOfRangeException: Cannot set percent because PercentComplete cannot be greater than 100. (Parameter 'value')
Actual value was 6368.
   at System.Management.Automation.ProgressRecord.set_PercentComplete(Int32 value)
   at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.<>c__DisplayClass61_0.<UploadBlobwithSdk>b__0(Int64 finishedBytes)
   at System.Progress`1.InvokeHandlers(Object state)
   at System.Threading.SynchronizationContext.<>c.<Post>b__8_0(ValueTuple`2 s)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback`1.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

System.Management.Automation.PSArgumentOutOfRangeException: Cannot set percent because PercentComplete cannot be greater than 100. (Parameter 'value')
Actual value was 6368.
   at System.Management.Automation.ProgressRecord.set_PercentComplete(Int32 value)
   at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.<>c__DisplayClass61_0.<UploadBlobwithSdk>b__0(Int64 finishedBytes)
   at System.Progress`1.InvokeHandlers(Object state)
   at System.Threading.SynchronizationContext.<>c.<Post>b__8_0(ValueTuple`2 s)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback`1.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Environment data

PS /> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.2
PSEdition                      Core
GitCommitId                    7.2.2
OS                             Linux 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

PS /> Get-Module Az*

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.7.5                 Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount…}

Error output

PS /> Resolve-AzError
WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[blueww]

@yvzn
Would you please share which Az.Storage module version do you use (run Get-Module and see it)?
And how frequently do you meet the error?

I have tried to repro the issue with latest Az.Storage 4.4.1, by running a similar script as yours 100 times.
But the issue not repro, it always upload success.
I test with Powershell 7.2.2 on Windows.

[yvzn]

@blueww
I am running Az.Storage 4.4.0 with Powershell 7.2.2 on Linux in a Docker container.

Could it make a difference ? Possibly in the way the code computes the file size ?

We are using this cmdlet a lot every day, as part of our CI/CD build ; most of the time it runs ok, but we get the exception dozen times a day, approximately. The only workaround for now is to retry the faulty upload, until it eventually succeeds.

I was wondering if adding an extra check to the pr.PercentComplete = ... instruction would help. It would certainly fix the issue for us. Is it possible to submit a pull request ?

[blueww]

@yvzn

From the Powershell code, it looks the "pr.PercentComplete" can't be bigger than 100, since finishedBytes should <= fileSize, until there are some SDK issue cause the finishedBytes is not accurate.

However, to unblock you, we can add code to limit the "pr.PercentComplete" to <=100.

[yvzn]

@blueww
Yes that would be perfect for us if you could add that extra code to limit pr.PercentComplete

[blueww]

@yvzn
The fix for this issue is already in Az.Storage 4.5.0.
https://www.powershellgallery.com/packages/Az.Storage/4.5.0