Set-AzStorageFileContent fails with HTTP 403 Not Authorized when uploading file with a SAS token that lacks both Read and Write permissions.

[ziesemer]

Description

Set-AzStorageFileContent fails with HTTP 403 Not Authorized when uploading file with a SAS token that lacks both "Read" and "Write" permissions - where only "Create" should be sufficient.

This is similar to #21595 - but looks to be a different issue with a different error message and a different set of circumstances.

1. Create an Azure Storage account and container through the Azure portal with default parameters.
2. Create a Shared Access Signature (SAS) Token / URL that contains on the "Create" and "List" permissions, e.g. https://azbug20240113.blob.core.windows.net/upload?sp=cl&st=2024-01-13T06:00:00Z&se=2024-02-01T06:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=******** .
3. Observe that Azure Storage Explorer can successfully upload a file, using the above SAS URL.
   i. Note that Storage Explorer gives a warning that "Your credential does not authorize AzCopy to read the contents of the destination. AzCopy will therefore attempt to overwrite if there are conflicts and you will not be able to check content length at the end of each trasnsfer. Do you want to proceed?" However, this appears to be another issue to be addressed separately - as the file successfully uploads - and even with the "List" permission, is able to determine both that a file first did not exist, and is able to see the completed uploaded file - including size / content length.
   ii. Using the latest version as of this report - 1.32.1 (96), with AzCopy Version 10.20.1. Build Number: 20231114.10, Commit: e9351da25b.
4. Observe that AzCopy itself (10.22.2) can also successfully upload a file without issue or even a warning, using the same SAS URL.

& azcopy.exe copy .\Test.txt 'https://azbug20240113.blob.core.windows.net/upload?sp=cl&st=2024-01-13T06:00:00Z&se=2024-02-01T06:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=********'

5. Observe that using Set-AzStorageBlobContent from the Az.Storage PowerShell module fails attempting the same.

$sasToken = 'sp=cl&st=2024-01-13T06:00:00Z&se=2024-02-01T06:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=...'
$storageContext = New-AzStorageContext -BlobEndpoint 'https://azbug20240113.blob.core.windows.net' -SasToken $sasToken
Get-AzStorageBlob -Context $storageContext -Container 'upload'
Set-AzStorageBlobContent -Context $storageContext -Container 'upload' -File .\Test.txt -Blob 'Test.txt' -Force

The first 3 lines complete without error. However, the last line (Set-AzStorageBlobContent) fails with:

Set-AzStorageBlobContent: This request is not authorized to perform this operation using this permission. HTTP Status Code: 403 - HTTP Error Message: This request is not authorized to perform this operation using this permission.
ErrorCode: AuthorizationPermissionMismatch
ErrorMessage: This request is not authorized to perform this operation using this permission.
RequestId:c61539bb-801e-0073-761d-475746000000
Time:2024-01-14T19:11:58.9037435Z

Attempting to add only either one of the "Read" or "Write" permissions is not sufficient. However, after adding both:

$sasToken = 'sp=rcwl&st=2024-01-13T06:00:00Z&se=2024-02-01T06:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=...'

... the upload succeeds as initially expected.

When this was first noticed during development, the version of Az.Storage on the system was 5.4.0 - so this goes back at least that far. However, the same issue occurs even with the latest 6.1.0.

To further complicate debugging - I have been unable to find a way to capture the HTTPS requests issued from Set-AzStorageBlobContent, which I may open a separate issue for. Following https://learn.microsoft.com/en-us/powershell/azure/az-powershell-proxy?view=azps-11.2.0 and setting HTTPS_PROXY (and even HTTP_PROXY), I'm able to capture the complete requests of GetAzStorageBlob with Fiddler. However, while I can see encrypted requests to Set-AzStorageBlobContent through Wireshark, I am unable to immediately find a working configuration to direct these requests through a proxy.

I'm wondering if Set-AzStorageBlobContent is first searching for an existing blob of the same name ("list") then also inspecting it ("read"), before creating an empty stub ("create"), and then finally populating it ("write"). This wouldn't be surprising, given the "Set" vs. "Add" verb name. If this is the case, at a minimum, this then needs to be further documented at https://learn.microsoft.com/en-us/powershell/module/az.storage/set-azstorageblobcontent?view=azps-11.2.0 . This would still leave this at a feature disparity with AzCopy, which should be addressed.

One of the primary reasons I'm attempting to use PowerShell instead of AzCopy here is because AzCopy still does not have a valid mechanism to pass SAS tokens without exposing them on the command line (Azure/azure-storage-azcopy#1103 , Azure/azure-storage-azcopy#1990). The "improved login support" documented https://github.com/Azure/azure-storage-azcopy/wiki/Improved-login-support-for-AzCopy-commands-(with-in-memory-secret-store) unfortunately still does not support SAS keys - which are also the only way to limit permissions to this granularity.

My next attempt here will be to bypass the Azure PowerShell module, and call the HTTP API directly...

Issue script & Debug output

DEBUG: 1:20:14 PM - NewAzureStorageContext begin processing with ParameterSet 'SasTokenServiceEndpoint'.
DEBUG: AzureQoSEvent:  Module: Az.Storage:6.1.0; CommandName: New-AzStorageContext; PSVersion: 7.4.1; IsSuccess: True; Duration: 00:00:00.0012638
DEBUG: 1:20:14 PM - NewAzureStorageContext end processing.
DEBUG: 1:20:14 PM - Init Operation Context for 'SetAzureBlobContentCommand' with client request id  Azure-Storage-PowerShell-b0d91ffd-85b4-47d6-a08b-f7a3b87c9e6e. If you want to get more details, please add "-Debug" to your command.
DEBUG: 1:20:14 PM - Use storage account '[SasToken]' from storage context.
DEBUG: 1:20:15 PM - SetAzureBlobContentCommand end processing, Start 4 remote calls. Finish 4 remote calls. Elapsed time 754926.61 ms. Client operation id: Azure-Storage-PowerShell-b0d91ffd-85b4-47d6-a08b-f7a3b87c9e6e.
DEBUG: 1:20:15 PM - SetAzureBlobContentCommand end processing.

Environment data

PS> $PSVersionTable                    

Name                           Value
----                           -----
PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

PS> Get-Module Az*

ModuleType Version    PreRelease Name
---------- -------    ---------- ----
Script     2.15.0                Az.Accounts
Script     6.1.0                 Az.Storage

Error output

PS> Resolve-AzError

   HistoryId: 1

Message        : This request is not authorized to perform this operation using this permission. HTTP Status Code: 403 - HTTP Error Message: This request is not authorized to perform this operation using this permission.
                 ErrorCode: AuthorizationPermissionMismatch
                 ErrorMessage: This request is not authorized to perform this operation using this permission.
                 RequestId:6ba6dbc6-301e-003b-221f-474a71000000
                 Time:2024-01-14T19:23:51.4802266Z
StackTrace     : 
Exception      : Microsoft.Azure.Storage.StorageException
InvocationInfo : {Set-AzStorageBlobContent}
Line           : Set-AzStorageBlobContent -Context $storageContext -Container 'upload' -File .\Test.txt -Blob 'Test.txt' -Force
Position       : At line:4 char:1
                 + Set-AzStorageBlobContent -Context $storageContext -Container 'upload' …
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 1

Message        : This request is not authorized to perform this operation using this permission.
StackTrace     :    at Microsoft.WindowsAzure.Commands.Storage.Common.DataMovementTransferHelper.DoTransfer(Func`1 doTransfer, ProgressRecord record, TaskOutputStream outputStream)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.Upload2Blob(Int64 taskId, IStorageBlobManagement localChannel, String filePath, CloudBlob blob)
                    at Microsoft.WindowsAzure.Commands.Storage.Common.LimitedConcurrencyTaskScheduler.RunConcurrentTask(Int64 taskId, Task task)
Exception      : Microsoft.Azure.Storage.StorageException
InvocationInfo : {Set-AzStorageBlobContent}
Line           : Set-AzStorageBlobContent -Context $storageContext -Container 'upload' -File .\Test.txt -Blob 'Test.txt' -Force
Position       : At line:4 char:1
                 + Set-AzStorageBlobContent -Context $storageContext -Container 'upload' …
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 1

[ziesemer]

Looks like it's getting stuck at

azure-powershell/src/Storage/Storage/Blob/Cmdlet/SetAzureStorageBlobContent.cs

Line 284 in 6cf75e1

await DataMovementTransferHelper.DoTransfer(() =>

- but I'm getting lost from there. (At least until I get this up and running in an IDE that I can click through..)

[ziesemer]

So the "Write" permission is required to use the "Put Block" functionality. Still looking to justify the need for "Read".

[yifanz7]

Mark A. Ziesemer (@ziesemer) Could you please share the size of the blob that you're uploading?
The PowerShell cmdlet Set-AzStorageBlobContent is calling DMLib and as we've tested, it seems like when a blob is bigger than 8MB, DMLib would re-calculate the upload block size for performance reason. And with this calculation, API "put block list" will be called, which will need write permission.

For the read permission, could you confirm if you're using "-Force" when uploading the blob? One possible reason is that without -Force, DMLib will first fetch the attributes of the blob which will need Read permission. With -Force, DMLib will write the blob directly without the checking.

[ziesemer]

Mark A. Ziesemer (@ziesemer) Could you please share the size of the blob that you're uploading? The PowerShell cmdlet Set-AzStorageBlobContent is calling DMLib and as we've tested, it seems like when a blob is bigger than 8MB, DMLib would re-calculate the upload block size for performance reason. And with this calculation, API "put block list" will be called, which will need write permission.

Thank you. My current efforts here are focused on files ranging from 100 MB - 100 GB, so this makes sense.

For the read permission, could you confirm if you're using "-Force" when uploading the blob? One possible reason is that without -Force, DMLib will first fetch the attributes of the blob which will need Read permission. With -Force, DMLib will write the blob directly without the checking.

I was already attempting this with "-Force" - as included in the example I provided in the original submission above.

Thank you for also calling out DMLib. That hadn't yet caught my attention as an alternative, and may be a good intermediate option to consider as well. In the meantime, I had a version of things mostly working that called the HTTP API directly - though without the multi-threading yet (and am seeing the expected drop in performance to match).

[yifanz7]

Mark A. Ziesemer (@ziesemer) Great to know that a workaround with http API works :)
Would you please let us know if there're other questions regarding the issue? Otherwise we will close the issue. Thanks!

[ziesemer]

If -Force is to allow an upload without first checking the destination ("Read" permission), that is not working. This is what is demonstrated in the original submission above.

[yifanz7]

Mark A. Ziesemer (@ziesemer) Hmm I'm not able to repro the issue with -Force. Does the -Force issue happen to all sizes of blobs for you?

This is what I have on my machine:
Create SAS token and Storage context:

$sas = New-AzStorageContainerSASToken -Name testc1 -Permission clw -Context $ctx -Protocol HttpsOnly -ExpiryTime 2222-01-01
# $sas looks like sv=2023-08-03&spr=https&se=2221-12-31T16%3A00%3A00Z&sr=c&sp=cwl&sig=xxx
$sascontext = New-AzStorageContext -SasToken $sas -BlobEndpoint "https://testuploadfile1.blob.core.windows.net"

Upload a blob of size 333MB which is successful:

PS C:\> Set-AzStorageBlobContent -File D:\sourcefile\test333MB -Container testc1 -Blob test203 -Context $sascontext -Force



   AccountName: testuploadfile1, ContainerName: testc1

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId                  
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------                  
test203              BlockBlob -1                                             2024-01-17 08:57:43Z                                         False                                 

Try to get this uploaded blob which should fail as Read permission is missing:

PS C:\> Get-AzStorageBlob -Blob test203 -Container testc1 -Context $sascontext
Get-AzStorageBlob : Service request failed.
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch
Headers:
Transfer-Encoding: chunked
x-ms-request-id: 7f587c9c-101e-0079-2224-499baf000000
x-ms-client-request-id: ad7fa42b-4a76-4709-a8ed-45527bc2fe59
x-ms-version: 2023-08-03
x-ms-error-code: AuthorizationPermissionMismatch
Date: Wed, 17 Jan 2024 09:04:17 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
At line:1 char:1
+ Get-AzStorageBlob -Blob test203 -Container testc1 -Context $sascontex ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Get-AzStorageBlob], RequestFailedException
    + FullyQualifiedErrorId : RequestFailedException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand

I think the steps are the same as described in the issue above. Please let me know if I'm missing anything in the repro steps