AzureFileCopy@4 does not work, but AzureFileCopy@3 does when copying files to blog storage

[samsmithnz]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: AzureFileCopy@4

Environment

Server - Azure Pipelines

Agent - Hosted: Windows 2017

Issue Description

This is my yaml definition:

- task: AzureFileCopy@4
  displayName: 'Copy ARM Templates to Azure Storage'
  inputs:
    SourcePath: ArmTemplateTestProject/ArmTemplateTestProject
    azureSubscription: 'my connection to Azure Portal'
    Destination: AzureBlob
    storage: mydatastorage
    ContainerName: armtemplate

Task logs

Uploading files from source path: 'd:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject\azuredeploy.json' to storage account: 'mydatastorage' in container: 'armtemplate' with blob prefix: ''
& "AzCopy\AzCopy.exe" copy "d:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject\azuredeploy.json" "https://mydatastorage.blob.core.windows.net/armtemplate"
Error: wrong number of arguments, please refer to the help page on usage of this command

Troubleshooting

Version 3 of the task, with the same settings appears to work. I can't see anything wrong with the arguments, it appears from the logs to be using the same azcopy command

[20shivangi]

@samsmithnz Are you giving the inputs as a yaml file or in the azure task itself ?
Also, please make sure that you have given all necessary permissions to your SPN before using AzureFileCopyV4. Please refer to this doc : https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10#option-1-use-azure-active-directory

[20shivangi]

Please let us know the version of AzureFileCopyV4 used if you face the error again

[samsmithnz]

Permissions should be all set, I use this same service connection for other deployments and AzureFileCopyV3 is working on the same connection? Saying that, I get a different error now, which does indicate permissions. Not sure that makes sense, given it's the same settings/storage/etc from v3 to v4, and the SPN has full permissions.

I created another job, with the same settings and V3 and V4. Here are the logs, V3 is working, V4 is not.

V3 log:

2020-06-24T11:20:11.5065143Z ##[section]Starting: Copy ARM Templates to Azure Storage V3
2020-06-24T11:20:11.5213684Z ==============================================================================
2020-06-24T11:20:11.5214345Z Task : Azure file copy
2020-06-24T11:20:11.5214646Z Description : Copy files to Azure Blob Storage or virtual machines
2020-06-24T11:20:11.5215125Z Version : 3.167.4
2020-06-24T11:20:11.5215366Z Author : Microsoft Corporation
2020-06-24T11:20:11.5216421Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2020-06-24T11:20:11.5216882Z ==============================================================================
2020-06-24T11:20:13.2593551Z ##[command]Import-Module -Name C:\Modules\azurerm_2.1.0\AzureRM\2.1.0\AzureRM.psd1 -Global
2020-06-24T11:20:21.1096058Z ##[warning]The names of some imported commands from the module 'AzureRM.Websites' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-06-24T11:20:21.1177009Z ##[warning]The names of some imported commands from the module 'AzureRM' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-06-24T11:20:21.1316923Z ##[command]Import-Module -Name C:\Modules\azurerm_2.1.0\AzureRM.Profile\2.1.0\AzureRM.Profile.psm1 -Global
2020-06-24T11:20:21.5020252Z ##[command]Add-AzureRMAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -EnvironmentName AzureCloud
2020-06-24T11:20:22.7103675Z ##[command] Set-AzureRmContext -SubscriptionId 07db7d0b-a6cb-4e58-b07e-e1d541c39f5b -TenantId ***
2020-06-24T11:20:25.7192971Z Uploading files from source path: 'D:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject' to storage account: 'samsappdataeustorage' in container: 'armtemplate' with blob prefix: '4496A'
2020-06-24T11:20:25.7499837Z ##[command] & "AzCopy\AzCopy.exe" /Source:"D:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject" /Dest:"https://samsappdataeustorage.blob.core.windows.net/armtemplate/4496A" /@:"D:\a_temp\23abf1f9-eb45-4b54-938a-979fda8af775" /XO /Y /SetContentType /Z:"AzCopy" /V:"AzCopy\AzCopyVerbose_0612900c-dafd-413e-9141-7e8eb059ce13.log" /S
2020-06-24T11:20:26.8463231Z [2020/06/24 11:20:26] Transfer summary:
2020-06-24T11:20:26.8464588Z -----------------
2020-06-24T11:20:26.8465406Z Total files transferred: 8
2020-06-24T11:20:26.8466307Z Transfer successfully: 8
2020-06-24T11:20:26.8466805Z Transfer skipped: 0
2020-06-24T11:20:26.8467345Z Transfer failed: 0
2020-06-24T11:20:26.8467992Z Elapsed time: 00.00:00:01
2020-06-24T11:20:26.8666396Z Uploaded files successfully from source path: 'D:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject' to storage account: 'samsappdataeustorage' in container: 'armtemplate' with blob prefix: '4496A'
2020-06-24T11:20:27.1814707Z ##[section]Finishing: Copy ARM Templates to Azure Storage V3

V4 log:

2020-06-24T11:20:27.1856426Z ##[section]Starting: Copy ARM Templates to Azure Storage V4
2020-06-24T11:20:27.2098030Z ==============================================================================
2020-06-24T11:20:27.2098417Z Task : Azure file copy
2020-06-24T11:20:27.2098749Z Description : Copy files to Azure Blob Storage or virtual machines
2020-06-24T11:20:27.2099032Z Version : 4.170.2
2020-06-24T11:20:27.2099298Z Author : Microsoft Corporation
2020-06-24T11:20:27.2099845Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2020-06-24T11:20:27.2100243Z ==============================================================================
2020-06-24T11:20:29.1726101Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Accounts\1.8.1\Az.Accounts.psd1 -Global
2020-06-24T11:20:31.7490286Z ##[command]Clear-AzContext -Scope Process
2020-06-24T11:20:32.6149263Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2020-06-24T11:20:33.1417703Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
2020-06-24T11:20:34.4756399Z ##[command] Set-AzContext -SubscriptionId 07db7d0b-a6cb-4e58-b07e-e1d541c39f5b -TenantId ***
2020-06-24T11:20:36.1571794Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Resources\1.8.0\Az.Resources.psd1 -Global
2020-06-24T11:20:37.1179808Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Storage\1.9.0\Az.Storage.psd1 -Global
2020-06-24T11:20:38.6126351Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Compute\3.1.0\Az.Compute.psd1 -Global
2020-06-24T11:20:39.6787093Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Network\2.1.0\Az.Network.psd1 -Global
2020-06-24T11:20:45.4504022Z ##[command] & "AzCopy\AzCopy.exe" login --service-principal --application-id "" --tenant-id=""
2020-06-24T11:20:45.5018786Z INFO: If you set an environment variable by using the command line, that variable will be readable in your command line history. Consider clearing variables that contain credentials from your command line history. To keep variables from appearing in your history, you can use a script to prompt the user for their credentials, and to set the environment variable.
2020-06-24T11:20:45.6707780Z INFO: SPN Auth via secret succeeded.
2020-06-24T11:20:45.8973389Z INFO: AzCopy.exe: A newer version 10.4.3 is available to download
2020-06-24T11:20:45.8973856Z
2020-06-24T11:20:46.0263614Z INFO: AzCopy.exe: A newer version 10.4.3 is available to download
2020-06-24T11:20:46.0264042Z
2020-06-24T11:20:46.0363342Z Uploading files from source path: 'D:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject' to storage account: 'samsappdataeustorage' in container: 'armtemplate' with blob prefix: '4496B'
2020-06-24T11:20:46.0445023Z ##[command] & "AzCopy\AzCopy.exe" copy "D:\a\1\s\ArmTemplateTestProject\ArmTemplateTestProject" "https://samsappdataeustorage.blob.core.windows.net/armtemplate/4496B" --log-level=INFO --recursive
2020-06-24T11:20:46.1049615Z INFO: Scanning...
2020-06-24T11:20:46.1051628Z INFO: Using OAuth token for authentication.
2020-06-24T11:20:46.3742753Z INFO: AzCopy.exe: A newer version 10.4.3 is available to download
2020-06-24T11:20:46.3743499Z
2020-06-24T11:20:46.8298976Z
2020-06-24T11:20:46.8302476Z Job d6543a78-36f8-7340-74ad-cf508d2d60f6 has started
2020-06-24T11:20:46.8306994Z Log file is located at: C:\Users\VssAdministrator.azcopy\d6543a78-36f8-7340-74ad-cf508d2d60f6.log
2020-06-24T11:20:46.8307712Z
2020-06-24T11:20:46.8309230Z
2020-06-24T11:20:47.3917191Z 0.0 %, 0 Done, 0 Failed, 8 Pending, 0 Skipped, 8 Total,
2020-06-24T11:20:47.3925556Z INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.7.0/azblob/zc_storage_error.go:42
2020-06-24T11:20:47.3927214Z ===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
2020-06-24T11:20:47.3930476Z Description=This request is not authorized to perform this operation using this permission.
2020-06-24T11:20:47.3935525Z RequestId:9d97bcbc-f01e-006a-3f19-4a308a000000
2020-06-24T11:20:47.3939127Z Time:2020-06-24T11:20:47.3877686Z, Details:
2020-06-24T11:20:47.3943435Z Code: AuthorizationPermissionMismatch
2020-06-24T11:20:47.3951444Z PUT https://samsappdataeustorage.blob.core.windows.net/armtemplate/4496B/ArmTemplateTestProject/templates/azureStorageTemplate.parameters.json?timeout=901
2020-06-24T11:20:47.3954519Z Authorization: REDACTED
2020-06-24T11:20:47.3957435Z Content-Length: [323]
2020-06-24T11:20:47.3961519Z User-Agent: [TFS_useragent AzCopy/10.3.3 Azure-Storage/0.7 (go1.13; Windows_NT)]
2020-06-24T11:20:47.3964222Z X-Ms-Blob-Cache-Control: []
2020-06-24T11:20:47.3966905Z X-Ms-Blob-Content-Disposition: []
2020-06-24T11:20:47.3971952Z X-Ms-Blob-Content-Encoding: []
2020-06-24T11:20:47.3973506Z X-Ms-Blob-Content-Language: []
2020-06-24T11:20:47.3984912Z X-Ms-Blob-Content-Md5: []
2020-06-24T11:20:47.3988367Z X-Ms-Blob-Content-Type: [application/json]
2020-06-24T11:20:47.3991081Z X-Ms-Blob-Type: [BlockBlob]
2020-06-24T11:20:47.4037704Z X-Ms-Client-Request-Id: [54779a9b-b0f0-4eff-762d-61ee0ecc2f18]
2020-06-24T11:20:48.0888385Z X-Ms-Version: [2018-03-28]
2020-06-24T11:20:48.0889677Z --------------------------------------------------------------------------------
2020-06-24T11:20:48.0890645Z RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
2020-06-24T11:20:48.0891554Z Content-Length: [279]
2020-06-24T11:20:48.0891977Z Content-Type: [application/xml]
2020-06-24T11:20:48.0892416Z Date: [Wed, 24 Jun 2020 11:20:46 GMT]
2020-06-24T11:20:48.0892815Z Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
2020-06-24T11:20:48.0898032Z X-Ms-Error-Code: [AuthorizationPermissionMismatch]
2020-06-24T11:20:48.0900060Z X-Ms-Request-Id: [9d97bcbc-f01e-006a-3f19-4a308a000000]
2020-06-24T11:20:48.0901772Z X-Ms-Version: [2018-03-28]
2020-06-24T11:20:48.0902351Z
2020-06-24T11:20:48.0912507Z
2020-06-24T11:20:48.0912902Z
2020-06-24T11:20:49.6006870Z 0.0 %, 0 Done, 0 Failed, 8 Pending, 0 Skipped, 8 Total,
2020-06-24T11:20:49.6007691Z
2020-06-24T11:20:49.6008244Z
2020-06-24T11:20:49.6008626Z Job d6543a78-36f8-7340-74ad-cf508d2d60f6 summary
2020-06-24T11:20:49.6008876Z Elapsed Time (Minutes): 0.0334
2020-06-24T11:20:49.6009527Z Total Number Of Transfers: 8
2020-06-24T11:20:49.6010327Z Number of Transfers Completed: 0
2020-06-24T11:20:49.6011196Z Number of Transfers Failed: 1
2020-06-24T11:20:49.6014409Z Number of Transfers Skipped: 0
2020-06-24T11:20:49.6016114Z TotalBytesTransferred: 0
2020-06-24T11:20:49.6018240Z Final Job Status: Cancelled
2020-06-24T11:20:49.6019409Z
2020-06-24T11:20:49.6029414Z ##[command] & "AzCopy\AzCopy.exe" logout
2020-06-24T11:20:49.6029862Z INFO: Logout succeeded.
2020-06-24T11:20:49.6030571Z INFO: AzCopy.exe: A newer version 10.4.3 is available to download
2020-06-24T11:20:49.6030928Z
2020-06-24T11:20:49.6031299Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2020-06-24T11:20:49.6031997Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2020-06-24T11:20:50.1190294Z ##[error]Upload to container: 'armtemplate' in storage account: 'samsappdataeustorage' with blob prefix: '4496B' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
2020-06-24T11:20:50.2044048Z ##[section]Finishing: Copy ARM Templates to Azure Storage V4

YAML:

I tried it with both the classic UI and YAML, and get the same result. The only difference is that I appended an A and B to the blobprefix, so that each task would copy to a unique location

pool:
  vmImage: 'windows-latest'

steps:
- task: AzureFileCopy@3
  displayName: 'Copy ARM Templates to Azure Storage V3'
  inputs:
    SourcePath: ArmTemplateTestProject/ArmTemplateTestProject
    azureSubscription: 'SamLearnsAzure connection to Azure Portal'
    Destination: AzureBlob
    storage: samsappdataeustorage
    ContainerName: armtemplate
    BlobPrefix: '$(Build.BuildId)A'
    sasTokenTimeOutInMinutes: 240
- task: AzureFileCopy@4
  displayName: 'Copy ARM Templates to Azure Storage V4'
  inputs:
    SourcePath: ArmTemplateTestProject/ArmTemplateTestProject
    azureSubscription: 'SamLearnsAzure connection to Azure Portal'
    Destination: AzureBlob
    storage: samsappdataeustorage
    ContainerName: armtemplate
    BlobPrefix: '$(Build.BuildId)B'

[20shivangi]

@samsmithnz This error [AuthorizationPermissionMismatch] comes up only if we do not provide storage blob owner and storage blob contributor permissions to the SPN. Can you please check if these permissions are given to SPN?

[samsmithnz]

I've confirmed that the SPN has contributor access to the subscription.

Does V3 use the SPN in a different way? Why do I get errors with the new version and not the old version?

[20shivangi]

@samsmithnz Yes it is different. AzureFileCopyV3 uses azcopy.exe version 7, and AzureFileCopyV4 uses azcopy.exe version 10.
So in AzureFileCopyV4, we have to some extra permissions to our SPN as documented in the document :
https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10#option-1-use-azure-active-directory

[samsmithnz]

Looks like I already have contributor and owner access to the storage, (inherited from the subscription), which I wouldn't normally think is enough. Adding the storage contributor role worked, but I have to admit, it's odd having to add an extra role. Any idea why they made this change? Otherwise this issue can be closed.

[20shivangi]

@samsmithnz This permission thing came as a document with new Azcopy.exe version. I can't say why this change has been made. But you have to give these required permissions to run AzureFileCopyV4 task. I hope all your queries are solved now.

[20shivangi]

Closing this one, feel free to reopen if face any issue.

[samsmithnz]

Thank you @20shivangi. One last thought: Should any of this be in the v4 task docs? I don't see anything, but it's a change.

[20shivangi]

@samsmithnz I will check this, if it is not documented, I will take care of it

[jdthorpe]

Seems weird that (A) AzCopy v10 (and hence AzureFileCopy@4) enforces permissions differently than the underlying blob storage data plane, and (B) that granting permissions via the "this pipeline needs permissions to run" popup when running the Azure Pipeline grants sufficient permissions for AzureFileCopy@3 task but not the AzureFileCopy@4 task.

For those unfortunate enough to be as confused about this as I was, the solution is to:

• open the Devops pipeline and note the name of the service connection
• open the project settings at the bottom of the Project page, click "Service Connections", select the service connection, and then click "Manage Service Principal" and then get the display name (optionally changing it to something unique -- as all the service principals for service connections in a project have the same default name which can be confusing for the next step)
• open the Storage resource (Account or container) in the azure portal, then select "Access Control (IAM)" in the blade on the left, then click "Add a role assignment" add the "Storage Blob Data Contributor" role to the service principal