Azure file copy fails to copy to Azure Blob #12154
Assignees
Labels
Comments
|
@trevonmckay can you try to execute the task on the Hosted VS2017 image and see if you are facing the same issue. |
|
I'm getting the same issue, this was running fine before, but recently has started to fail. Logs are follows: |
|
@arjgupta received the same error on VS2017 |
|
@trevonmckay: Thanks for reporting this issue. We have heard similar issues from other customers as well. There seems to be compat break involving AzureRM 2.1.0 module. While we will work on fixing this, it might take a while before the fix is rolled out. As a work around, can you please add a task before file copy task. Add a script in that task to prepend env:psmodulepath with the latest AzureRM module available on the machine? |
|
I've just tried this with my instance and it seems to now be getting another error |
|
Been trying today to get the Azure file copy task working, tried loads of options and installing different versions of the Azure PowerShell modules but just lots of errors. I decided to give up and write the PowerShell myself to just copy the files 👍. The workaround for me was to just add a Azure PowerShell task and use the following inline script: Then just set the |
|
Bit of an update as the above script uploaded the files but some were the incorrect content type and if you're using Blob storage to host a static website then Chrome doesn't like been given the incorrect data type. Add-Type -AssemblyName "System.Web"
$StorageAcountObj = Get-AzStorageAccount -ResourceGroupName $(ResourceGroup) -Name $(StorageAccountName);
$StorageContext = $StorageAcountObj.Context
$items = Get-ChildItem "$(System.DefaultWorkingDirectory)/where-your-files-are" -File -Recurse
foreach ($item in $items) {
$mineType = [System.Web.MimeMapping]::GetMimeMapping($item.FullName)
Write-Host "Uploading $($item.FullName) $($mineType)"
Set-AzStorageBlobContent -Context $StorageContext -File $item.FullName -Container "`$web" -Force -Properties @{"ContentType" = $mineType}
}Just thought I'd share for anyone else with similar problems and needing a workaround 👍 |
|
@trevonmckay We have fixed the issue in latest version of AzureFileCopy. It will take at least 5-6 weeks to get deployed. |
|
Closing this issue as the fix has been rolled out which will be deployed in 4-5 weeks. |
|
@20shivangi we are having this issue starting today. I have tried it with V2, V3, and V4 of the task and they all fail. I have tried it on the vs2017-win2016 and windows-2019 Hosted Agents and they also both fail. Logs below: Here is the start of the task for version #s: We were using V2 and the older task version worked yesterday. YESTERDAY (worked): TODAY (broken): Also note that based on the diffs of the task, I think the issue is likely caused by using a new version of AzureRM through this commit: 9c1e149#diff-10c0386b7ba2862cd7e655dfcf5dbb96 Based on the logs, it previously imported AzureRM 2.1.0 and it is now using AzureRM 6.13.1 |
|
@chandlerkent I tried to revert the change and test it using 2.1.0 module. The task failed even with 2.1.0 version of the module. Can you confirm that you were using storage account of type Premium File Storage and were able to execute the task successfully with versions V2 and V3 ? |
|
@asranja Thank you for taking a look. We were previously using the V2 task and have been successfully for many months without any changes to the storage account (it is part of a pipeline that runs each day). In fact, we are still using V2 only because that was the latest version that was out when the pipeline was created. I don't believe we are using a Premium File Storage account. The SKU is "StorageV2 (general purpose v2)". We were not using the V3 version of the task so I do not know if it used to work. I do know that V3 did not work as we tried upgrading to it after V2 stopped working yesterday. Is there anything we can do temporarily to get our pipelines working while the bug gets investigated or do we need to wait for a fix to the task? I don't think there is a way for us to use the older version of the task, is there? |
|
@chandlerkent Can you point out the error you got with Azure File Copy V4 task ? |
|
@asranja V4 task gives us a permissions error. Task info: Error: The Service Principal is actually an owner of the entire subscription with no deny statements on this Storage Account. |
|
Yes, this was a breaking change with Azure File Copy V4 (hence a new version). The Service Principal needs to also have Storage Blob Data Contributor permissions on the Storage account. This is because of AzCopy.exe logging in using the Service Principal Credentials in V4 task. Can you assign the permission and re-queue the pipeline. More details on Azure File Copy V4 task and roles. AzureFileCopyV4 task is based on new Az modules as opposed to older AzureRm modules. V4 is the way going forward. That being said, we are working on finding the root cause in V2 and V3, and will come with a fix asap. |
|
@asranja Ah OK, thank you! I will try that and let you know! |
|
@asranja Just to follow up that did fix the issue and the V4 task is now working after I gave the Service Principal the correct permissions on the Storage Account. Thank you! If you need any tests of the V2/V3 tasks, I would still be happy to test those and verify they will fix the reported issue. Thank you again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: Azure file copy
Environment
Server - Azure Pipelines
Account name - dalgs
Team project name - Digital Workplace
Release definition - directory-webapp-release/202001.5
Agent - Hosted windows-2019
Issue Description
Task logs
ReleaseLogs_561.zip
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
2020-01-15T16:49:40.7822699Z ##[debug][Azure Call]Getting resource details for azure storage account resource: dgsappdirectoryint with resource type: Microsoft.Storage/storageAccounts 2020-01-15T16:49:42.1074825Z ##[debug]System.NullReferenceException: Object reference not set to an instance of an object. 2020-01-15T16:49:42.1096413Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.Models.PSStorageAccount..ctor(StorageAccount storageAccount) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\Models\PSStorageAccount.cs:line 27 2020-01-15T16:49:42.1111457Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.Models.PSStorageAccount.Create(StorageAccount storageAccount, IStorageManagementClient client) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\Models\PSStorageAccount.cs:line 87 2020-01-15T16:49:42.1132108Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.StorageAccountBaseCmdlet.<>c__DisplayClass2.<WriteStorageAccountList>b__1(StorageAccount storageAccount) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\StorageAccount\StorageAccountBaseCmdlet.cs:line 161 2020-01-15T16:49:42.1146318Z ##[debug] at Microsoft.WindowsAzure.Commands.Utilities.Common.GeneralUtilities.ForEach[T](IEnumerable1 sequence, Action1 action) in d:\workspace\powershell-sign\src\Common\Commands.Common\GeneralUtilities.cs:line 147 2020-01-15T16:49:42.1167097Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.StorageAccountBaseCmdlet.WriteStorageAccountList(IEnumerable1 storageAccounts) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\StorageAccount\StorageAccountBaseCmdlet.cs:line 1612020-01-15T16:49:42.1180004Z ##[debug] at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord() in d:\workspace\powershell-sign\src\Common\Commands.Common\AzurePSCmdlet.cs:line 610
2020-01-15T16:49:42.3306724Z ##[debug]Processed: ##vso[task.logissue type=error;code={"Task_Internal_Error":"TemporaryCopyingToBlobContainerFailed"};]
2020-01-15T16:49:42.3351803Z ##[debug]Trying to disconnect from Azure and clear context at process scope
2020-01-15T16:49:42.5623196Z ##[debug]Caught exception from task script.
2020-01-15T16:49:42.5655643Z ##[debug]Error record:
2020-01-15T16:49:42.6254818Z ##[debug]Get-AzureRmStorageAccount : Object reference not set to an instance of an object.
2020-01-15T16:49:42.6267065Z ##[debug]At D:\a_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\2.154.16\AzureUtilityGTE1.0.ps1:27 char:47
2020-01-15T16:49:42.6279224Z ##[debug]+ ... ResourceDetails = Get-AzureRmStorageAccount -ErrorAction Stop | Where ...
2020-01-15T16:49:42.6290268Z ##[debug]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-01-15T16:49:42.6303078Z ##[debug] + CategoryInfo : CloseError: (:) [Get-AzureRmStorageAccount], NullReferenceException
2020-01-15T16:49:42.6314916Z ##[debug] + FullyQualifiedErrorId : Microsoft.Azure.Commands.Management.Storage.GetAzureStorageAccountCommand
2020-01-15T16:49:42.6326337Z ##[debug]
2020-01-15T16:49:42.6345394Z ##[debug]Script stack trace:
2020-01-15T16:49:42.6379888Z ##[debug]at Get-AzureStorageAccountResourceGroupName, D:\a_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\2.154.16\AzureUtilityGTE1.0.ps1: line 27
2020-01-15T16:49:42.6392058Z ##[debug]at Get-AzureStorageKeyFromARM, D:\a_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\2.154.16\AzureUtilityRest.ps1: line 28
2020-01-15T16:49:42.6406408Z ##[debug]at Get-StorageKey, D:\a_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\2.154.16\Utility.ps1: line 149
2020-01-15T16:49:42.6422458Z ##[debug]at , D:\a_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\2.154.16\AzureFileCopy.ps1: line 101
2020-01-15T16:49:42.6434089Z ##[debug]at , : line 1
2020-01-15T16:49:42.6446172Z ##[debug]at , : line 22
2020-01-15T16:49:42.6459269Z ##[debug]at , : line 18
2020-01-15T16:49:42.6470994Z ##[debug]at , : line 1
2020-01-15T16:49:42.6489785Z ##[debug]Exception:
2020-01-15T16:49:42.6511296Z ##[debug]System.NullReferenceException: Object reference not set to an instance of an object.
2020-01-15T16:49:42.6523029Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.Models.PSStorageAccount..ctor(StorageAccount storageAccount) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\Models\PSStorageAccount.cs:line 27
2020-01-15T16:49:42.6535296Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.Models.PSStorageAccount.Create(StorageAccount storageAccount, IStorageManagementClient client) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\Models\PSStorageAccount.cs:line 87
2020-01-15T16:49:42.6547081Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.StorageAccountBaseCmdlet.<>c__DisplayClass2.b__1(StorageAccount storageAccount) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\StorageAccount\StorageAccountBaseCmdlet.cs:line 161
2020-01-15T16:49:42.6558457Z ##[debug] at Microsoft.WindowsAzure.Commands.Utilities.Common.GeneralUtilities.ForEach[T](IEnumerable
1 sequence, Action1 action) in d:\workspace\powershell-sign\src\Common\Commands.Common\GeneralUtilities.cs:line 1472020-01-15T16:49:42.6569978Z ##[debug] at Microsoft.Azure.Commands.Management.Storage.StorageAccountBaseCmdlet.WriteStorageAccountList(IEnumerable
1 storageAccounts) in d:\workspace\powershell-sign\src\ResourceManager\Storage\Commands.Management.Storage\StorageAccount\StorageAccountBaseCmdlet.cs:line 161 2020-01-15T16:49:42.6581642Z ##[debug] at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord() in d:\workspace\powershell-sign\src\Common\Commands.Common\AzurePSCmdlet.cs:line 610 2020-01-15T16:49:42.6641627Z ##[error]Object reference not set to an instance of an object. 2020-01-15T16:49:42.6642015Z ##[debug]Processed: ##vso[task.logissue type=error]Object reference not set to an instance of an object.The text was updated successfully, but these errors were encountered: