Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure File Copy build task not available on hosted Ubuntu build agent #8920

Open
benne opened this issue Nov 27, 2018 · 28 comments
Open

Azure File Copy build task not available on hosted Ubuntu build agent #8920

benne opened this issue Nov 27, 2018 · 28 comments

Comments

@benne
Copy link

@benne benne commented Nov 27, 2018

Environment

  • Server - Azure Pipelines
    Account name: chaosbyte
    Team project name: D Test
    Build definition name: D Test-CI
    Build number: 9

  • Agent - Hosted:
    Queue name: Not sure
    It is hosted a Ubuntu 16.04 build agent

Issue Description

I got a project building some pulling from an external Git repository, and building it with gcc through a simple bash script - no problem.
However, when I attempt to use the Azure File Copy task to copy over the single output file to a Azure Blob Storage container, I am told that this task won't run on Linux.
Is there any plans to make Azure File Copy work on the hosted Ubuntu 16.04 build agent, or do you have any other suggestion on how to achieve the same result with existing tasks that will run on this hosted build server?

Error logs

2018-11-27T18:03:41.7033273Z ##[section]Starting: AzureBlob File Copy
2018-11-27T18:03:41.7038323Z ==============================================================================
2018-11-27T18:03:41.7038468Z Task         : Azure File Copy
2018-11-27T18:03:41.7038567Z Description  : Copy files to Azure blob or VM(s)
2018-11-27T18:03:41.7038647Z Version      : 2.0.15
2018-11-27T18:03:41.7038738Z Author       : Microsoft Corporation
2018-11-27T18:03:41.7038823Z Help         : [More Information](https://aka.ms/azurefilecopyreadme)
2018-11-27T18:03:41.7038939Z ==============================================================================
2018-11-27T18:03:41.7244544Z ##[error]The current operating system is not capable of running this task. That typically means the task was written for Windows only. For example, written for Windows Desktop PowerShell.
2018-11-27T18:03:41.7258675Z ##[section]Finishing: AzureBlob File Copy
@chrispat
Copy link
Contributor

@chrispat chrispat commented Nov 27, 2018

@benne you could use the az cli to do the file copy and that task will work on Linux

@benne
Copy link
Author

@benne benne commented Nov 27, 2018

@chrisrpatterson az cli might be my best option for now, using a non-interactive service principal. I am gonna experiment a bit with this.

Still interested in if this is gonna make it to the road-map at some point. It would be nice to use the build-in build task for future projects :)

@chrispat
Copy link
Contributor

@chrispat chrispat commented Nov 27, 2018

@benne I will leave the roadmap for that task up to the team that owns the task. The az cli task should work with the same service principal you would have setup for the Azure File Copy task.

@benne
Copy link
Author

@benne benne commented Nov 27, 2018

@chrisrpatterson I went with a service principal with specific permissions to contribute to a specific storage account. Combined with an Azure CLI build task I was able to upload the assets to the blob storage.
Success!

@kmkumaran
Copy link
Member

@kmkumaran kmkumaran commented Nov 29, 2018

@benne - Making Azure File copy xPlat is in our backlog. Adding @RoopeshNair . Until then using Azure CLI will be the right way to solve.

@RoopeshNair
Copy link
Member

@RoopeshNair RoopeshNair commented Jan 16, 2019

Bouncing an idea here, we are evaluating whether we can expose Azure storage as a file drive. When available you can copy the files over using SSH or Remote PS.

@luyseyal
Copy link

@luyseyal luyseyal commented Sep 22, 2019

azcopy is available on Linux
https://aka.ms/downloadazcopy-v10-linux

@chshrikh chshrikh assigned niadak and N-Usha and unassigned RoopeshNair and kmkumaran Dec 5, 2019
@mounarshisantra
Copy link

@mounarshisantra mounarshisantra commented Dec 8, 2019

if we are using the az cli to achieve the same functionality, we are not getting the same flexibility as the Azure File Copy task in terms of getting the storage uri and sastoken as output. We need to make additional cli calls to achieve the same . So, it would be great if the file copy task works xPlat.

@motowilliams
Copy link

@motowilliams motowilliams commented Feb 14, 2020

The AzureFileCopy task and the az cli workaround is not at feature parity with each other. I wonder how many teams, like I just did, spent time adding the task to the pipeline, watching it fail and then searching the internet for a couple of hours. This is how I found this issue. The docs on this are also weak.

@titimoby
Copy link

@titimoby titimoby commented Mar 4, 2020

Is there any progress to this?
It is very strange that tasks like JavaToolInstaller are able to copy installer file from Azure Storage but we can't use AzureFileCopy to copy files to Azure Storage.

@minhtrieu-aemo
Copy link

@minhtrieu-aemo minhtrieu-aemo commented Mar 31, 2020

Trying to migrate our pipelines to use self-hosted Linux agent and discovered that its failing due to AzureFileCopy not being supported on Linux. It would be great if this can be made to work with Linux!

@weznagwama
Copy link

@weznagwama weznagwama commented Apr 30, 2020

Another bump for an update on this.

@alexangas
Copy link

@alexangas alexangas commented May 1, 2020

I've written a YAML task template for Azure Pipelines that fills this gap for now. You can find it on my blog here: https://www.alexangas.com/blog/missing-azurefilecopy-azcopy-task-solved/

Feedback is very welcome! Please raise as an issue on the repo for my blog rather than polluting this thread, thanks.

@veton
Copy link

@veton veton commented May 3, 2020

For the time being, using windows image as a workaround:

  - deployment: deploy_staging
    environment: staging
    pool:
      vmImage: windows-latest
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: $(artifactName)

          - task: AzureFileCopy@3
            inputs:
              sourcePath: $(Pipeline.Workspace)/$(artifactName)
              azureSubscription: $(azureSubscription)
              destination: azureBlob
              storage: memes
              containerName: $web

@requilium
Copy link

@requilium requilium commented May 14, 2020

This also left me very confused. This obviously should work on linux hosts.

@tckb
Copy link

@tckb tckb commented Jun 2, 2020

More than a year later, the docs are still stale. Thanks for the work around @veton

@jamiehaywood
Copy link

@jamiehaywood jamiehaywood commented Jun 8, 2020

is there an update on this?

@jamiehaywood
Copy link

@jamiehaywood jamiehaywood commented Jun 8, 2020

I wanted to share my own solution to this that works with vmImage:ubuntu-latest:

  - task: AzureCLI@1
    displayName: Az File Copy to Storage
    inputs:
      azureSubscription: $(azureSubscription)
      scriptLocation: inlineScript
      inlineScript: |
        az storage blob upload-batch \
          --destination \$web \
          --account-name "$(storageAccountName)" \
          --source "$(Agent.BuildDirectory)/$(outputDir)"

@sebastianrogers
Copy link

@sebastianrogers sebastianrogers commented Jun 11, 2020

Critical thing here is that Azure File Copy will return a SAS Token allowing you to then access these files via ARM deployment.

@trondkla
Copy link

@trondkla trondkla commented Aug 21, 2020

Any progress? This should be a thing for linux too :)

@xortim
Copy link

@xortim xortim commented Aug 21, 2020

Not to +1 and "me too" but if the guidance from MS is to use azcopy, then they should support it in their products. What happened to MS ❤️ Linux? :trollface:

@beauof
Copy link

@beauof beauof commented Oct 2, 2020

Thanks for sharing your workarounds! Yet, I am still waiting for the AzureFileCopy task on Linux agents because I can't always remember this issue until I see another failed pipeline. :)

@jwbrindle
Copy link

@jwbrindle jwbrindle commented Nov 30, 2020

umm, have been hoping to see some kind of update on this for over a year, what gives guys, can we get some support on this? GA opened since azcopy v10 for linux, why couldnt this be integrated?

@cee-money
Copy link

@cee-money cee-money commented Dec 3, 2020

+1, it would be great to see an update on this

@ascott18
Copy link

@ascott18 ascott18 commented May 6, 2021

This is a terrible experience for your customers. Why is something so basic as copying files not supported on Linux agents the same as it is supported on Windows?

@nadesu nadesu assigned vijayma and unassigned N-Usha May 7, 2021
@nadesu
Copy link

@nadesu nadesu commented May 7, 2021

@vijayma pls triage

@dancooktoptal
Copy link

@dancooktoptal dancooktoptal commented Jul 1, 2021

+1

1 similar comment
@ThePiz
Copy link

@ThePiz ThePiz commented Aug 26, 2021

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet