Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to install AWS CLI on VSTS Hosted windows agent or will that be a default feature of hosted agent in the future. #17

Open
tankthomas opened this issue Sep 11, 2017 · 20 comments

Comments

Projects
None yet
@tankthomas
Copy link

commented Sep 11, 2017

ran into this issue when running aws cli task on hosted windows agent "2017-09-11T20:45:32.5563058Z ##[error]AWS CLI is not installed on this machine."
it would make more sense that it comes by default. I'm wondering if it will be released in the future?

@steveataws

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2017

Given the different environments and ways to install the CLI, and the requirement to install Python etc (at least I think that still needs to be done) we're thinking about documenting the install process as part of the task reference, as a script that can be copied into a shell command task. I don't think we are going to try and do the install automatically (but you never know).

@steveataws steveataws added the question label Sep 27, 2017

@J00MZ

This comment has been minimized.

Copy link

commented Dec 6, 2017

Just had this issue on Linux Hosted agent.
##[error]AWS CLI is not installed on this machine.
Must the 'AWS CLI' step be dependent on a previous manual installation task on the remote agent?
@steveataws what is the roadmap plan for adding this task you mentioned?

@coreyperkins

This comment has been minimized.

Copy link

commented Apr 5, 2018

Still no answer?

@renatorcosta

This comment has been minimized.

@J00MZ

This comment has been minimized.

Copy link

commented Apr 19, 2018

@renatorcosta the issue here is not how to install the AWS CLI. we've all done that.
The issue is that we would expect the official VSTS AWS integration to include the AWS CLI installation by default.

@Rabadash8820

This comment has been minimized.

Copy link

commented May 30, 2018

@steveataws This all makes me wonder what use case you were even trying to address with this Task. There's no way to get the return values from a get-* or list-* operation; we'd be better off writing a custom script to do that and write the output to a temporary file. And now I find that the AWS CLI doesn't come even come pre-installed, so we have to write a separate script to install it first. So again, what is even the point of having this Task?

Sorry for sounding harsh, I love all your other AWS Tools Tasks. This one in particular just seems like it was a waste of man hours...

@srini85

This comment has been minimized.

Copy link

commented Aug 14, 2018

I couldnt get this to work either on hosted VSTS windows box. I tried numerous ways of installing without luck. My alternative was to just use the AWS SDK in node and create lamda functions. Used the invoke lamda functions to do various devops tasks. Bit of a long winded approach, but its a workaround until there is a proper solution :)

@steveataws

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2018

Huge apologies, I wasn't aware of any follow up correspondence on this issue and have been mostly focused elsewhere.

I totally agree we haven't done a great job of addressing the 'why' of this task, together with finding some convenient way to make it work better in a hosted environment, and integration with the environment itself. Our integration test setup uses a Team Foundation Server setup, so we have the CLI pre-installed for the build agents. This is possible with VSTS too, but we don't document it (I did find a useful blog post on this while discussing it with a colleague yesterday, which also reminded me I'd not recalled any activity on the issue for a while).

The task was intended (like the PowerShell task) as a catch-all for scenarios for working with AWS that our other tasks didn't fulfill at the time. We can't bundle the cli with the tools as there is a size limit in the marketplace for extensions therefore our only options are either to document the process (which we also haven't done a great job on) or find some way to do an automatic install, like we do with the AWS Tools for Windows PowerShell module - although that too has caused its fair share of issues and now we have users wanting us to install PowerShell 6 and the corresponding AWS module for use on Linux hosts too.

The cli with its Python dependencies requires a slightly more involved install process but, if we were to be sure that Python was available pre-installed on all hosted agents, it might be possible for us to run the 'pip' install automatically - I don't know whether Python is in fact always available on all hosts however. Is someone more familiar with all the different hosts able to more quickly confirm or deny this is the case?

Regarding the output issue from the task, my current suggestion is for us to extend the task so that the output can be captured into a build variable and thus processed further downstream in builds. Alternatively, the AWS shell script task that we recently added could be used to install the cli (for now, if necessary), run the command(s) and do the output processing, using the formatted console output to inject the data you need into VSTS build variables for downstream tasks.

Wish I had better news at this stage but the task isn't 'dead', we're just not at all sure of the best way to address the auto-install issue on multiple platforms with the Python dependency and are certainly open to suggestions.

@srini85

This comment has been minimized.

Copy link

commented Oct 2, 2018

@steveataws perhaps a discussion with the azure pipelines team for them to increase the size limit would all that would be required? They are obviously pushing their service far and are embracing the market place and perhaps would be open to an exception to the file size limitation?

@steveataws

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2018

Even if we managed to get the size limit raised we'd then find ourselves in a bind on the CLI version bundled with the tools. The AWS SDKs and the CLI update almost daily. The VSTS tools rev much more slowly. So if we bundled the CLI we'd then potentially have users confused as to why recent service APIs or new services were not present.

This (plus size) is why we set the PowerShell task to auto-install instead of bundling (plus it was much easier since we know the PowerShellGet commands to perform installs are present on the Windows hosts). A similar process which would always pull down the latest CLI if not installed would avoid bundled version issues and is the approach I think we need to follow (perhaps with an option to try to do an update if already installed).

@Rabadash8820

This comment has been minimized.

Copy link

commented Oct 4, 2018

@steveataws Great points, thanks for the update. In reference to whether Python is always available, there is a Microsoft Docs article that describes the software available on hosted VSTS agents. Unfortunately, it seems to only describe the operating systems. Maybe one of the authors of that topic can give more specifics on pre-installed tools/frameworks?

@steveataws

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2018

I've heard from others inside AWS that apparently Python is available on all hosts (but haven't confirmed it) but that 'pip install' errors out about a new version of pip being available. My hunch is that this is not actually an error, just a misinterpretation of an exit code - I see the same message when installing the CLI on my Windows laptop.

If that's the case, it should be possible for the task to be updated to trigger an install (or update if the CLI is found) and squelch the pip output, but it needs some testing to confirm.

@jasonwhetton

This comment has been minimized.

Copy link

commented Oct 23, 2018

This worked for me, with a Linux build agent:

pip install setuptools
pip install wheel
pip install awscli --upgrade

The pip version output is just a notice, not an error.

@vinla

This comment has been minimized.

Copy link

commented Dec 23, 2018

@jasonwhetton
How did you get this to work. I have a command line task to run those commands, but even directly after that I still get aws command not found
The log shows the cli getting installed ok, but it still doesn't work.

@jasonwhetton

This comment has been minimized.

Copy link

commented Dec 24, 2018

@vinla I was using the Hosted Linux Preview, it's not working on the Ubuntu 16.04 agent. Still trying to get it working there (hopefully before they remove the preview!).

@vinla

This comment has been minimized.

Copy link

commented Jan 2, 2019

@jasonwhetton I got this working on Ubuntu 1604 by prefixing the 3 commands with sudo

@jasonwhetton

This comment has been minimized.

Copy link

commented Jan 2, 2019

@vinla thanks for coming back and posting - you just saved my afternoon :)

@SPhu

This comment has been minimized.

Copy link

commented Jan 20, 2019

I had trouble getting aws cli installed on the agent also but it wasnt an aws issue. I will try Vinia's suggestion. I did work around this already by using my own container on the build agent. I find it to be more consistent and predictable that the vsts environment. I find vsts is alot of slow and painful trial and error and the documentation doesn't really cover real build scenarios just small pieces

docker pull your-container
docker run --name container-name -dt image-name
docker exec aws-cli aws sts get-caller-identity
docker stop container-name

@danparker276

This comment has been minimized.

Copy link

commented Feb 26, 2019

This is broken now again, it used to work until yesterday on CMD for me, just says:
Script contents:
pip install awscli
##[command]"C:\windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL
"D:\a_temp\XXXXXXXXXXXXXXXXXXXXX.cmd""
##[error]Cmd.exe exited with code '1'.
##[section]Finishing: Add AWS CLI

@danparker276

This comment has been minimized.

@justnance justnance added guidance and removed question labels Apr 20, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.