Having issues with AWS OpsWorks installation

[sainejob]

I'm trying to get the agents to install through AWS OpsWorks (Chef 11.10) and receiving the following:

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of ./al-agent provision --key [KEY REDACTED] --inst-type host ----
STDOUT: 
STDERR: May 11 22:40:20 2016 al-agent[24242]: ALC00089I [alc_config_set_api_key] setting api_key
May 11 22:40:20 2016 al-agent[24242]: ALC00083I [unregister] clean up registration artefacts
May 11 22:40:20 2016 al-agent[24242]: ALC00315E [alc_socket_send_request] Response http code is 301
May 11 22:40:20 2016 al-agent[24242]: ALC00074E [alc_config_provision_host] Could not issue provisioning request
May 11 22:40:20 2016 al-agent[24242]: ALC00079E [alc_config_init] Could not provision the host
---- End output of ./al-agent provision --key [KEY REDACTED] --inst-type host ----
Ran ./al-agent provision --key [KEY REDACTED] --inst-type host returned 1

Not sure where the 301 Moved Permanently is coming from....

Any ideas?

[motobob]

@sainejob which agent version are you using?
301/307 are for cross-DC support and are valid responses from AL backend where agent have to follow the redirects. latest agents should handle it gracefully.
In AWS env please check outbound rules, probably a redirect is handled but it cannot connect to other DC.

[sainejob]

Using al_agents 1.3.2 --- I'm assuming it installs the latest agent as of yesterday https://scc.alertlogic.net/software/al-agent_LATEST_amd64.deb

We verified the outbound rules on the on-boarding call. I can try and recheck them.

[motobob]

@sainejob well. that seems not the right version and very old one. the version of the al_agents and not the agent it self. but if you just installed it should pick the latest

Please also check that you are using the correct API_KEY and you AWS environment has outbound rule internet gateway so the agent to reach all Alert Logic DCs.
Have you tried the agent prov command manually and does it give the same result?

If the key is valid I would recommend contacting AL support directly to debug agent installation issue.

[sainejob]

I just downloaded https://scc.alertlogic.net/software/al-agent_LATEST_amd64.deb
and verified that manually installation worked fine. No config commands were run previously.

[sainejob]

I was able to install it manually on one host but when I tried again on a separate host, I see the 301. Trying now with a fresh host.

[sainejob]

OK - not sure why this occurs but I have a work-around which seems to work consistently with a fresh instance. I separated out the recipes instead of just using simply al_agents or al_agents::default

I use the following with for_imaging: true config so the install recipe doesn't automatically run the configure_agent for me. This code (https://github.com/alertlogic/al_agents/blob/master/recipes/install.rb#L31):

include_recipe 'al_agents::configure_agent' unless for_imaging
include_recipe 'al_agents::provision_agent' unless for_imaging

Here is the order in which I run the recipes for successful provisioning and agent start-up:

---

OpsWorks 'configure' lifecycle:

If I run al_agents by itself, I get the 301 and it leaves the host in a bad state. I am even unable to manually install after this fails. I had to log into the host and dpkg --purge al-agent; rm -rf /var/alertlogic before I could successfully do a manual install.

My guess is that the configure_agent recipe leaves it in a bad state for some reason. I cut it out of the equation and it works for me.

Chef 11.10 - AWS OpsWorks - Ubuntu 14.04

[jearly]

@sainejob We have been able to reproduce this behavior and are working towards a resolution. We will update you once we have a solution.

[jearly]

@sainejob This issue is fixed in the latest Pull Request #65 Once all the tests complete, we will merge and release the latest version to the supermarket.
Essentially, the issue was in the configure command. By default the egress helper was validating the egress url and then being passed into the configure_options as --host host:port This was valid at one point, but the agent package now requires --host and --port be passed separately to the configure command in order to properly redirect to the correct location to be provisioned. I have tested this on all supported OS's and this fix should resolve your issue. Please let us know if you continue to have issues after we release this fix to the supermarket later today.

[jearly]

@sainejob This change has been published to the supermarket. Please let us know if you have any further issues.

https://supermarket.chef.io/cookbooks/al_agents

[sainejob]

@jearly Looks like you packaged a version missing my latest change a08589a

This is required to work on Chef 11.10

[jearly]

@sainejob I apologize. You are correct. I did not rebase your change prior to build. I have rebased with your change and bumped to version 1.3.4 in the supermarket. Chef 11.x compatibility should be restored. I apologize for the inconvenience.