CodeDeploy Fails on ApplicationStop - host agent restart solves it

[jasong1987]

We have blue/green setup and have found this intermittent issue across multiple accounts we have.

Windows server 2019 and the launch template installs the latest version of codedeploy host agent on boot
sometimes codedeploy fails on ApplicationStop with no error details at all. a quick restart of the codedeploy host agent service on the existing ec2 instance and the retry works. There must still be an issue with codedeploy going into sleep mode or crashing in windows server 2019 - please help this stops our whole codepipeline form being automated currently

[philstrong]

Need more information. Are there any logs on the host?

[jasong1987]

Need more information. Are there any logs on the host?

this is exactly the same issue I was having - there are no logs at all from the deployment on the ec2 instance - it is literally like it fails because the host agent service has crashed - so effectively code deploy cant connect to it to even start the logs let alone the deployment. I have found no way to get any useful error messages.

The strange thing is if I check the service it shows as a running state but without fail everytime if I restart the service and try the deployment again it works fine with no errors or issues

[philstrong]

Does it happen after a recent update to the agent? Reboot of the machine?

[jasong1987]

We are auto installing the latest version of the agent with user data (on a windows 2019 AMI) via a launch template. I would say it started doing this around November but is very intermittent - sometimes it works as expected. The longer between deployments the more the error seems to happen which is what makes me think the service is crashing out in windows somehow after a while

[philstrong]

Concerning,
Do you have it configured to use VPCe or proxy settings that might be relevant?

mind sharing C:\ProgramData\Amazon\CodeDeploy\conf.yml from one of the affected hosts?

[jasong1987]

The EC2 is setup and joined to a VPC within AWS yes but no proxy setup

below is the conf file:

:log_dir: 'Amazon/CodeDeploy/log'
:root_dir: 'Amazon/CodeDeploy'
:verbose: true
:wait_between_runs: 1
:wait_after_error: 1
:bundle_name: 'artifact_bundle.tar'

and these lines are repeated in the codedeploy-agent-log file:
2021-02-04T12:11:23 DEBUG [codedeploy-agent(1588)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command = nil
2021-02-04T12:11:24 DEBUG [codedeploy-agent(1588)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2021-02-04T12:11:24 INFO [codedeploy-agent(1588)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.3.1.1880_msi.

Also within the event viewer I have found a warning on the new ec2 instance - event ID 7039:
A service process other than the one launched by the Service Control Manager connected when starting the CodeDeploy Host Agent Service service. The Service Control Manager launched process 624 and process 1588 connected instead.

Note that if this service is configured to start under a debugger, this behavior is expected.

[jasong1987]

Just had the same issue again. Does the above information help in anyway?

[jasong1987]

for anyone else having the same issue I have got around this by adding a scheduled task to stop and start the service everyday
the below code can be added to userdata in a launch template
#Setup scheduled task to stop and restart the codedeploy host agent service $action = New-ScheduledTaskAction -Execute 'powershell' -Argument 'powershell.exe -Command Stop-Service -Name codedeployagent;powershell.exe Start-Sleep -Seconds 20;powershell.exe -Command Start-Service -Name codedeployagent;' $trigger = New-ScheduledTaskTrigger -Daily -At 7am Register-ScheduledTask -Action $action -Trigger $trigger -TaskName "ResetCodeDeployHostAgent" -Description "Daily Reset of CodeDeployHostAgent service"

[mwjones-aws]

Please reopen if you're still experiencing this issue with the latest version of CodeDeploy Agent