New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CodeDeploy support #14
Comments
I am not using Code Deploy, so I'm not familiar with the hooks, but please try it out and let me know if it works out of the box, or otherwise I'll try to come up with a solution for supporting this. Do you know how are these CodeDeploy hooks interacting with new instances attached manually to the group after they were running for a while outside the group? That's essentially what autospotting is doing under the hood. |
Hmm I will have to check that our, that will take a few days... |
This is the lifecycle hook of the CodeDeploy integration:
|
I read a bit through the CodeDeploy docs and I think it should work, but please test it first on a non-production environment.
But you may get into trouble in the very unlikely case that...
There is a chance that all the live nodes would get deployed at the same time and their service is restarted and may cause some downtime. To reduce this risk, I would recommend to use a rolling procedure which updates one node at a time, to block deployments when you first enable AutoSpotting on your existing AutoScaling group, or to increase the group's capacity to reduce the chance to deploy all of the live nodes at once.
This should also work, except for the not quite so unlikely case in which...
In this case I recommend to do something to speed up your deployments or make your ELB checks take longer to kill the instances than the time you need for the deployment. Another approach may be to switch your AutoScaling group to EC2 health checks. I would recommend to test both scenarios if you can, but I think that a deployment group based on EC2 tags would be a better idea because it would be more unlikely to get into that 'perfect storm' failure scenario. Let me know what you think, or if I am missing anything. |
Did a little testing with this today and thought I'd share the results. In my setup our CodeDeploy groups are based on AutoScale groups. Once the new spot instance was added to the AutoScale group nothing happened. CodeDeploy hooks into the lifecycle process of the AutoScale group, so I expect that Autospotting is missing out a critical piece of the process. If our AMI was pre-baked, this wouldn't be an issue, but then we wouldn't use CodeDeploy either. |
I'm not that familiar with CodeDeploy so correct me if I'm wrong, but according to the introduction's page: http://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html It looks like to me, when CodeDeploy manages the instance to create, it also injects meta-data regarding the deployment (version, repo, etc), these being used by the agent to deploy the code. What you could do is check the presence of the agent and metada on those 2 types of instances (spawned from CodeDeploy and the Spot instances). I think it's worth checking those elements and provide further feedback. |
As per this document https://aws.amazon.com/blogs/devops/under-the-hood-aws-codedeploy-and-auto-scaling-integration/
At the time those notifications are sent for your new spot instances they are not yet added to the group so CodeDeploy simply ignores them when configured based on group membership. @nerk01 try configuring CodeDeploy based on an instance tag set on your group, since it turns out the AutoScaling group membership method is broken due to the way CodeDeploy handles the lifecycle hooks. If that works for you I would like to have this documented as a known issue. |
CodeDeploy doesn't handle newly launched instances, even with tags. So, having a deployment group with both autoscaling membership and instance tags doesn't solve the issue. Here is a workaround: Whenever an instance is launched, trigger a deployment (via CloudTrail+SNS) in the same group with option |
Thanks @acsrujan, please create a PR with a documentation update in which to mention this workaround |
@acsrujan can you give a bit more detail on how you accomplished doing this? I understand the context but not sure I understand the implementation. We are having this exact same issue and would love to have a solution. |
I have no experience with CodeDeploy so I am just shooting in the dark here, but at a first glance I think the simplest way to do it is using the userdata script, all instances may execute a deployment using the aws command line tool and the flag mentioned above by @acsrujan. |
I was able to get this working via a simple bash script that gets my tags of the instance then uses them to run:
That will return a deployment-id that checks to see if any instances within that deployment group haven't had any code deployed to them. This could probably be converted to a simple lambda function that registers the new instances, checks for tags like: deployment_group and application_name and registers a new deployment to it. I didn't have the time to try and work that out, but I appreciate @acsrujan pointing me in the right direction! |
We're experiencing the same problem and the above command worked as expected. Any ideas around how to automate executing that command when a spot instance is launched? |
@binarylogic What I did was set the deployment group to use tag groups. Then added these two scripts to the ami:
We remove the .sh and store in /usr/bin. Once those are added and customized to what you need just add something like:
Or wherever you want. Can also be added a user-run script but I don't like putting things in there so did it this way. |
@binarylogic, @sc-chad this is really useful information, so let's document it properly. Please any of you create a PR for this. |
@cristim trade ya - I'll write up a PR to fully document what I've done if you take a look at my Name:tag issue! :) I'll totally write it up anyway, but figured I'd try! |
@sc-chad I gave you some suggestions for the Name tag, this should be relatively easy to implement, although it may require a bit of refactoring. Please create a PR and I will do my best to help you out with the implementation. |
I might have some time as well to take a look at issues |
The current version of AutoSpotting trunk implements CodeDeploy support against ASG deployment groups, so the previous workaround using tag-based deployment groups should no longer be necessary. Please try it out and report back if you notice any issues. |
My current autoscaling groups use CodeDeploy to deploy the web applications. CodeDeploy uses Hooks in the AutoScaling Group for that, is this supported out of the box?
The text was updated successfully, but these errors were encountered: