New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Developer experience #136

Open
kaihendry opened this Issue Jul 19, 2016 · 16 comments

Comments

Projects
None yet
5 participants
@kaihendry

kaihendry commented Jul 19, 2016

Sorry for the question here, though I am trying to understand how to facilitate the developer workflow for our deployment on AWS ECS.

In the company I work for, we will be setting up several staging "services", that need to be "Updated" ideally on a git push to Github. Does the amazon-ecs-cli help here with the hook?

And then we need an interface to push (ideally with some sort of "sign-off") approved staging code to production service. Is amazon-ecs-cli the tool to use here?

Many thanks,

@uttarasridhar

This comment has been minimized.

Show comment
Hide comment
@uttarasridhar

uttarasridhar Jul 20, 2016

Member

Hello @kaihendry ,
Unfortunately amazon-ecs-cli, as of v0.4.1 doesn't have a github hook natively.

To unblock yourself, you can integrate with existing automated build pipelines (like Jenkins, AWS Code pipelines, CodeShip) for stage managements with approvals. But at each stage, once you have an image, amazon-ecs-cli just requires you to execute ecs-cli compose service up to update your services.

Hope that helps.

Member

uttarasridhar commented Jul 20, 2016

Hello @kaihendry ,
Unfortunately amazon-ecs-cli, as of v0.4.1 doesn't have a github hook natively.

To unblock yourself, you can integrate with existing automated build pipelines (like Jenkins, AWS Code pipelines, CodeShip) for stage managements with approvals. But at each stage, once you have an image, amazon-ecs-cli just requires you to execute ecs-cli compose service up to update your services.

Hope that helps.

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 20, 2016

Thank you! However I still have more questions: 😁

ecs-cli compose service up ... I'm a little confused by this command. Does it map to Docker compose or to the way I've setup the service in the ECS console?

Does it update so that there is no downtime, assuming i am running 2 tasks on separate instances?
Is there downtime when "1 task on 1 instance" is updated out of interest or is that agent smart enough to swap around the containers with connection draining et al.?

How do I roll back in case there is an issue? I was kindof hoping "task revisions" would be for this, but I don't think it tracks or binds to a particular image identifier, or does it?

Are there ENVs or tricks to expose what image I am using & what instance I am on to make sure my deployment is doing what it should do before I roll out to production and millions of users? Thank you!

kaihendry commented Jul 20, 2016

Thank you! However I still have more questions: 😁

ecs-cli compose service up ... I'm a little confused by this command. Does it map to Docker compose or to the way I've setup the service in the ECS console?

Does it update so that there is no downtime, assuming i am running 2 tasks on separate instances?
Is there downtime when "1 task on 1 instance" is updated out of interest or is that agent smart enough to swap around the containers with connection draining et al.?

How do I roll back in case there is an issue? I was kindof hoping "task revisions" would be for this, but I don't think it tracks or binds to a particular image identifier, or does it?

Are there ENVs or tricks to expose what image I am using & what instance I am on to make sure my deployment is doing what it should do before I roll out to production and millions of users? Thank you!

@uttarasridhar

This comment has been minimized.

Show comment
Hide comment
@uttarasridhar

uttarasridhar Jul 20, 2016

Member

Sure.

  1. ecs-cli compose service up : command works with a Docker compose file to create a task definition associated with that file (each "service" of docker compose translates to a "container" in the task definition). Service up command also creates/updates an ECS service with that task definition. Does that make sense?
  2. no downtime: You can specify your own deployment configuration with the up command (help doc link). You can provide --deployment-min-healthy-percent option for specifying the lower limit of the number of running tasks that must remain running and healthy in a service during a deployment.
  3. roll back: As for v0.4.1 release, ecs-cli does not have an automatic roll back. However, you will just need to udpate the compose yml file with the previous image and execute ecs-cli compose service up and it will take care of rolling out the previous version.
  4. env variable: ecs-cli supports Docker compose variable substitution. So you can specify image option in your compose yml as image: "$NEW_IMAGE" and ecs-cli will interpolate it for you.

Does that answer all your questions?

Member

uttarasridhar commented Jul 20, 2016

Sure.

  1. ecs-cli compose service up : command works with a Docker compose file to create a task definition associated with that file (each "service" of docker compose translates to a "container" in the task definition). Service up command also creates/updates an ECS service with that task definition. Does that make sense?
  2. no downtime: You can specify your own deployment configuration with the up command (help doc link). You can provide --deployment-min-healthy-percent option for specifying the lower limit of the number of running tasks that must remain running and healthy in a service during a deployment.
  3. roll back: As for v0.4.1 release, ecs-cli does not have an automatic roll back. However, you will just need to udpate the compose yml file with the previous image and execute ecs-cli compose service up and it will take care of rolling out the previous version.
  4. env variable: ecs-cli supports Docker compose variable substitution. So you can specify image option in your compose yml as image: "$NEW_IMAGE" and ecs-cli will interpolate it for you.

Does that answer all your questions?

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 20, 2016

Thank you again! To be clear, I get no downtime even running just 1 task on 1 instance? Because in the past we have to have a two instance setup to achieve zero downtime on our current setup.

You talk a lot about the Docker compose file. I assume it maps to the "Service" I've created in the AWS ECS console. However where do I find it in the Web interface?!

I do hope to make notes on this personal blog of mine: http://dabase.com/blog/ECS_questions/

I also don't understand CPU/memory allocation, but we can save that for another conversation.

kaihendry commented Jul 20, 2016

Thank you again! To be clear, I get no downtime even running just 1 task on 1 instance? Because in the past we have to have a two instance setup to achieve zero downtime on our current setup.

You talk a lot about the Docker compose file. I assume it maps to the "Service" I've created in the AWS ECS console. However where do I find it in the Web interface?!

I do hope to make notes on this personal blog of mine: http://dabase.com/blog/ECS_questions/

I also don't understand CPU/memory allocation, but we can save that for another conversation.

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 20, 2016

Yes, so I've bothered to create the service from within the AWS console.

So now I'm left wondering how to get the docker-compose.yml out or am I expected to start again? Is there no mapping between ecs-cli and the Web console ?!??

~$ ecs-cli compose service up
ERRO[0000] Failed to find the compose file: docker-compose.yml
ERRO[0000] Unable to open ECS Compose Project            error=open docker-compose.yml: no such file or directory
FATA[0000] Unable to create and read ECS Compose Project  error=open docker-compose.yml: no such file or directory

kaihendry commented Jul 20, 2016

Yes, so I've bothered to create the service from within the AWS console.

So now I'm left wondering how to get the docker-compose.yml out or am I expected to start again? Is there no mapping between ecs-cli and the Web console ?!??

~$ ecs-cli compose service up
ERRO[0000] Failed to find the compose file: docker-compose.yml
ERRO[0000] Unable to open ECS Compose Project            error=open docker-compose.yml: no such file or directory
FATA[0000] Unable to create and read ECS Compose Project  error=open docker-compose.yml: no such file or directory
@uttarasridhar

This comment has been minimized.

Show comment
Hide comment
@uttarasridhar

uttarasridhar Jul 20, 2016

Member

no downtime even running just 1 task on 1 instance?: If you always want 1 task to be up and healthy, in your case --deployment-min-healthy-percent should be 100%, which means ECS Scheduler would try to run a 2nd task on the same instance before it kills the 1st one. If there are enough resources on the instance (enough cpu, memory, no host-port conflict), then it would succeed, else you might need a second instance.

Member

uttarasridhar commented Jul 20, 2016

no downtime even running just 1 task on 1 instance?: If you always want 1 task to be up and healthy, in your case --deployment-min-healthy-percent should be 100%, which means ECS Scheduler would try to run a 2nd task on the same instance before it kills the 1st one. If there are enough resources on the instance (enough cpu, memory, no host-port conflict), then it would succeed, else you might need a second instance.

@uttarasridhar

This comment has been minimized.

Show comment
Hide comment
@uttarasridhar

uttarasridhar Jul 20, 2016

Member

Unfortunately, amazon-ecs-cli as of v0.4.1, provides only a one-way transformation from compose.yml to task definition. There's a feature request to convert task definition to compose.yml: #20.

What ecs-cli provides you is a quick and easy way to create and update clusters (using ecs-cli up). Using docker-compose like commands and files, you can also create and manage multi-container applications using the cli. And any cluster, service, task that is created using the cli shows up in the web console.

But mapping an existing service (created using the console) to the cli doesn't exist today because of the aforementioned one-way transformation.

Member

uttarasridhar commented Jul 20, 2016

Unfortunately, amazon-ecs-cli as of v0.4.1, provides only a one-way transformation from compose.yml to task definition. There's a feature request to convert task definition to compose.yml: #20.

What ecs-cli provides you is a quick and easy way to create and update clusters (using ecs-cli up). Using docker-compose like commands and files, you can also create and manage multi-container applications using the cli. And any cluster, service, task that is created using the cli shows up in the web console.

But mapping an existing service (created using the console) to the cli doesn't exist today because of the aforementioned one-way transformation.

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 21, 2016

If there are enough resources on the instance (enough cpu, memory, no host-port conflict), then it would succeed, else you might need a second instance.

I don't quite understand. Tasks define a port and running two of the same tasks on one instance will always be a failure since the ports will always conflict. Or have I missed a way of a random or sequential port allocation trick? (similar to http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/)

Finally I still don't quite understand how ecs-cli compose service up can be triggered from "Jenkins, AWS Code pipelines, CodeShip" after a successful docker build & placement of the image on appropriate image service.

https://www.youtube.com/watch?v=Imeb-_g_CtU <--- btw made a video about ECS

kaihendry commented Jul 21, 2016

If there are enough resources on the instance (enough cpu, memory, no host-port conflict), then it would succeed, else you might need a second instance.

I don't quite understand. Tasks define a port and running two of the same tasks on one instance will always be a failure since the ports will always conflict. Or have I missed a way of a random or sequential port allocation trick? (similar to http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/)

Finally I still don't quite understand how ecs-cli compose service up can be triggered from "Jenkins, AWS Code pipelines, CodeShip" after a successful docker build & placement of the image on appropriate image service.

https://www.youtube.com/watch?v=Imeb-_g_CtU <--- btw made a video about ECS

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 26, 2016

Another query i have is how to keep the ECS optimized OS uptodate? Is the workflow supposed to be killing an instance and bringing up a new one? What's the tool flow for that to prevent down time please?

kaihendry commented Jul 26, 2016

Another query i have is how to keep the ECS optimized OS uptodate? Is the workflow supposed to be killing an instance and bringing up a new one? What's the tool flow for that to prevent down time please?

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 27, 2016

Earlier you said But at each stage, once you have an image, amazon-ecs-cli just requires you to execute ecs-cli compose service up to update your services.

I didn't find this was the case. The new image only came online with a {down,up}!
http://s.natalian.org/2016-07-27/ecs.txt

kaihendry commented Jul 27, 2016

Earlier you said But at each stage, once you have an image, amazon-ecs-cli just requires you to execute ecs-cli compose service up to update your services.

I didn't find this was the case. The new image only came online with a {down,up}!
http://s.natalian.org/2016-07-27/ecs.txt

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jul 27, 2016

RE roll back: As for v0.4.1 release, ecs-cli does not have an automatic roll back. However, you will just need to udpate the compose yml file with the previous image and execute ecs-cli compose service up and it will take care of rolling out the previous version.

How do you specify an older image please? For e.g. with sha256:a014554178cac80bd08b1daf36238f408a0396bc5189197c5175a153e6b446d3?

kaihendry commented Jul 27, 2016

RE roll back: As for v0.4.1 release, ecs-cli does not have an automatic roll back. However, you will just need to udpate the compose yml file with the previous image and execute ecs-cli compose service up and it will take care of rolling out the previous version.

How do you specify an older image please? For e.g. with sha256:a014554178cac80bd08b1daf36238f408a0396bc5189197c5175a153e6b446d3?

@NoumanSaleem

This comment has been minimized.

Show comment
Hide comment
@NoumanSaleem

NoumanSaleem Aug 20, 2016

@kaihendry

I don't quite understand. Tasks define a port and running two of the same tasks on one instance will always be a failure since the ports will always conflict.
This is correct. If you have an application which specifies the host port to bind, you will need a second instance to support no downtime. Run a second instance and set your desired container count to 2 and deployment configuration to 50%.

Finally I still don't quite understand how ecs-cli compose service up can be triggered from "Jenkins, AWS Code pipelines, CodeShip" after a successful docker build & placement of the image on appropriate image service.

Configure a jenkins job with two steps

  1. https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Build+and+Publish+plugin Configure a tag, such as ${BUILD_NUMBER} which will create a container tag for each build.
  2. Bash script which executes ecs-cli commands in the workspace. Obviously you will need to have the job configured to pull source code.

Also, you'll want to update the docker-compose.yml file in your source code to set image to an environment variable which you will export in the bash script as well.

//yml
my-container:
  image: ${image}
# jenkins shell step
export image="hub.docker.com/foo/bar:$BUILD_NUMBER"
ecs-cli  ...

How do you specify an older image please? For e.g. with
You are already one step there if you commit your docker-compose.yml file with image referencing an environment variable. You can get creative how you want to rollback. Maybe you want to only roll forward, even if that means reverting a commit. If that's the case, your existing jenkins job will work. If you want to roll back specifying a specific container tag, make a new jenkins job which accepts user input variables. It will function the same as the script above, you will instead pass the provded build variable as the tag.

Now, I know docker-compose supports env variable for the image property. if ecs-cli doesn't, you can use envsubst to replace the references before running ecs-compose

hope that helps

NoumanSaleem commented Aug 20, 2016

@kaihendry

I don't quite understand. Tasks define a port and running two of the same tasks on one instance will always be a failure since the ports will always conflict.
This is correct. If you have an application which specifies the host port to bind, you will need a second instance to support no downtime. Run a second instance and set your desired container count to 2 and deployment configuration to 50%.

Finally I still don't quite understand how ecs-cli compose service up can be triggered from "Jenkins, AWS Code pipelines, CodeShip" after a successful docker build & placement of the image on appropriate image service.

Configure a jenkins job with two steps

  1. https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Build+and+Publish+plugin Configure a tag, such as ${BUILD_NUMBER} which will create a container tag for each build.
  2. Bash script which executes ecs-cli commands in the workspace. Obviously you will need to have the job configured to pull source code.

Also, you'll want to update the docker-compose.yml file in your source code to set image to an environment variable which you will export in the bash script as well.

//yml
my-container:
  image: ${image}
# jenkins shell step
export image="hub.docker.com/foo/bar:$BUILD_NUMBER"
ecs-cli  ...

How do you specify an older image please? For e.g. with
You are already one step there if you commit your docker-compose.yml file with image referencing an environment variable. You can get creative how you want to rollback. Maybe you want to only roll forward, even if that means reverting a commit. If that's the case, your existing jenkins job will work. If you want to roll back specifying a specific container tag, make a new jenkins job which accepts user input variables. It will function the same as the script above, you will instead pass the provded build variable as the tag.

Now, I know docker-compose supports env variable for the image property. if ecs-cli doesn't, you can use envsubst to replace the references before running ecs-compose

hope that helps

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jun 27, 2017

Since there are no results for https://github.com/search?utf8=%E2%9C%93&q=%22ecs-cli+compose+service+up%22+filename%3A.travis.yml&type= I am starting to worry that continuous integration (CI) is just too difficult with ECS.

I did notice one CI script that does a down and then an up https://github.com/masschallenge/impact-api-deploy-demo/blob/b165c808329b746c05a968dac8874d3989aaf424/.travis.yml#L30 but that obviously misses the point of using a cluster (fronted with a load balancer) for a red/black aka blue/green deployment. I don't want to drop a request!

kaihendry commented Jun 27, 2017

Since there are no results for https://github.com/search?utf8=%E2%9C%93&q=%22ecs-cli+compose+service+up%22+filename%3A.travis.yml&type= I am starting to worry that continuous integration (CI) is just too difficult with ECS.

I did notice one CI script that does a down and then an up https://github.com/masschallenge/impact-api-deploy-demo/blob/b165c808329b746c05a968dac8874d3989aaf424/.travis.yml#L30 but that obviously misses the point of using a cluster (fronted with a load balancer) for a red/black aka blue/green deployment. I don't want to drop a request!

@yinshiua

This comment has been minimized.

Show comment
Hide comment
@yinshiua

yinshiua Jun 28, 2017

Contributor

@kaihendry Here is our recommended way of setting CI deployment. See link. We also have other container partners that provide softwares to help users setup CI/CD deployments

Today, in the ECS CLI, we support using an ELB and ALB in ecs cli compose service up and you can configure your deployment preferences to --deployment-min-healthy-percent 100 --deployment-max-healthy-percent 200 to avoid downtime.

Contributor

yinshiua commented Jun 28, 2017

@kaihendry Here is our recommended way of setting CI deployment. See link. We also have other container partners that provide softwares to help users setup CI/CD deployments

Today, in the ECS CLI, we support using an ELB and ALB in ecs cli compose service up and you can configure your deployment preferences to --deployment-min-healthy-percent 100 --deployment-max-healthy-percent 200 to avoid downtime.

@kaihendry

This comment has been minimized.

Show comment
Hide comment
@kaihendry

kaihendry Jun 29, 2017

I did look at https://github.com/awslabs/ecs-refarch-continuous-deployment but I found the Cloudformation templates far too unwieldly to manage at a startup. We need good simple generic Cluster setup and tear down examples for Travis/CircleCI, ideally CLI driven and not 100s of lines of CloudFormation template.

kaihendry commented Jun 29, 2017

I did look at https://github.com/awslabs/ecs-refarch-continuous-deployment but I found the Cloudformation templates far too unwieldly to manage at a startup. We need good simple generic Cluster setup and tear down examples for Travis/CircleCI, ideally CLI driven and not 100s of lines of CloudFormation template.

@codepj

This comment has been minimized.

Show comment
Hide comment
@codepj

codepj Oct 10, 2017

Specific to Jenkins Pipelines, Is there an image to install Jenkins workers on that will have ecs-cli installed on it, if not, should I install it myself, or should I spin up a container to execute ecs-cli
these commands?

codepj commented Oct 10, 2017

Specific to Jenkins Pipelines, Is there an image to install Jenkins workers on that will have ecs-cli installed on it, if not, should I install it myself, or should I spin up a container to execute ecs-cli
these commands?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment