Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws ecs - how to update a task definition #406

Closed
extrawurst opened this issue Dec 29, 2017 · 50 comments
Closed

aws ecs - how to update a task definition #406

extrawurst opened this issue Dec 29, 2017 · 50 comments
Assignees
Labels
ecs feature-request New feature or request service-api This issue pertains to the AWS API

Comments

@extrawurst
Copy link

Is there a way to add a new revision to a ECS task definition?

In my case I want to update the container URL in my CD pipeline using the command line. Either the documentation is missing how to do that or it is only possible currently using the management console?

see aws/amazon-ecs-cli#91 (but i am not using docker compose)

@jamesls
Copy link
Member

jamesls commented Jan 2, 2018

Hmm, I'm not sure. In general, the best place for usage related questions would be on stack overflow, as mentioned in our README.

Looking over the docs though, it seems like just using register-task-definition would create a new revision. From https://docs.aws.amazon.com/cli/latest/reference/ecs/register-task-definition.html#output:

          revision -> (integer)
              The revision of the task in a particular family. The revision is
              a version number of a task definition in a family. When you reg-
              ister  a task definition for the first time, the revision is 1 ;
              each time you register a new revision of a  task  definition  in
              the  same  family,  the  revision  value always increases by one
              (even if you have deregistered previous revisions in  this  fam-
              ily).

I would give that a shot first.

@jamesls jamesls closed this as completed Jan 2, 2018
@extrawurst
Copy link
Author

thanks, sorry for the noise. I was expecting it to be a separate command consistent with others like update-service

@menocomp
Copy link

Hello @extrawurst I have the same issue. I was wondering if you found a solution.

@extrawurst
Copy link
Author

hi @menocomp see @jamesls answer. just use register-task-definition and it will create a new revision if the family already exists

@diehlaws diehlaws added the guidance Request for help or guidance label Jan 4, 2019
@shatil
Copy link

shatil commented Apr 4, 2019

register-task-definition isn't a satisfactory way to accomplish a mere bump of the revision number.

register-task-definition requires the Task Definition parameters or JSON as input. Procuring that from describe-task-definition adds an additional dictionary layer and fields that are invalid for use with register-task-definition.

After flattening the top-level dictionary:

$ aws ecs register-task-definition --family name --cli-input-json file:///tmp/task-definition.json

Parameter validation failed:
Unknown parameter in input: "taskDefinitionArn", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration
Unknown parameter in input: "revision", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration
Unknown parameter in input: "status", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration
Unknown parameter in input: "requiresAttributes", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration
Unknown parameter in input: "compatibilities", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration

@m-radzikowski
Copy link

I think this should be reopened as a feature request. Currently, CLI for ECS Task update, required for CI/CD, is not straight forward and requires additional tricks.

Flow most commonly found on the web is to call describe-task-definition, get container definition from it, change image parameter and push it back with register-task-definition --family $FAMILY --container-definitions $UPDATED_DEFINITION.

Using Fargate I came across a problem with this flow - task couldn't be updated with the same JSON, it required additional parameters, and I basically had to put all top-level values from describe-task JSON as register-task-definition parameters (like cpu, memory and so on).

So the better solution for me was to use JSON as full input for the task, but it has it own problems mentioned by @shatil. Finally I ended with script like this:

TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_FAMILY" --region "us-east-1")
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$NEW_IMAGE" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities)')
aws ecs register-task-definition --region "us-east-1" --cli-input-json "$NEW_TASK_DEFINTIION"

@oappicgi
Copy link

oappicgi commented Jul 23, 2019

Thanks @m-radzikowski I slightly modified your solution so it also works when using git hash as image tag instead of "latest". Needs some some environmental variables to work though

#!/bin/bash

#Script to get current task definition, and based on that add new ecr image address to old template and remove attributes that are not needed, then we send new task definition, get new revision number from output and update service
set -e
ECR_IMAGE="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${CODEBUILD_RESOLVED_SOURCE_VERSION}"
TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_FAMILY" --region "$AWS_DEFAULT_REGION")
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$ECR_IMAGE" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities)')
NEW_TASK_INFO=$(aws ecs register-task-definition --region "$AWS_DEFAULT_REGION" --cli-input-json "$NEW_TASK_DEFINTIION")
NEW_REVISION=$(echo $NEW_TASK_INFO | jq '.taskDefinition.revision')
aws ecs update-service --cluster ${ECS_CLUSTER} \
                       --service ${SERVICE_NAME} \
                       --task-definition ${TASK_FAMILY}:${NEW_REVISION}```

@ORESoftware
Copy link

ORESoftware commented Aug 13, 2019

What's the status on this? how can we create a task definition revision? I would think there would be something like:

aws ecs update-task-definition --task-definition "$task_fam" --revision "ecr/name:tag"

so yes I am looking for:

aws ecs update-task-definition

I checked using aws ecs help and there is no such utility:

Screen Shot 2019-08-13 at 10 31 13 AM

I created this ticket: aws/aws-cli#4401

@coolxeo
Copy link

coolxeo commented Mar 13, 2020

aws-cli support, if you don't like us to discuss just don't use github, sorry, this is still valid and should be reopened

@acro5piano
Copy link

acro5piano commented Mar 24, 2020

You might not need jq:

aws describe-task-definition --task-definition myapp \
     --query 'taskDefinition.{containerDefinitions:containerDefinitions}' \
     > /tmp/task-definitions.json 

Fargate users might need long commands like this:

aws ecs register-task-definition \
    --family myapp \
    --cli-input-json file:///tmp/task.json \
    --requires-compatibilities FARGATE \
    --network-mode awsvpc \
    --cpu 1024 --memory 2048 \
    --task-role-arn <arn> --execution-role-arn <arn>

@goelinsights
Copy link

goelinsights commented Apr 21, 2020

As a fargate user, I've found a hack using for ecs cli and --tags.

Whatever you put in to the version number, it'll set the revision as the next one.

ecs-cli compose --project-name PROJECT-NAME --file DOCKER_COMPOSE_FILE --ecs-params ECS_PARAMS_FILE --region REGION create --launch-type FARGATE --tags 'TaskDefinition=TASK:VERSION_NUMBER'```

@acro5piano
Copy link

I've switched to Fargate CLI. It automatically updates a suitable task definition.

https://github.com/awslabs/fargatecli

@goelinsights
Copy link

I've switched to Fargate CLI. It automatically updates a suitable task definition.

https://github.com/awslabs/fargatecli

Thanks for sharing that - hadn't seen it. Look forward to checking it out.

@kostyrev
Copy link

kostyrev commented Jun 3, 2020

fargatecli does not work if you have volumes defined

@sashokbg
Copy link

sashokbg commented Jun 4, 2020

You might not need jq:

aws describe-task-definition --task-definition myapp \
     --query 'taskDefinition.{containerDefinitions:containerDefinitions}' \
     > /tmp/task-definitions.json 

Fargate users might need long commands like this:

aws ecs register-task-definition \
    --family myapp \
    --cli-input-json file:///tmp/task.json \
    --requires-compatibilities FARGATE \
    --network-mode awsvpc \
    --cpu 1024 --memory 2048 \
    --task-role-arn <arn> --execution-role-arn <arn>

Here is a full version that does not require JQ:

NEW_TASK_DEFINTION=\$(aws ecs describe-task-definition --task-definition {task def} 
      --query '{  containerDefinitions: taskDefinition.containerDefinitions,
                  family: taskDefinition.family,
                  taskRoleArn: taskDefinition.taskRoleArn,
                  executionRoleArn: taskDefinition.executionRoleArn,
                  networkMode: taskDefinition.networkMode,
                  volumes: taskDefinition.volumes,
                  placementConstraints: taskDefinition.placementConstraints,
                  requiresCompatibilities: taskDefinition.requiresCompatibilities,
                  cpu: taskDefinition.cpu,
                  memory: taskDefinition.memory}')

aws ecs register-task-definition --cli-input-json "$NEW_TASK_DEFINTION"

@six0h
Copy link

six0h commented Aug 5, 2020

Adding to @sashokbg's answer, register-task-definition is complaining that it needs strings for the cpu and memory values. The --query command uses JMESPath syntax, which supports to_string($arg) type casting, so just wrap those two in to_string()

@oalagtash
Copy link

If you just want to update your ECS cluster to use the latest image (tag with "latest"), this should be enough:

aws ecs update-service --cluster your-cluster-name --service your-service-name --force-new-deployment

The --force-new-deployment param does the trick.

@acro5piano
Copy link

@oalagtash Thanks for the info. I'll try it.

@alto9
Copy link

alto9 commented Jan 1, 2021

What if I don't want to use the latest tag, and my task definitions contain secrets? I have figured out, thanks to this thread, how to generate a new task definition revision, but it always strips out my secrets. Is there a simple way to just create an exact replica of the task def but with updated image tag?

@acro5piano
Copy link

@alto9 For secret, you should store it to S3 (with proper policy) and get it when you start a container. It's more flexible.
Or you can store them as environment variables, but in that case anyone with console access can view these values

@RamiAwar
Copy link

fargatecli does not work if you have volumes defined

Or autoscaling

@RamiAwar
Copy link

RamiAwar commented Feb 17, 2021

@alto9 Have you looked into SSM parameter store? You can link those ARNs inside the fargate task definition.

@alto9
Copy link

alto9 commented Feb 17, 2021

@RamiAwar that's actually what I was doing. the problem is the secrets are literally stripped out of the task definition while it gets created and I don't get why.

@RamiAwar
Copy link

RamiAwar commented Feb 17, 2021

@alto9 That's weird. I just set it up for the first time and it works out of the box. I made 100% sure that the environment variables were read from SSM and passed to the container. One thing that you notice is that environment is different from secrets. Secrets is where I add the SSM params.

This is the task definition that I'm using, along with the commands I use to update it:

{ "family": "myproj", "networkMode": "awsvpc", "memory": "512", "cpu": "256", "executionRoleArn": "my_ex_role_arn", "taskRoleArn": "my_task_role_arn", "containerDefinitions": [ { "name": "myproj", "image": "myimg_arn", "essential": true, "portMappings": [ { "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/myproj", "awslogs-region": "us-west-1", "awslogs-stream-prefix": "ecs" } }, "environment": [ { "name": "SENTRY_ENV", "value": "DEVELOPMENT" } ], "secrets": [ { "name": "SENTRY_URL", "valueFrom": "arn_here" } ] } ], "requiresCompatibilities": [ "FARGATE" ] }

aws ecs register-task-definition --region us-west-1 --cli-input-json file://./task-def-dev.json

@pwillis-els
Copy link

pwillis-els commented Feb 23, 2021

If you are that bizarre person who wants to use a single one-liner to update the task definition, with no intermediate files, here you go.....

CLUSTER="my-cluster"
SERVICE="my-service-name"
aws ecs describe-services \
    --cluster "$CLUSTER" \
    --services "$SERVICE" \
    --query 'services[].taskDefinition[]' \
    --output text \
| xargs -n1 \
    aws ecs describe-task-definition \
        --query 'taskDefinition.taskDefinitionArn' \
        --output text \
        --task-definition \
| xargs -n1 \
    aws ecs describe-task-definition \
        --query '{
                containerDefinitions: taskDefinition.containerDefinitions, 
                family: taskDefinition.family, 
                taskRoleArn: taskDefinition.taskRoleArn, 
                executionRoleArn: taskDefinition.executionRoleArn, 
                networkMode: taskDefinition.networkMode, 
                volumes: taskDefinition.volumes, 
                placementConstraints: taskDefinition.placementConstraints, 
                requiresCompatibilities: taskDefinition.requiresCompatibilities, 
                cpu: taskDefinition.cpu, 
                memory: taskDefinition.memory
            }' \
        --task-definition \
| xargs -0 \
    aws ecs register-task-definition \
        --cli-input-json \
| cat

@acro5piano
Copy link

acro5piano commented Apr 10, 2021

Thanks to @sashokbg 's comment, my deploy script become like this:

.
|-- bin
|   `-- update-fargate-service.sh
`-- my-api
    `-- Dockerfile

bin/update-fargate-service.sh

#!/bin/bash

docker build -t $ECR_URI $SOURCE 
aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_URI
docker push $ECR_URI

# @see https://github.com/aws/aws-sdk/issues/406
NEW_TASK_DEFINTION=$(aws ecs describe-task-definition --task-definition $NAME \
      --query '{  containerDefinitions: taskDefinition.containerDefinitions,
                  family: taskDefinition.family,
                  taskRoleArn: taskDefinition.taskRoleArn,
                  executionRoleArn: taskDefinition.executionRoleArn,
                  networkMode: taskDefinition.networkMode,
                  volumes: taskDefinition.volumes,
                  placementConstraints: taskDefinition.placementConstraints,
                  requiresCompatibilities: taskDefinition.requiresCompatibilities,
                  cpu: taskDefinition.cpu,
                  memory: taskDefinition.memory}')

aws ecs register-task-definition  --family $NAME --cli-input-json "$NEW_TASK_DEFINTION"
aws ecs update-service --cluster $NAME --service $NAME --task-definition $NAME

And run

$ env \
	  ECR_URI=<account_id>.dkr.ecr.us-west-1.amazonaws.com/my-api \ # ECR URL
	  NAME=my-api \
	  SOURCE=my-api \
	  bin/update-fargate-service.sh

Hope this helps someone

@olivatooo
Copy link

image

@mdhornet90
Copy link

This is buried in the documentation for aws ecs update-service but confirms #406

If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service. The service scheduler uses the minimum healthy percent and maximum percent parameters (in the service's deployment configuration) to determine the deployment strategy.
NOTE:
If your updated Docker image uses the same tag as what is in the
existing task definition for your service (for example,
my_image:latest ), you do not need to create a new revision of your
task definition. You can update the service using the forceNewDe-
ployment option. The new tasks launched by the deployment pull the
current image/tag combination from your repository when they start.

@fourgates
Copy link

fourgates commented Feb 11, 2022

Are there any updates from the ECS team on these issues? Would it be possible to just expose an interface that would just create a new task def from the current task def and just update the image?

aws ecs update-service --cluster ${cluster} --service ${serviceName} --task-definition ${taskFamily}: --desired-count ${count} --new-image ${newImageName}

It seems like that functionality is already in place. When you click "Create new revision" it creates a new revision with a copy of the current. Just allow us to update the image and that would make so many of us happy!!

thoward-godaddy referenced this issue in thoward-godaddy/aws-cli Feb 12, 2022
* sam pipeline bootstrap (aws#2811)

* two-stages-pipeline plugin

* typos

* add docstring

* make mypy happy

* removing swap file

* delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility

* remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function

* sam pipeline bootstrap command

* move the pipelineconfig.toml file to .aws-sam

* UX - rewriting

Co-authored-by: Chris Rehn <crehn@outlook.com>

* UX improvements

* make black happy

* apply review comments

* UX - rewriting

Co-authored-by: Chris Rehn <crehn@outlook.com>

* refactor

* Apply review comments

* use python way of array elements assignments

* Update samcli/lib/pipeline/bootstrap/stage.py

Co-authored-by: _sam <3804518+aahung@users.noreply.github.com>

* apply review comments

* typo

* read using utf-8

* create and user a safe version of the save_config method

* apply review comments

* rename _get_command_name to _get_command_names

* don't save generated ARNs for now, will save during init

* Revert "don't save generated ARNs for now, will save during init"

This reverts commit d184e164022d9560131c62a826436edbc93da189.

* Notify the user to rotate periodically rotate the IAM credentials

* typo

* Use AES instead of KMS for S3 SSE

* rename Ecr to ECR and Iam to IAM

* Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images

Co-authored-by: Chris Rehn <crehn@outlook.com>
Co-authored-by: _sam <3804518+aahung@users.noreply.github.com>

* sam pipeline init command (aws#2831)

* sam pipeline init command

* apply review comments

* apply review comments

* display a message that we have successfully created the pipeline configuration file(s).

* doc typo

* Let 'sam pipeline init'  prefills pipeline's infrastructure resources… (aws#2894)

* Let 'sam pipeline init'  prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap'  results.

* save bootstrapped sateg region

* make black happy

* exclude non-dict keys from samconfig.get_env_names method.

* Rename the pipeline 'Stage' concept to 'Environment' (aws#2908)

* Rename the pipeline 'Stage' concept to 'Environment'

* typo

* Rename --environment-name argument to --environment

* Sam pipelines ux rename ecr repo to image repository (aws#2910)

* Rename ecr-repo to image-repository

* UT Fixes

* typo

* typo

* feat: Support creating pipeline files directly into . without hooks (aws#2911)

* feat: Support creating pipeline files directly into . without hooks

* Integration test for pipeline init and pipeline bootstrap (aws#2841)

* Expose Environment._get_stack_name for integ test to predict stack name

* Add integ test for pipeline bootstrap

* Add init integ test

* small UX improvements: (aws#2914)

* small UX improvements:
1. show a message when the user cancels a bootstrapping command.
2. Don't prompt for CI/CD provider or provider templates if there is only one choice.
3. Make PipelineFileAlreadyExistsError a UserError.
4. use the Colored class instead of fg='color' when prompting a colored message.
5. Fix a bug where we were not allowing empty response for not required questions.

* Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one

* Add docs for PipelineFileAlreadyExistsError

* make black happy

* Sam pipelines s3 security (aws#2975)

* Deny non https requests for the artifacts S3 bucket

* enable bucket serverside logging

* add integration tests for artifacts bucket SSL-only requests and access logging

* typo

* Ensure the ArtifactsLoggingBucket denies non ssl requests (aws#2976)

* Sam pipelines ux round 3 (aws#2979)

* rename customer facing message 'CI/CD provider' to 'CI/CD system'

* add a note about what 'Environment Name' is during the pipeline bootstrap guided context

* Apply suggestions from code review

typo

Co-authored-by: Chris Rehn <crehn@outlook.com>

Co-authored-by: Chris Rehn <crehn@outlook.com>

* let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (aws#2982)

* Adding AWS_ prefix to displayed out. (aws#2993)

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Add region to pipeline bootstrap interactive flow (aws#2997)

* Ask AWS region in bootstrap interactive flow

* Read default region from boto session first

* Fix a unit test

* Inform write to pipelineconfig.toml at the end of bootstrap (aws#3002)

* Print info about pipelineconfig.toml after resources are bootstrapped

* Update samcli/commands/pipeline/bootstrap/cli.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

Co-authored-by: Chris Rehn <crehn@outlook.com>

* List detected env names in pipeline init when prompt to input the env name (aws#3000)

* Allow question.question can be resolved using key path

* Pass the list of env names message (environment_names_message) into pipeline init interactive flow context

* Update samcli/commands/pipeline/init/interactive_init_flow.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Fix unit test (trigger pr builds)

* Fix integ test

* Fix integ test

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Adding account id to bootstrap message. (aws#2998)

* Adding account id to bootstrap message.

* adding docstring

* Addressing PR comments.

* Adding unit tests.

* Fixing unit tests.

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Cfn creds fix (aws#3014)

* Removing pipeline user creds from cfn output. This maintains same user exp.

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Ux bootstrap revamp 20210706 (aws#3021)

* Add intro paragraph to bootstrap

* Add switch account prompt

* Revamp stage definition prompt

* Revamp existing resources prompt

* Revamp security prompt

* Allow answers to be changed later

* Add exit message for bootstrap

* Add exit message for bootstrap (1)

* Add indentation to review values

* Add "Below is the summary of the answers:"

* Sweep pylint errors

* Update unit tests

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/guided_context.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update samcli/commands/pipeline/bootstrap/cli.py

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Update unit tests

* Add bold to other literals

Co-authored-by: Chris Rehn <crehn@outlook.com>

* Adding account condition for CFN execution role. (aws#3027)

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* pipeline UX revamp 20210707 (aws#3031)

* Allow running bootstrap inside pipeline init

* Select account credential source within bootstrap

* Add bootstrap decorations within pipeline init

* Removing ip range option from bootstrap. (aws#3036)

* Removing ip range option from bootstrap.

* Fixing unit test from UX PR.

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Fix toml file incorrect read/write in init --bootstrap (aws#3037)

* Temporarily removing account fix. (aws#3038)

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Rename environment to stage (aws#3040)

* Improve account source selection (aws#3042)

* Fixing various cosmetics UX issues with pipeline workflow. (aws#3046)

* Fixing credential to credentials

* Forcing text color to yellow.

* Adding new line after stage diagram.

* Adding extra line after checking bootstrap message.

* Renaming config -> configuration

* account source -> credential source

* Removing old message.

* Fixing indentation in list.

* Fixing bunch of indentation.

* fixing f string

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Auto skip questions if stage detected (aws#3045)

* Autofill question if default value is presented

* Allow to use index to select stage names (aws#3051)

* Updating message when bootstrap stages are missing. (aws#3058)

* Updating message when bootstrap stages are missing.

* Fixing indendation

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Fixing bootstrap integ tests. (aws#3061)

* Fixing bootstrap integ tests.

* Cleaning up some integ tests.

* Using environment variables when running integ test on CI.

* Using expression instead of full loop.

* Adding instruction to use default profile on local.

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Fix bootstrap test region (#3064)

* Fix bootstrap region in integ test

* Fix regions in non-interactive mode as well

* Add more pipeline init integ test (aws#3065)

* Fix existing pipeline init integ test

* Add more pipeline init integ tests

* Config file bug (aws#3066)

* Validating config file after bootstrap stack creation.

* Validating config file after bootstrap.

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

* Fix pipeline init integ test because of pipelineconfig file exists (aws#3067)

* Make stage name randomized to avoid race condition among multi canary runs (aws#3078)

* Load number of stages from pipeline template (aws#3059)

* Load number of stages from templates

* Rename variable and add debug log

* Add encoding to open()

* Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (aws#2950)

* pipeline init UX: Ask to confirm when file exists (aws#3079)

* Ask to confirm overriding if files already exist, or save to another directory

* Add doc links (aws#3087)

* Adding accidentally removed tests back. (aws#3088)

Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>

Co-authored-by: elbayaaa <72949274+elbayaaa@users.noreply.github.com>
Co-authored-by: Chris Rehn <crehn@outlook.com>
Co-authored-by: Ahmed Elbayaa <elbayaaa@amazon.com>
Co-authored-by: Tarun <c2tarun@users.noreply.github.com>
Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>
@fszymanski-blvd
Copy link

I recently ran into a similar, but more frustrating issue. Due to an inefficient build pipeline, our developers currently rollback to a previous task definition via the console. Since this is a number of manual steps, one of our engineers attempted to recreate the manual process via the CLI.

Despite the engineers having the permissions to perform the action via the console, running the command in the CLI produces a permission error that the role denies iam:PassRole. My main question here is why does this work fine in the console but not the CLI?

@fatso83
Copy link

fatso83 commented Jul 1, 2022

@fszymanski-blvd That is vaguely related, but definitely not something that is suitable for this issue. Just pay up the 29 USD for developer support and create a case. They will figure it out. Best 29$ I have ever spent.

@fszymanski-blvd
Copy link

@fatso83 We're following that path as well. Just wasn't sure if the console had a different capability than the CLI, which would be slightly more related to this issue.

@stefanfuchs
Copy link

stefanfuchs commented Jul 1, 2022 via email

@zyphlar
Copy link

zyphlar commented Jul 7, 2022

It's amazing to me how cumbersome it is to create a task definition considering that it's provided so readily in JSON all over the place. Here I have task definitions "backed up" locally but it's like pulling teeth to "restore" them.

@Heatmanofurioso
Copy link

How after so many years we still don't have functionality to do this natively?

@ryandiamond23
Copy link

Is there any update on this? Surely there has to be a simple command that could be used to duplicate the latest task definition and update specific inputs.

Currently I'm doing this by describing and then using jq to manipulate the JSON. Not ideal.

@5er9e1
Copy link

5er9e1 commented Oct 13, 2022

Same problem.
I have a lot of information inside of Task Definition that created outside by automated scripts. And now I need to collect all this information just to be able update image tag? It is ridiculous.

Tons of bash scripts needed just to be able run single container...

That is a first time when I clearly felt heavy inconvenience of ECS in comparison with Kubernetes even for small projects

By the way here is my workaround for FARGATE:

TD="arn:aws:ecs:ap-northeast-1:123456789:task-definition/name"
OLD_TAG="oldest"
NEW_TAG="latest"
ROLE_ARN="arn:aws:iam::123456789:role/role-name"
EXEC_ROLE_ARN="arn:aws:iam::123456789:role/exec-role-name"
FAMILY="famaily-name"
CLUSTER="cluster-name"

JSON="$(aws ecs describe-task-definition --task-definition "${TD}" | jq -r '.taskDefinition | {containerDefinitions: .containerDefinitions, cpu: (.cpu)|tonumber, memory: (.memory)|tonumber, compatibilities: .compatibilities, requiresCompatibilities: .requiresCompatibilities, ephemeralStorage: .ephemeralStorage, networkMode: .networkMode}' | sed "s/${OLD_TAG}/${NEW_TAG}/g")"
CPU=$(echo "$JSON" | jq -r '.cpu')
MEMORY=$(echo "$JSON" | jq -r '.memory')
DEF_JSON=$(echo "$JSON" | jq -r '.containerDefinitions')
STORAGE=$(echo "$JSON" | jq -r '.ephemeralStorage')
NET=$(echo "$JSON" | jq -r '.networkMode')
REQ=$(echo "$JSON" | jq -r '.requiresCompatibilities[]' | xargs)
aws ecs register-task-definition --task-role-arn "${ROLE_ARN}" --network-mode "${NET}" --requires-compatibilities ${REQ} --ephemeral-storage "${STORAGE}" --memory "${MEMORY}" --cpu "${CPU}" --family "${FAMILY}" --container-definitions "${DEF_JSON}" --execution-role-arn "${EXEC_ROLE_ARN}" --tags "[{\"key\": \"Name\", \"value\": \"${FAMILY}\"}, {\"key\": \"Comment\", \"value\": \"updated by Me\"}]"
aws ecs deregister-task-definition --task-definition "${TD}"
aws ecs update-service --cluster "${CLUSTER}" --service "${FAMILY}" --task-definition "${FAMILY}" --force-new-deployment

Freaking Frankenstein sewn piece by piece

In my lucky case ROLE_ARN and EXEC_ROLE_ARN are predefined, but it should be also taken by CLI in common cases

@acro5piano
Copy link

@5er9e1 I think you can use --cli-input-json

@tim-finnigan
Copy link

tim-finnigan commented Nov 14, 2022

We still haven't received an update from the ECS team regarding this feature request but I have once again reached out to them for feedback. You can also consider reaching out through AWS Support if this is something you'd like to escalate further. I'm going to transfer this to our cross-SDK repository as it relates to a service API rather than the CLI directly. I will leave a comment here when we hear back from the ECS team. (P88746615)

@tim-finnigan tim-finnigan transferred this issue from aws/aws-cli Nov 14, 2022
@tim-finnigan tim-finnigan self-assigned this Nov 14, 2022
@itskingori
Copy link

I have built a tool called ecs-toolkit that could potentially help with this situation. Using it is quite simple 🤞🏽; taking the below example definition of an Rails application with a database migration and a web-server:

---
version: v1
cluster: example
tasks:
  pre:
    - family: app-database-migrate
      count: 1
      containers:
        - rails
      launch_type: ec2
services:
  - name: app-web-server
    force: false
    containers:
      - rails

It is easy to deploy a new version of your application using the deploy command:

$ ecs-toolkit deploy --image-tag=49779134ca1dcef21f0b5123d3d5c2f4f47da650
INFO[0000] using config file: .ecs-toolkit.yml          
INFO[0000] reading .ecs-toolkit.yml config file         
INFO[0000] starting rollout of pre-deployment tasks      cluster=example
INFO[0001] building new task definition from app-database-migrate:24  cluster=example task=app-database-migrate
WARN[0001] skipping container image tag update, no changes  cluster=example container=rails task=app-database-migrate
WARN[0001] skipping registering new task definition, no changes  cluster=example task=app-database-migrate
INFO[0001] preparing running task parameters             cluster=example task=app-database-migrate
INFO[0001] no changes to previous task definition, using latest  cluster=example task=app-database-migrate
INFO[0001] running new task, desired count: 1            cluster=example task=app-database-migrate
INFO[0001] watching task [1] ... last status: pending, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0004] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0007] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0010] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] watching task [1] ... last status: stopped, desired status: stopped, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] successfully stopped task [1], reason: essential container in task exited  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] tasks ran to completion, desired count: 1     cluster=example task=app-database-migrate
INFO[0013] tasks report - total: 1, successful: 1, failed: 0  cluster=example
INFO[0013] completed rollout of pre-deployment tasks     cluster=example
INFO[0013] starting rollout to services                  cluster=example
INFO[0014] building new task definition from app-web-server:103  cluster=example service=app-web-server
WARN[0014] skipping container image tag update, no changes  cluster=example container=rails service=app-web-server
WARN[0014] skipping registering new task definition, no changes  cluster=example service=app-web-server
INFO[0014] no changes to previous task definition, using latest  cluster=example service=app-web-server
INFO[0015] updated service successfully                  cluster=example service=app-web-server
INFO[0015] watch service rollout progress                cluster=example service=app-web-server
INFO[0015] watching ... service: active, deployment: primary, rollout: 1/1 (0 pending)  cluster=example deployment-id=ecs-svc/6300252591410027253 service=app-web-server
INFO[0015] checking if service is stable                 cluster=example service=app-web-server
INFO[0016] service is stable                             cluster=example service=app-web-server
INFO[0016] services report - total: 1, successful: 1, failed: 0  cluster=example
INFO[0016] completed rollout to services                 cluster=example
WARN[0016] skipping rollout of post-deployment tasks, none found  cluster=example

Obviously, I've focused on my use-case but I have tried to leave room for other use-cases. Supported functionality:

  • Takes inputs as a config file which allows you to:
    • Define all the services and tasks that make up an application in a concise configuration file that's easy to grok at a glance i.e. .ecs-toolkit.yml, you could think of this as an application.yml file.
    • Define all the important configuration options for tasks and services that are most applicable for deployments.
  • As part of the deployment process, it:
    • Registers a new task definition with a new image cloning all the concurrent configuration from the old one to the new one.
    • Runs new tasks and update services with the new task definition.
    • Runs the tasks and services asynchronously for faster deployments.
    • Watches a service until it's stable or a task until it's stopped.
    • Provides extensive logging and sufficient reporting throughout the process to catch failures and monitor progress.
  • It can also:
    • Update the image of only select containers in the new task definition.
    • Run pre-deployment tasks before updating services e.g. database migrations, asset syncing.
    • Run post-deployment tasks after updating services e.g. cleanup.
    • Optionally skip pre-deployment and post-deployment tasks during deployment.
    • Perform redeploys using the same image with an option forcing a pull of the image.

@acro5piano
Copy link

@itskingori Your ecs-toolkit looks really promising! I'll try it for a future project.

@hamza-saqib
Copy link

i have Blue/Green deployment implemented
now i am getting this while updating service
An error occurred (InvalidParameterException) when calling the UpdateService operation: Cannot force a new deployment on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.

@acro5piano
Copy link

FYI: Copilot CLI looks a better replacement for the CLI

https://github.com/aws/copilot-cli

@jritsema
Copy link

For those who have looked at fargatecli, here is a fork that is more actively maintained that gives you a one-liner to deploy a new image. It also has several other useful commands for working with envvars, secrets, logs, ops, and also works with docker-compose.yml files.

fargate service deploy --image xyz

@susahin80
Copy link

Thanks @m-radzikowski I slightly modified your solution so it also works when using git hash as image tag instead of "latest". Needs some some environmental variables to work though

#!/bin/bash

#Script to get current task definition, and based on that add new ecr image address to old template and remove attributes that are not needed, then we send new task definition, get new revision number from output and update service
set -e
ECR_IMAGE="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${CODEBUILD_RESOLVED_SOURCE_VERSION}"
TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_FAMILY" --region "$AWS_DEFAULT_REGION")
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$ECR_IMAGE" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities)')
NEW_TASK_INFO=$(aws ecs register-task-definition --region "$AWS_DEFAULT_REGION" --cli-input-json "$NEW_TASK_DEFINTIION")
NEW_REVISION=$(echo $NEW_TASK_INFO | jq '.taskDefinition.revision')
aws ecs update-service --cluster ${ECS_CLUSTER} \
                       --service ${SERVICE_NAME} \
                       --task-definition ${TASK_FAMILY}:${NEW_REVISION}```

It seems the following attributes should also be removed.

del(.registeredAt) | del(.registeredBy)

@tim-finnigan
Copy link

We heard back from the ECS team that this is not something currently on their roadmap and they recommended using this public roadmap for tracking updates: https://github.com/aws/containers-roadmap

Looking through this issue, several workarounds have been noted. But if those do not suffice then I recommend letting the ECS team know in this new issue I created for them to track: aws/containers-roadmap#2058

@github-actions
Copy link

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ecs feature-request New feature or request service-api This issue pertains to the AWS API
Projects
None yet
Development

No branches or pull requests