[ECS] [Proposal]: ECS Execute-Command proposal

[fierlion]

The ECS team is planning on implementing an execute-command functionality.

We would like comments on:
How the above approach will or will not fit your use case.
Your thoughts/concerns related to either the bind-mount or the use of SSM agent.

Containers Roadmap link: #187

Problem statement

Presently without workarounds the only way to access containers on ECS EC2 is with privilege and credentials to SSH onto the Container Instance, as well as user-level access to the Docker daemon to docker exec a container directly. On Fargate, such container level execute-command access is currently not supported. Sometimes, it’s difficult to get a full picture of an application from the CloudWatch logs alone. There are times when direct container access is necessary.

We’re planning on adding an ECS execute-command capability, which will make it possible to gain immediate access to containers running as part of ECS EC2 and Fargate tasks.

ECS Execute-Command Solution Proposal

In order to support execute-command functionality for ECS, we’ll use the SSM Agent and its session management capabilities.

We plan on bind-mounting an SSM Agent and its dependencies https://github.com/aws/amazon-ssm-agent into the container at container start. A customer’s execute-command session will be directly linked to their container. We’ll also mount the SSM Agent logs from inside the container to a unique directory on the EC2 host.

                             <-HOST->
                             
                             aws ecs run-task ------------------|
                                                                V
                                                           _____________
                                                           |           |
      ___customer_task___________    |---------------------| ECS Agent |
      | _______________________ |    |(1.mount)            |           |    
      | | customer container  | |    V                     -------------
      | | /exec-deps/    *<---|-|--/SSM Agent (Exec Agent)
      | |                *<---|-|--/session-worker & logger
      | |                *<---|-|--/certs
      | |                *<---|-|--/configuration
      | |                     | |
      | |/var/log/amazon/ssm/-|-|->/var/log/ecs/execAgent/<containerID>/
      | ----------------------- |
      ---------------------------

In this scenario, the ECS Agent takes on the same supervisor role over its owned containers that the init (systemd, upstart) process does on the host, watching over the SSM Agent and its dependent processes. This responsibility includes starting up the process using docker exec and ensuring that this startup is successful; watching over the process while it’s expected to be running, restarting on failure; stopping the process when required.

So what happens to my container?

If I enable this feature on my task/service, all containers launched in that task/service will have the SSM Agent dependencies directory /exec-deps (subject to change) mounted at startup, as well as the log directory /var/log/ecs/execAgent/<containerID> (subject to change) mount added to the host.

An ECS execute-command call from my laptop would establish a session-manager session directly inside my container.

[tjhiggins]

Great work to everyone involved! Excited to see that it is coming soon. I saw the agent version is live, but is there a rough ETA on when it will be available in the cli/api? Trying to plan a couple product features that would be impacted by this being an option.

[mreferre]

Hey @tjhiggins thanks for the interest. Unfortunately we don't share publicly more precise hints re dates than "coming soon". This is also to protect you from taking a dependency on a roadmap item that may slip last minute.

[jtatum]

As of this exact moment, the documentation links don't work yet but this just showed up https://aws.amazon.com/about-aws/whats-new/2021/03/amazon-ecs-now-allows-you-to-execute-commands-in-a-container-running-on-amazon-ec2-or-aws-fargate/
Edit: Looks like someone jumped the gun on this, the post got pulled. Here's hoping it shows up again soon.
Edit 2: It's back, with working links!

[dennisroche]

The documentation links are now working and impressively comprehensive.

However, unfortunately, the latest aws cli (2.1.29 published) doesn't have the aws ecs execute-command command. It appears that it is in the v1 version 84ff89523079ee5b826a192d7024f6a9a4f9c46b.

UPDATE: Just installed the https://github.com/aws/aws-cli/tree/v2#cli-dev-version and the command was there 🎉.

[SaloniSonpal]

Like @jtatum said, we launched ECS Exec on 3/15 based on the above design proposal. The execute-command CLI and ExecuteCommand API allow you run interactive shell or single commands in an ECS container running on Fargate or EC2.

[SaloniSonpal]

@dennisroche - ECS Exec support is available in 2.1.31 that just shipped
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

api-change:ecs: This is for ecs exec feature release which includes two new APIs - execute-command and update-cluster and an AWS CLI customization for execute-command API

[husain-aljamri]

While the commnds are there in aws cli, I couldn't get it to work because IAM keeps telling me that the action ecs:ExecuteCommand doesn't exist. When I add that action to a policy anyways, I keep getting an error about not having the permission to perform that action.

[jmteachw]

While the commnds are there in aws cli, I couldn't get it to work because IAM keeps telling me that the action ecs:ExecuteCommand doesn't exist. When I add that action to a policy anyways, I keep getting an error about not having the permission to perform that action.

I experienced the same thing, for whatever reason the policy builder in AWS console doesn't think the command exists, but if you add as JSON it will stick. You also need to make sure you allow action ecs:ExecuteCommand for the cluster and task. Here's an example:

{
  "Effect": "Allow",
   "Action": "ecs:ExecuteCommand",
   "Resource": "arn:aws:ecs:us-east-1:{MY_ACCOUNT}:task/{MY_CLUSTER}/*"
},
{
  "Effect": "Allow",
  "Action": "ecs:ExecuteCommand",
  "Resource": "arn:aws:ecs:us-east-1:{MY_ACCOUNT}:cluster/{MY_CLUSTER}"
}

[kylemacfarlane]

Does anyone else have a problem with tasks launched through CodePipeline deployments not having Execute Command enabled despite it being enabled on the service? I have to scale up and back down to replace the tasks with ones I can login to which is no good if I want to debug something already running.

The permission for both the cluster and the tasks gave me problems too. The blog post says to do one and the manual says to do the other, neither says to do both.

Using the amazon-ecs-exec-checker script I also find the agent often fails to launch at boot but does eventually start itself 20-30 mins later. Maybe something to do with maxing out the CPU at boot time.

[husain-aljamri]

@jmteachw Thanks. I reached out to AWS support, and they told me the same thing. Something I had to figure out though is that we also need the ecs:DescribeTasks action allowed for the same cluster. So your example should instead look like:

  "Effect": "Allow",
   "Action": "ecs:ExecuteCommand",
   "Resource": "arn:aws:ecs:us-east-1:{MY_ACCOUNT}:task/{MY_CLUSTER}/*"
},
{
  "Effect": "Allow",
  "Action": [
                    "ecs:ExecuteCommand",
                    "ecs:DescribeTasks"
                ]
  "Resource": "arn:aws:ecs:us-east-1:{MY_ACCOUNT}:cluster/{MY_CLUSTER}"
}

[MrChrisRodriguez]

Was this intended to be compatible with the docker compose using an ecs context? I can't seem to find any documentation for making these work together.

[JohnPreston]

Was this intended to be compatible with the docker compose using an ecs context? I can't seem to find any documentation for making these work together.

Do you mean in the context of using the ecs plugin in docker to use docker exec ?

[mreferre]

@MrChrisRodriguez the request to support ecs exec in the context of the docker compose using the ecs context (no pun intended) is tracked here: docker-archive/compose-cli#670

I am wondering if there is a way, using x-aws-cloudformation extensions to plug the proper CFN entries required (see here for an example of how these extensions work).