-
Notifications
You must be signed in to change notification settings - Fork 605
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ECS hosts lose ability to start new tasks (inactivity time exceeded timeout / No such container in logs) #547
Comments
@jhovell The error "Error retrieving stats for container xxxx: inactivity time exceeded timeout" is caused by slow docker response(as docker ps takes 10~20s). And the error "No such container" could occur once for each container when agent received docker stop events but before stopping collecting the metrics. These errors are not the reason for the ELB health check fail. When docker became slow, it will take longer to finish the deployment, as you said this problem occurred when you try to deploy a new version of service. So the ELB health check would fail as the service is not ready. Currently we don't support this kind of detection. One way you can do now is to configure your ELB to wait longer to health check for this situation. Thanks |
@richardpen what would/could be the cause of the slowness? Surely it is not necessarily an ECS agent issue; it could be an underlying Docker issue or a issue with my app in general. Hard for me to debug since I don't necessarily know all the internals of the ECS Optimized AMI. Normally an app might take 20-30 seconds to boot, and the ELB will not fail a task for at least 2 minutes. Increasing the timeout on the ELB to get around this transient issue doesn't seem like a very good one, right? I'm guessing there is some underlying problem with the agent/docker/ECS... not sure where to start diagnosing. |
After examining this in a bit more detail I think this is actually a duplicate of #537 It looks like due to a ECS service that gets stuck in a loop restarting, our 66 IOPS/sec budget on our 22GB GP2 volumes is exceeded by all the container restarts and our burst balance is drained to zero. Next, when eventually another service or services is updated to a newer image, due to not being able to burst in EBS the container does not start in time for the ELB health check to pass, and these services get stuck in a restart loop as well, keeping the EBS credit at ~0 and causing degradation across any service on the host requiring more than very light disk access. In my case I will likely address this by adding Cloudwatch alerts to monitor if EBS read/write IOPS budgets are exceeded for a lengthy period of time such as an hour, and trigger other downstream actions to deal with the issue. Adding this here in case it helps anyone else as I would imagine this is a common failure mode for anyone using the ECS Optimized AMI that accidentally deploys a task definition that cannot or does not pass ELB healthchecks. EBS docs on IO credit balances: Forum issue on similar topic: |
Thanks @jhovell for root-causing this. Yes you are right that this is one of the common failure mode we are observing in ECS. If this becomes a problem we recommend moving to a SSD with provisioned IOPS. We will improve our documentation around this so that it is much clear. Thanks again. |
@jhovell Do you know what triggers the restart loop in the first place? We have been experiencing this same thing out of no where and I can't find the root cause of these events happening. It's very strange because we have basically 3 clusters that are identical (Prod, Staging, Dev). All clusters have 3 hosts running t2.larges. We had 20 tasks running on Prod and 12 on Staging and Dev. Staging and Dev started failing across the board yesterday seemingly out of no where. The only way to fix it was to terminate all hosts and have new ones be created. I initially thought that the hosts were over committed on memory, but that wasn't the case. I looked at what you had said after some googling and you were correct in our case. The burst credits were at 0 and the server was bricked. Agent wasn't connected, couldn't ssh, etc. This just seems like a big problem that happens out of no where and I would prefer to find a solution that prevents this instead of creating alarms that handle it after the fact. Is PIOPs the only recommended way of fixing this? |
As an update to my situation, Switching to PIOPs fixed our problems. I doubled what we were provisioned with a regular gp2 volume (so essentially 6 IOPs per GB on the new volume). This seems like the simplest and most cost effective way to resolve the issue. We only switched to PIOPs for the docker volume when using the ECS Optimized AMI. The root partitions are still gp2. I think the cause of our issue was a mix of high IO docker applications that caused restart loops to start happening and it just spiraled out of control from there. Still monitoring, but it seems like PIOPs is a quick and easy solution. |
Version: Amazon ECS-Optimized Amazon Linux AMI 2016.09.i
Periodically hosts in our cluster become unable to launch new tasks. SSH'ing to the instance show that:
Simple commands like
docker ps
will hang for 10-20 seconds but then return.A command like
curl localhost:8888/myService
which would normally return 200/OK givescurl: (56) Recv failure: Connection reset by peer
. Not surprisingly ELB healthchecks fail every few minutes and the ECS agent dutifully restarts each task. It's worth noting existing containers continue to run and operate healthily (we normally run into this issue when trying to deploy a new version of a service).ECS Agent log has no obvious errors though periodically this is seen:
Docker logs contain
also occasionally:
This is happening about once per day across roughly ~20 ECS hosts & is difficult to detect / automate restarting
I can't tell if this is a duplicate of an existing issue though almost all issues here seem to say they are resolved in 1.12.2 version of the agent which we are running, so maybe this issue still persists? Happy to send entire docker log files, ECS agent files.
The text was updated successfully, but these errors were encountered: