AWS EC2 instance never comes up - ASG reports 1 instance, but stuck in pending

[dhalperi]

This sure looks like an AWS bug not a BK bug.

We have new autoscaler-based deployment in us-west-1. Our ASG goes all the way down to 0 instances, up to 15. Usually things work great; in the last few days we've had VMs that don't come up for 20+ minutes.

When there's only 1 step queued for an instance, this can be frustrating - that blocked step hangs seemingly "forever". BK does not try to scale up, EC2 does not actually have the instance.

The group is big enough that new VMs are spun up in both AZs multiple times per day, which rules out basic config errors like one subnet not being networked correctly. It really seems like an AWS issue.

[dhalperi]

[dhalperi]

ASG: ~20 minutes after the attempt to upscale.

[dhalperi]

EC2 for that instance:

[dhalperi]

Manually terminating the instance works fine as a way to unblock.

[lox]

Thanks for the details @dhalperi, seems like a really frustrating issue. If it keeps happening we'll add a check for it in the scaling lambda.

[dbaggerman]

This sounds like it might be related to something we've been seeing.

We traced the issue back to cloud-init failing to bootstrap the instance, seemingly because of problems talking to the EC2 metadata endpoint.

The bootstrap failure doesn't appear to result in the instances getting flagged as unhealthy. On the other hand, it means the buildkite-agent service never gets started. Since the agent isn't started, the stop timeout and ExecStopPost shutdown trigger never occur. All this results in the problem described above where the instance is neither killed by the ASG, nor able to run BK jobs.

I'll raise a PR with a change that seems to have more or less solved this problem for us. It just adds an OnFailure to the cloud-final systemd unit to poweroff the instance. So it doesn't prevent the failures from occurring, but it allows the ASG to clean up and the replace the instances when the problem occurs.

[lox]

@dbaggerman in this instance the status was stuck at pending, despite the failure of the instance, is that what you are seeing?

[dbaggerman]

@lox, Yes that matches what we saw. The Instance State was pending with the Status Checks as Initializing, as can be seen in the screenshot above.