Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
AWS private instances intermittently fail to terminate agents when idle #397
Intermittently, EC2 test agents are failing to be be stopped - its not unusual on my private instance to have 4 or 5 instances that have not been stopped over the period of a week. These instances have been idle for long period of time (8 hours+).
So far, I have observed this in agents running on us-east-1 and eu-west-1.
Full history of this issue is here: http://www.webpagetest.org/forums/showthread.php?tid=13465.
I think I know what may be going on. The running instance detection code relies on the tags that identify the instances as WPT instances and which locations they are tied to. If the instances start but fail tagging for some reason it would be possible for them to become orphaned (though they really should still be connecting in unless they are completely dead).
I'll change the logic to also include any known AMI ID's and backfill the location information automatically which should catch all running instances even if they fail to get tagged.
Should have something ready later this afternoon.