Ignoring the capacity of errored servers

[andy-trimble]

In the case where a server cannot be created (due to an AWS issue, for example), the non-existent server's capacity is still used to calculate total capacity. This seems like undesirable behavior.

[bradrydzewski]

Hi Andy, this is actually by design. The reason is that in an error state the server may still exist in AWS, and we do not want Drone to keep adding servers on loop and running up bills.

Hypothetical example at #16 (comment)

[andy-trimble]

Ah, that makes sense. I ran into an issue where the EC2 instance could not be created (due to my misconfiguring the security group), but the autoscaler would not allocate additional resources since it believed the capacity had been filled. I wonder if there is a way to detect when the instance has not been created and ignore the capacity in that instance.

[tboerger]

For some cases I wish the autoscaler would exclude the errored nodes, at least for some providers it would be possible to check if there is really some server living.

[bradrydzewski]

We have a process that tries to automatically remove errored instances:
https://github.com/drone/autoscaler/blob/master/engine/reaper.go

We should not ignore the error state, which feels very risky. Instead we should continue to improve the reaper routine to try to understand why an instance is errored and resolve the issue by repairing or removing the instance.

[tboerger]

We have a process that tries to automatically remove errored instances:
https://github.com/drone/autoscaler/blob/master/engine/reaper.go

Good to know that it's currently not working for Hetzner, got to fix that.