Limit elastic IP association to pending, running instances

[rfletcher]

EC2 takes some time to clean up terminated instances. When replacing an instance, elastic IP association on the new instance often fails because the old instance is still hanging around in a "terminated" state.

This change limits instance lookup to pending or running instances for the purpose of elastic IP association.

[DavidS]

Wouldn't it be more sensible to disassociate the elastic IP from the instance when terminating, so it can be re-used quicker? Or am I totally on the wrong boat here?

[rfletcher]

This is a problem even if the terminated instance never had the elastic IP. The requirements are simply that you have two instances with the same name, one of which is in the "terminated" state. I've only seen that while replacing an instance (and AWS hasn't yet cleaned up the terminated instance), so I used that case as an example.

The general problem is that it never makes sense to associate an elastic IP with anything other than a pending or running instance, so the query should only consider those.

[DavidS]

Ah, so the problem is not that AWS rejects the configuration, but that puppet detects the elastic IP on the terminated instance and complains?

[rfletcher]

It tries to associate the IP with the terminated instance, and fails. I've been running with my patch so I haven't seen this error lately, but I can try and recreate it later today and paste the output. I'm not sure why I didn't do that when I filed this issue a few months back.

[rfletcher]

The most frustrating part was that it only failed some of the time. When there are multiple matching instances Puppet just grabs the first instance in the set. If that happened to be the 'running' instance, there was no issue. If it was the 'terminated' instance, it blew up.

[rfletcher]

Here's a pretty minimal case:

1. Launch an instance named 'sandbox' (with or without puppet).

2. Terminate it.

3. Launch another and leave it running (again, without or without puppet).

   • The terminated instance's ID must come first alphabetically. If it doesn't, terminate the running instance and try again until there's at least one terminated instance before the running instance.

4. Apply this manifest, replacing the IP and region with your values:

   ec2_elastic_ip { '1.2.3.4':
     ensure   => 'attached',
     instance => 'sandbox',
     region   => 'us-east-1',
   }

Instead of ignoring the terminated instance(s), provider/ec2_elastic_ip/v2.rb will try and associate the IP with the first instance, which is terminated. That produces an error like this (puppet --debug output shown here):

Debug: Prefetching v2 resources for ec2_elastic_ip
Info: Creating association for 1.2.3.4
Warning: Multiple instances found named sandbox, using i-28458888
Error: Could not set 'attached' on ensure: stopped waiting, encountered a failure state
  at 5:.../manifest.pp
Error: Could not set 'attached' on ensure: stopped waiting, encountered a failure state
  at 5:.../manifest.pp
Wrapped exception:
stopped waiting, encountered a failure state
Error: /Stage[main]/Ec2_elastic_ip[1.2.3.4]/ensure: change from detached to attached
  failed: Could not set 'attached' on ensure: stopped waiting, encountered a failure state
  at 5:.../manifest.pp

[CarmenAPuccio]

This is the same as my issue with autoscaling instances with the same name. Please see #164

An instance can terminate but still is not actually cleaned up for a bit in AWS. If it has the same name as the new autoscaled node, it will fail.