Failing authentication for some larger instances.

[pdunnavant]

I'm having what appears to be a timing issue with some larger EC2 instance types (that's the pattern I'm seeing anyway). I have some cases where I need larger instances (e.g. m3.large). Most of the time, when kitchen attempts to build them, an authentication failure happens after the "(ssh ready)" message:

-----> Starting Kitchen (v1.2.1)
-----> Cleaning up any prior instances of <foo-centos-65>
-----> Destroying <foo-centos-65>...
       Finished destroying <foo-centos-65> (0m0.00s).
-----> Testing <foo-centos-65>
-----> Creating <foo-centos-65>...
       EC2 instance <i-56fe0c5d> created.
..............       (server ready)
       Waiting for ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com:22...
       Waiting for ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com:22...
       (ssh ready)
       Finished creating <foo-centos-65> (1m49.42s).
-----> Converging <foo-centos-65>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 3.1.1...
       Removing non-cookbook files before transfer
-----> Destroying <foo-centos-65>...
       EC2 instance <i-56fe0c5d> destroyed.
       Finished destroying <foo-centos-65> (0m1.09s).
>>>>>> Converge failed on instance <foo-centos-65>.
>>>>>> Please see .kitchen/logs/foo-centos-65.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Authentication failed for user root@ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com
>>>>>> ----------------------

This is happening fairly consistently for me (at least with m3.large instances). If I edit the ec2 driver to add a 15 second delay (note: 15 seconds is an arbitrary number; I didn't try anything lower) like the following, everything seems to work better. I haven't had a failure with the delay added.

def create(state)
  return if state[:server_id]
  server = create_server
  state[:server_id] = server.id

  info("EC2 instance <#{state[:server_id]}> created.")
  server.wait_for { print '.'; ready? }
  print '(server ready)'
  state[:hostname] = hostname(server)
  wait_for_sshd(state[:hostname], config[:username])
  sleep 15 # <---- Added this delay
  print '(ssh ready)\n'
  debug("ec2:create '#{state[:hostname]}'")
rescue Fog::Errors::Error, Excon::Errors::Error => ex
  raise ActionFailed, ex.message
end

I'm not quite sure what the problem is, but it just looks like AWS is returning a "running" state that isn't quite ssh-able. I'm going to try to dig in a little bit further, but I figured I'd open up an issue in case someone else knows exactly how to fix this faster than I can take a look at it. :-)

[JigarJoshi]

I am frequently running into it as well

[pdunnavant]

Good to hear that I'm not going completely crazy. I'll try to continue looking into this when I have a few minutes then. I don't really think an arbitrary sleep is a good idea, so I'd rather look into a better solution.

[pdunnavant]

I'm pretty sure that this is fixed on master in test-kitchen itself, so the fix should come out with the next stable release. I'll go ahead and leave this open just so it can be verified (unless you'd rather close it here). I did run a couple tests with test-kitchen on master and everything seemed to work fine.

[sethvargo]

You may need to raise your ssh_timeout, which is now present in master of kitchen-ec2 and will be available in the next release.