version 3.1.0 does not properly wait for VM to be up before failing

[qubitrenegade]

Hello,

I've been using version 3.0.0 on one machine, and it works great. It takes a minute for the VM to come up, but that's easily dealt with by waiting for the vm to come up:

-----> Creating <default-centos-65>...
       OpenStack instance with ID of <411b5bb2-70d2-4947-ba0c-0d69393a58d2> is ready.
       Attaching floating IP from <EXT> pool
       Attaching floating IP <192.168.10.10>
       Waiting for server to be ready...
       Waiting for SSH service on 192.168.10.10:22, retrying in 3 seconds
       Waiting for SSH service on 192.168.10.10:22, retrying in 3 seconds
       Waiting for SSH service on 192.168.10.10:22, retrying in 3 seconds
       [SSH] Established
       Adding OpenStack hint for ohai

However, as of 3.1.0, this no longer appears to be the case, as it immediately fails on the first ssh attempt:

-----> Creating <default-centos-65>...
       OpenStack instance with ID of <e0956463-fe92-4f69-8c27-0b7815b976f4> is ready.
       Attaching floating IP from <EXT> pool
       Attaching floating IP <192.168.10.11>
       Waiting for server to be ready...
>>>>>> Server 192.168.10.11 (e0956463-fe92-4f69-8c27-0b7815b976f4) not reachable. Destroying server...
       OpenStack instance <e0956463-fe92-4f69-8c27-0b7815b976f4> destroyed.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #create action: [Broken pipe]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I know we can set server_wait, but I don't really want to wait 5mins every time when an extra 30s will do.

Is it possible to restore the functionality of waiting for SSH to be available for a certain period of time rather than just failing on the first attempt please?

Thanks!

[qubitrenegade]

Hmm... maybe I'm wrong, I downgraded to 3.0.0 and uninstalled 3.1.0 and that machine exhibits the same behavior of failing as on first ssh attempt... So I don't know what the difference is between the two workstatios as they are using identical kitchen configs.

[jjasghar]

Can you use nova or knife-openstack to bootstrap machines? It's very possible your image has a min to get sshkeys or other cloud-init stuff happening before it's up.

We have a "sleep" command for WinRM for this issue.

[qubitrenegade]

We're using OpenStack as an endpoint for Test Kitchen. I don't think knife-openstack works with Test Kitchen (at least nothing in the readme indicates how that would work), does it? And when you say nova do you mean creating the instances from the GUI? How would kitchen be able to manage that instance? Manually creating the images in the GUI then using kitchen-ssh kinda defeats the purpose as someone has to manually spin up and down the images at that point.

What I really don't understand is why I'm seeing different behavior on what I believe to be two identically configured workstations. On one workstation there's an interval to check if ssh is up yet, on the other workstation kitchen fails as soon as ssh fails once.

Ultimately, we're looking to increase velocity, so things like adding manual steps or long (5 mins is a long time if you're doing multiple kitchen runs an hour) wait times. I know most of our devs are more impatient than I.

[jjasghar]

The nova command and knife-openstack is to take out kitchen out of the problem. From the information you are saying the common denominator is your OpenStack cloud. And to answer your question:

Is it possible to restore the functionality of waiting for SSH to be available for a certain period of time rather than just failing on the first attempt please?

We've had a couple discussions about it. If you read through the issues and PRs you'll see that we/I ultimately decided not to do it. There are SO many different setups with consoles, enabled/disabled, different hypervisors with the way they offer that data, it would make this plugin unruly.

My ultimate goal is to make a plugin for kitchen that can support long term OpenStack, not something bespoke for someones specific cloud.

There is nothing wrong with you forking this and developing for your specify cloud, but it's very unlikely it'll be put upstream.

And unfortunately I can't access your cloud to verify anything. From your issues you've put in today it's something configured from your OpenStack team, not this plugin.

[jjasghar]

Not to mention, most of the time this issue pops up it's due to Glance caching on the different hypervisors around your cloud. That's why it's "fast" sometimes and "slow" other times.

I'd talk to your OpenStack team and look for a way to cache Glance images on your hypervisor and tweaking that option.

[jjasghar]

The Glance issue "fix/workaround" we have in this driver: https://github.com/test-kitchen/kitchen-openstack#glance_cache_wait_timeout

[qubitrenegade]

I don't understand. Every other kitchen driver waits for machine images to come up. vagrant, vsphere, docker, all do similar Waiting for SSH service on 192.168.10.10:22, retrying in 3 seconds. This driver used to wait for the image to come up.

It takes time for images to boot. If I perform the same kitchen converge with the same Image against vagrant/VBox or vSphere kitchen end points, the image takes the same time to boot, the difference is, vagrant and vSphere drivers don't fail on their first attempt to connect to the box.

I'm not asking for anything specific to my environment, I'm just requesting the OpenStack driver behave like the other kitchen drivers and perform more than one attempt to connect to the test image before failing.

Barring that, can you tell me which version removed the multiple attempt to ssh functionality and we can just run the old version of kitchen-openstack?

[jjasghar]

You'll need to roll back to probably 1.x or older. That'll remove the transport layer and you'll need older versions of test-kitchen too.

Ah so you want to have a "oh it failed, let me try another time or two before exiting 1" for SSH?

That should be taken care from upstream test-kitchen, it's possible that we are just using an older version in the gem file, I'll do some research.

[jjasghar]

Can you verify the test-kitchen gem you have in your environment? If you see we pin pretty old: https://github.com/test-kitchen/kitchen-openstack/blob/master/kitchen-openstack.gemspec#L24 when the newest is 1.9.x -> https://github.com/test-kitchen/test-kitchen/blob/master/lib/kitchen/version.rb#L20

I'm betting that's the problem you're seeing.

[simplyunix]

Hello there.

Did you find a resolution to this problem?