synchronize lookup and assignment of floating ip

[jgawor]

Test kitchen can be run in parallel mode (-p option). When the driver is configured with floating_ip_pool option, it might try to assign the same floating ip address to two or more instances (and that might cause for two different tests to run on the same instance at the same time).
The problem here is really a combination of issues. First is that the attach_ip_from_pool when executed at the same time can return the same free_addrs (and so that two threads will end up trying to associate the same address). Second, is that the server.associate_address address doesn't really return an error if the association fails. And, third, as far as I know the only way to verify the address assignment is to poll (e.g. by checking compute.addresses).

Since polling can be expensive and take some time or until somebody can suggest a better approach, for now I propose to synchronize the attach_ip_from_pool() method. That should at least ensure that the driver will not attempt to assign the same address twice. Obviously, this is not a foolproof solution as some other process might steal the address as the test kitchen runs but it makes the driver work a little better.

[hartmantis]

Ewww, that does sound ugly. The intra-process fix looks good; I'm just poking around in Fog a little, hoping to find a magical solution to the cross-process race condition. We hadn't hit it yet at $workplace, but surely will as all our cookbook tests are moved onto TK.

[hartmantis]

Looks like the associate_address limitations are how the OpenStack API works. It's doing an assignment and returning a 202 regardless of whether the IP was associated with another server beforehand or not. Even if polling happened here, it would still be possible to start up a dozen build processes and have them all fight over the same IP.

...unless there's something I missed, which I reeeaaally hope is the case, but going ahead and merging this.

[jgawor]

Thanks. I hope OpenStack will handle this better in the future.