fix #60239 Caught exception in wait_for_fun utils in OpenStack

[vdloo]

A PR that implements the fix from #60239

After upgrading to 3004.1 we noticed we could not create any nodes anymore on OpenStack. The fix proposed by @sblaisot in #60239 seems to fix the issue.

Looking at an earlier commit like 47ecb7a I don't think this code is thoroughly unit-tested (tests/unit/cloud/clouds/test_openstack.py maybe?)

[sblaisot]

Thanks fo proposing this PR which fixes a critical bug in salt-cloud openstack driver for years.

You probably also need to import salt.utils.cloud

[vdloo]

some debugging information from doing:

def _get_ips(node, addr_type="public"):
    ret = []
    for _, interface in node.addresses.items():
        for addr in interface:
            try:
                __utils__["cloud.is_public_ip"]
            except KeyError:
                import ipdb; ipdb.set_trace()
            if addr_type in ("floating", "fixed") and addr_type == addr.get(
                "OS-EXT-IPS:type"
            ):
                ret.append(addr["addr"])
            elif addr_type == "public" and __utils__["cloud.is_public_ip"](
                addr["addr"]
            ):
                ret.append(addr["addr"])
            elif addr_type == "private" and not __utils__["cloud.is_public_ip"](
                addr["addr"]
            ):
                ret.append(addr["addr"])
    return ret

looks like salt.utils.cloud is already available in this scope so import salt.utils.cloud is not needed:

ipdb> __utils__['cloud.is_public_ip']                                                          
*** KeyError: '__utils__'
ipdb> salt.utils.cloud.is_public_ip                                                            
<function is_public_ip at 0x7f219f07e680>
ipdb>

I don't know what populates this __utils__ variable but what's interesting is that not in all invocations of this function the 'cloud.is_public_ip' does not exist. An earlier invocation in the process shows:

py(444)_get_ips()
    443             import ipdb; ipdb.set_trace()
--> 444             if addr_type in ("floating", "fixed") and addr_type == addr.get(
    445                 "OS-EXT-IPS:type"

ipdb> __utils__                                                                                
<salt.loader.context.NamedLoaderContext object at 0x7f5afabe2e90>
ipdb> __utils__['cloud.is_public_ip']                                                          
<LoadedFunc name='cloud.is_public_ip'>
ipdb>                                                                                          

@aef mentions in #60239 that:

It seems __utils__ is unavailable as soon as __query_node(vm_) is run through cloud.wait_for_fun. Seems like it loses context in a threadpool or something like this.

the exact error happens here:

salt/cloud/clouds/openstack.py(452)_get_ips()
    451                 ret.append(addr["addr"])
--> 452             elif addr_type == "private" and not __utils__["cloud.is_public_ip"](
    453                 addr["addr"]

ipdb>                                                                                          
KeyError: '__utils__'

[waynew]

Got some tests started here:
https://github.com/waynew/salt/pull/new/tc/add-starter-tests-for-61918
There's a bit to do to finish them up, but that should be a good jumping off point 👍

[dwoz]

Closing this due to inactivity. Anyone should feel free to re-open it if they want to see it through to the end in one release cycle.