Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Salt cloud vultr integration is broken, and some of the cloud utilities function make it difficult to debug #60568

Open
fake-name opened this issue Jul 19, 2021 · 5 comments
Open

Salt cloud vultr integration is broken, and some of the cloud utilities function make it difficult to debug #60568

fake-name opened this issue Jul 19, 2021 · 5 comments
Assignees
@garethgreenaway
Labels
Bug broken, incorrect, or confusing behavior

Comments

@fake-name
Copy link
Contributor

fake-name commented Jul 19, 2021 •
edited

Description
A clear and concise description of what the bug is.

Setup
https://github.com/fake-name/AutoTriever/blob/master/marshaller/salt_runner.py

Steps to Reproduce the behavior
Attempt to deploy a Vultr VM.
Deploy fails.

Expected behavior
I expect to actually create the VM.

Screenshots

Main.VpsHerder.MainThread - INFO - Creating instance...
Main.VpsHerder.MainThread - INFO -      Client name: 'test-1'
Main.VpsHerder.MainThread - INFO -      using provider: 'vultr'
Main.VpsHerder.MainThread - INFO -      kwargs: '{'image': 270, 'private_networking': False, 'size': '201', 'location': 22, 'script': '/home/client/AutoTriever/marshaller/bootstrap-salt-delay.sh'}'
salt.utils.entrypoints.MainThread - DEBUG - Using pkg_resources to load entry points
salt.utils.entrypoints.MainThread - DEBUG - Using pkg_resources to load entry points
salt.utils.lazy.MainThread - DEBUG - Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
salt.utils.lazy.MainThread - DEBUG - LazyLoaded parallels.avail_locations
salt.utils.lazy.MainThread - DEBUG - LazyLoaded proxmox.avail_sizes
salt.cloud.MainThread - DEBUG - Generating minion keys for 'test-1'
salt.utils.lazy.MainThread - DEBUG - LazyLoaded cloud.fire_event
salt.utils.event.MainThread - DEBUG - MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
salt.utils.event.MainThread - DEBUG - MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
salt.utils.event.MainThread - DEBUG - Sending event: tag = salt/cloud/test-1/creating; data = {'provider': 'vultr:vultr', 'driver': 'vultr:vultr', 'name': 'test-1', 'profile': None, 'event': 'starting create', '_stamp': '2021-07-19T07:00:32.016509'}
salt.transport.ipc.MainThread - DEBUG - Closing IPCMessageClient instance
salt.utils.lazy.MainThread - DEBUG - LazyLoaded http.query
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/regions/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/os/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/plans/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
salt.loaded.int.clouds.vultrpy.MainThread - INFO - Creating Cloud VM test-1
salt.utils.event.MainThread - DEBUG - MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
salt.utils.event.MainThread - DEBUG - MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
salt.utils.event.MainThread - DEBUG - Sending event: tag = salt/cloud/test-1/requesting; data = {'kwargs': {'hostname': 'test-1', 'VPSPLANID': '201', 'enable_private_network': 'no', 'DCID': '22', 'OSID': 270, 'label': 'test-1'}, 'event': 'requesting instance', '_stamp': '2021-07-19T07:00:33.074577'}
salt.transport.ipc.MainThread - DEBUG - Closing IPCMessageClient instance
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/server/create?api_key=XXXXXXXXXX using POST method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
salt.loaded.int.utils.cloud.MainThread - DEBUG - Attempting function <function create.<locals>.wait_for_hostname at 0x7f998b660e50>
salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 497, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 315, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = list_nodes_full()
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 246, in list_nodes_full
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = _query("server/list")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 617, in _query
salt.loaded.int.utils.cloud.MainThread - DEBUG -         result = __utils__["http.query"](
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 78, in __getitem__
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return self.value()[item]
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 72, in value
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return loader.pack[self.name]
salt.loaded.int.utils.cloud.MainThread - DEBUG -     KeyError: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -
salt.loaded.int.utils.cloud.MainThread - DEBUG - Retrying function <function create.<locals>.wait_for_hostname at 0x7f998b660e50> on  (try 1)
salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 497, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 315, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = list_nodes_full()
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 246, in list_nodes_full
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = _query("server/list")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 617, in _query
salt.loaded.int.utils.cloud.MainThread - DEBUG -         result = __utils__["http.query"](
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 78, in __getitem__
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return self.value()[item]
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 72, in value
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return loader.pack[self.name]
salt.loaded.int.utils.cloud.MainThread - DEBUG -     KeyError: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -

<Repeat 900 (!) times>

Note that the above is with a patched wait_for_fun() implementation to actually output a useful traceback:

def wait_for_fun(fun, timeout=900, **kwargs):
    """
    Wait until a function finishes, or times out
    """
    start = time.time()
    log.debug("Attempting function %s", fun)
    trycount = 0
    while True:
        trycount += 1
        try:
            response = fun(**kwargs)
            if not isinstance(response, bool):
                return response
        except Exception as exc:  # pylint: disable=broad-except
            log.debug("Caught exception in wait_for_fun: %s", exc)
            for line in traceback.format_exc().split("\n"):
                log.debug("    %s", line)
            time.sleep(1)
            log.debug("Retrying function %s on  (try %s)", fun, trycount)
        if time.time() - start > timeout:
            log.error("Function timed out: %s", timeout)
            return False

I'm particulary confused by this exception, since vultrpy._query() is called successfully earlier in the deploy process. I suspect that something in the capture of wait_for_hostname() in wait_for_server_state() is breaking the reference to util somehow.

Versions Report

client@salt:~/AutoTriever/marshaller$ salt --versions-report
Salt Version:
          Salt: 3003.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Jun  2 2021, 10:49:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-77-generic
        system: Linux
       version: Ubuntu 20.04 focal

This is basically the current version from https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest
@fake-name fake-name added Bug broken, incorrect, or confusing behavior needs-triage labels Jul 19, 2021
@fake-name
Copy link
Contributor Author

fake-name commented Jul 19, 2021 •
edited

Instrumenting loader_context():

Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.utils.event.MainThread - DEBUG - MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
salt.utils.event.MainThread - DEBUG - MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
salt.utils.event.MainThread - DEBUG - Sending event: tag = salt/cloud/test-1/creating; data = {'name': 'test-1', 'provider': 'vultr:vultr', 'profile': None, 'driver': 'vultr:vultr', 'event': 'starting create', '_stamp': '2021-07-19T07:21:10.760720'}
salt.transport.ipc.MainThread - DEBUG - Closing IPCMessageClient instance
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.utils.lazy.MainThread - DEBUG - LazyLoaded http.query
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/regions/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/os/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/plans/list?api_key=XXXXXXXXXX using GET method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
salt.loaded.int.clouds.vultrpy.MainThread - INFO - Creating Cloud VM test-1
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.utils.event.MainThread - DEBUG - MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
salt.utils.event.MainThread - DEBUG - MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
salt.utils.event.MainThread - DEBUG - Sending event: tag = salt/cloud/test-1/requesting; data = {'kwargs': {'DCID': '2', 'hostname': 'test-1', 'OSID': 270, 'VPSPLANID': '201', 'enable_private_network': 'no', 'label': 'test-1'}, 'event': 'requesting instance', '_stamp': '2021-07-19T07:21:11.772218'}
salt.transport.ipc.MainThread - DEBUG - Closing IPCMessageClient instance
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.loaded.int.utils.http.MainThread - DEBUG - Requesting URL https://api.vultr.com/v1/server/create?api_key=XXXXXXXXXX using POST method
salt.loaded.int.utils.http.MainThread - DEBUG - Using backend: tornado
salt.loaded.int.utils.http.MainThread - DEBUG - Response Status Code: 200
Loader name: __utils__
loader.pack {'__utils__': <salt.loader.LazyLoader object at 0x7f0e03077790>, '__active_provider_name__': {}, '__grains__': {}, '__pillar__': {}, '__context__': {}}
salt.loaded.int.utils.cloud.MainThread - DEBUG - Attempting function <function create.<locals>.wait_for_hostname at 0x7f0e02f4b940>
Loader name: __utils__
loader.pack {'__context__': {}, '__proxy__': <function proxy at 0x7f0e04405670>, '__grains__': {}, '__pillar__': {}}
salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 497, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 315, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = list_nodes_full()
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 246, in list_nodes_full
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = _query("server/list")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 618, in _query
salt.loaded.int.utils.cloud.MainThread - DEBUG -         result = __utils__["http.query"](
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 81, in __getitem__
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return self.value()[item]
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 75, in value
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return loader.pack[self.name]
salt.loaded.int.utils.cloud.MainThread - DEBUG -     KeyError: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -
salt.loaded.int.utils.cloud.MainThread - DEBUG - Retrying function <function create.<locals>.wait_for_hostname at 0x7f0e02f4b940> on  (try 1)
Loader name: __utils__
loader.pack {'__context__': {}, '__proxy__': <function proxy at 0x7f0e04405670>, '__grains__': {}, '__pillar__': {}}
salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 497, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 315, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = list_nodes_full()
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 246, in list_nodes_full
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = _query("server/list")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 618, in _query
salt.loaded.int.utils.cloud.MainThread - DEBUG -         result = __utils__["http.query"](
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 81, in __getitem__
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return self.value()[item]
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 75, in value
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return loader.pack[self.name]
salt.loaded.int.utils.cloud.MainThread - DEBUG -     KeyError: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -
salt.loaded.int.utils.cloud.MainThread - DEBUG - Retrying function <function create.<locals>.wait_for_hostname at 0x7f0e02f4b940> on  (try 2)
Loader name: __utils__
loader.pack {'__context__': {}, '__proxy__': <function proxy at 0x7f0e04405670>, '__grains__': {}, '__pillar__': {}}
salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: '__utils__'
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 497, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 315, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = list_nodes_full()
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 246, in list_nodes_full
salt.loaded.int.utils.cloud.MainThread - DEBUG -         nodes = _query("server/list")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 618, in _query
salt.loaded.int.utils.cloud.MainThread - DEBUG -         result = __utils__["http.query"](
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 81, in __getitem__
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return self.value()[item]
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/loader_context.py", line 75, in value
salt.loaded.int.utils.cloud.MainThread - DEBUG -         return loader.pack[self.name]
salt.loaded.int.utils.cloud.MainThread - DEBUG -     KeyError: '__utils__'

So somehow, __utils__ is missing from the relevant test function loader context.

Similarly, going through and making the imports that are pointed at by __utils__ explicit, it then explodes on a __opts__ access:

salt.loaded.int.utils.cloud.MainThread - DEBUG - Caught exception in wait_for_fun: name '__opts__' is not defined
salt.loaded.int.utils.cloud.MainThread - DEBUG -     Traceback (most recent call last):
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 682, in wait_for_fun
salt.loaded.int.utils.cloud.MainThread - DEBUG -         response = fun(**kwargs)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 503, in wait_for_hostname
salt.loaded.int.utils.cloud.MainThread - DEBUG -         data = show_instance(vm_["name"], call="action")
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/cloud/clouds/vultrpy.py", line 325, in show_instance
salt.loaded.int.utils.cloud.MainThread - DEBUG -         salt.utils.cloud.cache_node(nodes[name], _get_active_provider_name(), __opts__)
salt.loaded.int.utils.cloud.MainThread - DEBUG -       File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 3073, in cache_node
salt.loaded.int.utils.cloud.MainThread - DEBUG -         __opts__.update(opts)
salt.loaded.int.utils.cloud.MainThread - DEBUG -     NameError: name '__opts__' is not defined
salt.loaded.int.utils.cloud.MainThread - DEBUG -
salt.loaded.int.utils.cloud.MainThread - DEBUG - Retrying function <function create.<locals>.wait_for_hostname at 0x7fccc7fbbc10> on  (try 5)

So all I can conclude is that something in some of the magic import stuff got changed, and no one tests the Vultr integrations at all.

@fake-name
Copy link
Contributor Author

fake-name commented Jul 19, 2021 •
edited

Possibly related to #58934 ? I don't think this is related.

@yun14u
Copy link

yun14u commented Aug 21, 2021 •
edited

I encounter the similar issue. One interesting observation is that the server instance is created on the Vultr side and I can log on to that server instance using root password created by Vultr.

$ salt --versions-report
Salt Version:
Salt: 3003.2

@yun14u
Copy link

yun14u commented Aug 25, 2021

I'm not sure if my comment is relevant; however, Vultr is no longer supporting V1 API (v1). The entire vultrpy.py driver needs to be revamped for V2 API (v2) which requires a lot of code rewrite.

@gizahNL
Copy link
Contributor

gizahNL commented Feb 16, 2022

Can confirm this is hitting us as well. And agree with the diagnosis that it doesn't seem to be an API issue.

wait_for_hostname ends up calling list_nodes_full last, which is also called by calling "salt-cloud -F", and that seems to work fine, and from what I can see it's the last API call made.

@gizahNL gizahNL mentioned this issue Feb 16, 2022
Closed
3 tasks
gizahNL added a commit to gizahNL/salt that referenced this issue Apr 7, 2022
@gizahNL
Fix saltstack#60568 & upgrade to V2 API
9e59578 
-Fix deploying of VM's
-Fix unimplemented functions (start/stop)

BREAKING CHANGES:
-V2 API regions are a string instead of a numeric ID
-V2 API plans (salt: sizes) are a string instead of a numeric ID
-ssh_key_names now actually takes the key names, instead of ID, as
 suggested by documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@garethgreenaway garethgreenaway

Labels
Bug broken, incorrect, or confusing behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix #60568 & upgrade to V2 API gizahNL/salt
4 participants
@garethgreenaway @fake-name @gizahNL @yun14u