Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run ESXi proxy minion #61982

Open
ggiesen opened this issue Apr 24, 2022 · 1 comment
Open

[BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run ESXi proxy minion #61982

ggiesen opened this issue Apr 24, 2022 · 1 comment
Labels
Bug broken, incorrect, or confusing behavior Proxy-Minion severity-high 2nd top severity, seen by most users, causes major problems
Milestone
Approved

Comments

@ggiesen
Copy link
Contributor

ggiesen commented Apr 24, 2022 •
edited

Description
Trying to run an ESXi proxy minion against a standalone ESXi 7.0.3 host. When I run salt-proxy --proxyid='hv01' -l debug, I get the following output over and over:

[CRITICAL] Unexpected error while connecting to salt01.example.com
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 79, in _get_top_file_envs
    return __context__["saltutil._top_file_envs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/context.py", line 78, in __getitem__
    return self.value()[item]
KeyError: 'saltutil._top_file_envs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1140, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1369, in connect_master
    yield self._post_master_init(master)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib64/python3.6/types.py", line 182, in send
    return self.__wrapped.send(val)
  File "/usr/lib/python3.6/site-packages/salt/metaproxy/proxy.py", line 133, in post_master_init
    self.functions["saltutil.sync_all"](saltenv=self.opts["saltenv"])
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1050, in sync_all
    ret["clouds"] = sync_clouds(saltenv, False, extmod_whitelist, extmod_blacklist)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 764, in sync_clouds
    ret = _sync("clouds", saltenv, extmod_whitelist, extmod_blacklist)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 101, in _sync
    saltenv = _get_top_file_envs()
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 85, in _get_top_file_envs
    envs = list(st_.top_matches(top).keys()) or "base"
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 3968, in top_matches
    ext_matches = self._master_tops()
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 3987, in _master_tops
    return self.client.master_tops()
  File "/usr/lib/python3.6/site-packages/salt/fileclient.py", line 1418, in master_tops
    return self.channel.send(load)
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 351, in send
    load, tries=tries, timeout=timeout, raw=raw
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 318, in _crypted_transfer
    ret = yield _do_transfer()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 299, in _do_transfer
    self._package_load(self.auth.crypticle.dumps(load)),
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
    toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
  File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
    msg, default=ext_type_encoder, use_bin_type=use_bin_type
  File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
    return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
  File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  [Previous line repeated 1 more time]
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] "salt01.example.com" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://[2001:db8:2:1516::11]:4506
[DEBUG   ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (1883)
[DEBUG   ] Setting zmq_reconnect_ivl to '1883ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://[2001:db8:2:1516::11]:4505
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/proxy/minion.pem
[DEBUG   ] No metaproxy key found in opts for id hv01. Defaulting to standard proxy minion
[DEBUG   ] LazyLoaded proxy.post_master_init
[DEBUG   ] subclassed LazyLoaded _post_master_init
[DEBUG   ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://[2001:db8:2:1516::11]:4506
[DEBUG   ] Trying to connect to: tcp://[2001:db8:2:1516::11]:4506
[ERROR   ] Exception getting pillar:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 242, in compile_pillar
    dictkey="pillar",
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 232, in crypted_transfer_decode_dictentry
    self._package_load(self.auth.crypticle.dumps(load)),
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
    toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
  File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
    msg, default=ext_type_encoder, use_bin_type=use_bin_type
  File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
    return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
  File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[ERROR   ] Error while bringing up minion for multi-master. Is master at salt01.example.com responding?

Setup
/srv/pillar/hv01.sls on master (salt01)

proxy:
  proxytype: esxi
  host: hv01.example.com
  username: < removed >
  passwords: 
    - < removed >

/etc/salt/proxy.d/proxy.conf on proxy host (salt-proxy02)

master:
  - salt01.example.com
  - salt02.example.com

Steps to Reproduce the behavior
Start the proxy minion: salt-proxy --proxyid='hv01' -l debug

Expected behavior
Proxy minion should connect to master

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: 0.26.8
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 0.26.4
        Python: 3.6.8 (default, Nov 17 2021, 16:10:06)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.5 Arctic Sphynx
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-348.20.1.el8_5.x86_64
        system: Linux
       version: AlmaLinux 8.5 Arctic Sphynx

Additional context
Both master and proxy host are running on AlmaLinux 8.5

pyVmomi version on proxy host is 7.0.3
@ggiesen ggiesen added Bug broken, incorrect, or confusing behavior needs-triage labels Apr 24, 2022
@ggiesen
Copy link
Contributor Author

ggiesen commented Apr 24, 2022 •
edited

If I disable IPv6 on the esxi host completely (not viable long-term), then the proxy successfully connects. However I then do a test.ping and get the following:

INFO    ] Starting a new job with PID 133834
[DEBUG   ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG   ] LazyLoaded test.ping
[WARNING ] The minion function caused an exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/metaproxy/proxy.py", line 483, in thread_return
    opts, data, func, args, kwargs
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/test.py", line 127, in ping
    return __proxy__[ping_cmd]()
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/proxy/esxi.py", line 456, in ping
    if DETAILS["mechanism"] == "userpass":
KeyError: 'mechanism'
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/proxy/minion_event_c11df35763_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/proxy/minion_event_c11df35763_pull.ipc
[DEBUG   ] Sending event: tag = _salt_error; data = {'message': 'The minion function caused an exception', 'args': ('The minion function caused an exception',), '_stamp': '2022-04-24T23:57:07.946579'}
[DEBUG   ] Closing IPCMessageClient instance

and an esx.cmd:

salt hv01 esxi.cmd system_info
hv01:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/metaproxy/proxy.py", line 483, in thread_return
        opts, data, func, args, kwargs
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/esxi.py", line 56, in cmd
        return __proxy__[proxy_cmd](command, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/proxy/esxi.py", line 497, in ch_config
        for k in kwargs:
    RuntimeError: dictionary changed size during iteration

Sending a pillar.get works however.

@ggiesen ggiesen changed the title [BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run esxi proxy minion [BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run ESXi proxy minion Apr 25, 2022
@OrangeDog OrangeDog added severity-high 2nd top severity, seen by most users, causes major problems Proxy-Minion and removed needs-triage labels Apr 25, 2022
@OrangeDog OrangeDog added this to the Approved milestone Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug broken, incorrect, or confusing behavior Proxy-Minion severity-high 2nd top severity, seen by most users, causes major problems
Projects
None yet
Milestone
Approved
Development

No branches or pull requests
2 participants
@OrangeDog @ggiesen