Highstate results in 'minion did not respond' in 2015.8.x #31837

cmeisinger · 2016-03-11T22:40:35Z

Description of Issue/Question

When bringing up a minion installed from the saltstack repo, state.highstates result in a "Minion did not return" error. This has been tested against two salt-masters and has carried through multiple versions of 2015.8.x.

Individual commands execute fine, only highstates generate this error.

It should be noted that minions installed via EPEL do not have this issue.

Setup:

Two new instances, one for master, one for a minion. Each installed using the saltstack repository. Sync over appropriate formula/pillar data and run a salt state.highstate.

Output:
[root@salt ~]# salt CInode\* state.highstate CInode99.scrubbeddomain.net: Minion did not return. [No response]

Versioning info:

Master version info

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-123.8.1.el7.x86_64
         system: CentOS Linux 7.2.1511 Core`

Client version info:

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: unknown
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.18.1.el6.x86_64
         system: CentOS 6.7 Final`

Debug Logging:

Error message from running the minion with -l debug.

`[INFO    ] Returning information for job: 20160311222639738441
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'slave99.ci.ourcentro.net', 'tcp://50.31.171.46:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'CInode99.scrubbeddomain.net', 'tcp://10.10.10.10:4506')
Process Process-1:2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1155, in _thread_return
    timeout=minion_instance._return_retry_timer()
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1305, in _return_pub
    ret_val = channel.send(load, timeout=timeout)
  File "/usr/lib/python2.6/site-packages/salt/utils/async.py", line 73, in wrap
    ret = self._block_future(ret)
  File "/usr/lib/python2.6/site-packages/salt/utils/async.py", line 83, in _block_future
    return future.result()
  File "/usr/lib64/python2.6/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.6/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.6/site-packages/salt/transport/zeromq.py", line 231, in send
    ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout)
  File "/usr/lib64/python2.6/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.6/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.6/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.6/site-packages/salt/transport/zeromq.py", line 199, in _crypted_transfer
    ret = yield _do_transfer()
  File "/usr/lib64/python2.6/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.6/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.6/site-packages/tornado/gen.py", line 230, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.6/site-packages/salt/transport/zeromq.py", line 183, in _do_transfer
    self._package_load(self.auth.crypticle.dumps(load)),
  File "/usr/lib/python2.6/site-packages/salt/crypt.py", line 1164, in dumps
    return self.encrypt(self.PICKLE_PAD + self.serial.dumps(obj))
  File "/usr/lib/python2.6/site-packages/salt/payload.py", line 140, in dumps
    return msgpack.dumps(msg)
  File "/usr/lib64/python2.6/site-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 223, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:223)
  File "msgpack/_packer.pyx", line 225, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:225)
  File "msgpack/_packer.pyx", line 184, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:184)
  File "msgpack/_packer.pyx", line 184, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:184)
  File "msgpack/_packer.pyx", line 184, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:184)
  File "msgpack/_packer.pyx", line 184, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:184)
  File "msgpack/_packer.pyx", line 184, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:184)
  File "msgpack/_packer.pyx", line 220, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:220)
TypeError: can't serialize set([' user'])
[INFO    ] User root Executing command saltutil.find_job with jid 20160311222839925637
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20160311222839925637', 'tgt': 'CInode-99*', 'ret': '', 'user': 'root', 'arg': ['20160311222639738441'], 'fun': 'saltutil.find_job'}

The text was updated successfully, but these errors were encountered:

jfindlay · 2016-03-14T18:16:05Z

@cmeisinger, thanks for reporting. This seems to be similar to #31692.

cachedout · 2016-04-07T19:53:56Z

OK, so what's happening here is that some state is returning a set somewhere in its output. Those are invalid and cannot be serialized by msgpack. Are you using any custom modules/states that might be doing this? If not, we might have let one slip in somewhere and we'll have to find it. It would be helpful to try to narrow your highstate down to the specific state that's causing this, if you can..

cmeisinger · 2016-04-09T00:02:53Z

@cachedout So we've got a lot of custom states, no custom modules. Any pointers on how to start running down the culprit?

Refs saltstack#31837

cachedout · 2016-06-03T18:15:48Z

@cmeisinger Sincere apologies for missing your reply. (I get kind of a crazy amount of email.)

I'd just grep your list of states for set or frozenset and see if anything comes back. You could also just watch the logs and see if you can determine which state is executing at the time that you see this happen.

However, I don't think this is a bug in Salt as this appears to be a custom state doing something that it shouldn't be. I have added a note to the documentation here: #33757

As such, I am closing this issue since I don't think we can resolve this by changing anything in Salt. However, we can still continue the discussion here to help you figure this out for your system. (I'll try not to wait two months to respond next time. Sorry again.) :]

jfindlay added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around Core relates to code central or existential to Salt P2 Priority 2 labels Mar 14, 2016

jfindlay added this to the Approved milestone Mar 14, 2016

jfindlay added the Needs-Testcase PR needs test cases written, or the issue is about a bug/feature that needs test cases label Apr 7, 2016

cachedout pushed a commit to cachedout/salt that referenced this issue Jun 3, 2016

Reminder not to return non-serializable data from states

500d4cc

Refs saltstack#31837

cachedout closed this as completed Jun 3, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Highstate results in 'minion did not respond' in 2015.8.x #31837

Highstate results in 'minion did not respond' in 2015.8.x #31837

cmeisinger commented Mar 11, 2016

jfindlay commented Mar 14, 2016

cachedout commented Apr 7, 2016

cmeisinger commented Apr 9, 2016

cachedout commented Jun 3, 2016

Highstate results in 'minion did not respond' in 2015.8.x #31837

Highstate results in 'minion did not respond' in 2015.8.x #31837

Comments

cmeisinger commented Mar 11, 2016

Description of Issue/Question

Setup:

Versioning info:

Debug Logging:

jfindlay commented Mar 14, 2016

cachedout commented Apr 7, 2016

cmeisinger commented Apr 9, 2016

cachedout commented Jun 3, 2016