salt-run produces many errors and warnings

[d101nelson]

Description of Issue/Question

Using 'salt-run' under salt-2016.11.2 produces many errors and warnings then just dies.

Setup

A functional master/minion setup.

Steps to Reproduce Issue

I've tried a number Salt supplied runners namely 'survey.hash', 'test.arg', and 'jobs.active'. They all produce the same errors.

On the salt-master:

# salt-run test.arg 'foo'                                                                                                                                                                                         
[ERROR   ] Exception raised when processing __virtual__ function for apache. Module will not be loaded 'os_family'                                                                                                               
[WARNING ] salt.loaded.int.module.apache.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'apache', please fix this.                      
[ERROR   ] Exception raised when processing __virtual__ function for bridge. Module will not be loaded 'kernel'                                                                                                                  
[WARNING ] salt.loaded.int.module.bridge.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'bridge', please fix this.                      
[ERROR   ] Exception raised when processing __virtual__ function for deb_apache. Module will not be loaded 'os_family'                                                                                                           
[WARNING ] salt.loaded.int.module.deb_apache.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'deb_apache', please fix this.              
[ERROR   ] Exception raised when processing __virtual__ function for debconfmod. Module will not be loaded 'os_family'                                                                                                           
[WARNING ] salt.loaded.int.module.debconfmod.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'debconfmod', please fix this.              
[ERROR   ] Exception raised when processing __virtual__ function for debian_ip. Module will not be loaded 'os_family'                                                                                                            
[WARNING ] salt.loaded.int.module.debian_ip.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'debian_ip', please fix this.                
[ERROR   ] Exception raised when processing __virtual__ function for debian_service. Module will not be loaded 'os'                                                                                                              
[WARNING ] salt.loaded.int.module.debian_service.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'debian_service', please fix this.      
[ERROR   ] Exception raised when processing __virtual__ function for dpkg. Module will not be loaded 'os_family'                                                                                                                 
[WARNING ] salt.loaded.int.module.dpkg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'dpkg', please fix this.                          
[ERROR   ] Exception raised when processing __virtual__ function for eix. Module will not be loaded 'os'                                                                                                                         
[WARNING ] salt.loaded.int.module.eix.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'eix', please fix this.                            
[ERROR   ] Exception raised when processing __virtual__ function for eselect. Module will not be loaded 'os'                                                                                                                     
[WARNING ] salt.loaded.int.module.eselect.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'eselect', please fix this.                    
[ERROR   ] Exception raised when processing __virtual__ function for freebsd_sysctl. Module will not be loaded 'os'                                                                                                              
[WARNING ] salt.loaded.int.module.freebsd_sysctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsd_sysctl', please fix this.      
[ERROR   ] Exception raised when processing __virtual__ function for freebsdjail. Module will not be loaded 'os'                                                                                                                 
[WARNING ] salt.loaded.int.module.freebsdjail.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsdjail', please fix this.            
[ERROR   ] Exception raised when processing __virtual__ function for freebsdkmod. Module will not be loaded 'kernel'                                                                                                             
[WARNING ] salt.loaded.int.module.freebsdkmod.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsdkmod', please fix this.            
[ERROR   ] Exception raised when processing __virtual__ function for freebsdpkg. Module will not be loaded 'os'                                                                                                                  
[WARNING ] salt.loaded.int.module.freebsdpkg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsdpkg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for freebsdports. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.freebsdports.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsdports', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for freebsdservice. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.freebsdservice.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'freebsdservice', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for gentoo_service. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.gentoo_service.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'gentoo_service', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for gentoolkitmod. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.gentoolkitmod.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'gentoolkitmod', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for groupadd. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.groupadd.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'groupadd', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for grub_legacy. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.grub_legacy.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'grub_legacy', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for kmod. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.kmod.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'kmod', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for layman. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.layman.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'layman', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for linux_ip. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.linux_ip.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'linux_ip', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for linux_sysctl. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.linux_sysctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'linux_sysctl', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for logadm. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.logadm.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'logadm', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for mac_sysctl. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.mac_sysctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'mac_sysctl', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for mac_xattr. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.mac_xattr.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'mac_xattr', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for makeconf. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.makeconf.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'makeconf', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for mdadm. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.mdadm.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'mdadm', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for netbsd_sysctl. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.netbsd_sysctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'netbsd_sysctl', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for netbsdservice. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.netbsdservice.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'netbsdservice', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for nspawn. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.nspawn.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'nspawn', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for openbsd_sysctl. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.openbsd_sysctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'openbsd_sysctl', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for openbsdpkg. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.openbsdpkg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'openbsdpkg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for openbsdrcctl. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.openbsdrcctl.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'openbsdrcctl', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for openbsdservice. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.openbsdservice.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'openbsdservice', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pacman. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.pacman.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pacman', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pkgin. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.pkgin.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pkgin', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pkgng. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.pkgng.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pkgng', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pkgutil. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.pkgutil.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pkgutil', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for poudriere. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.poudriere.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'poudriere', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pw_group. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.pw_group.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pw_group', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for pw_user. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.pw_user.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'pw_user', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for rh_ip. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.rh_ip.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'rh_ip', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for rh_service. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.rh_service.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'rh_service', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for service. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.service.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'service', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for smf. Module will not be loaded 'os_family'
[WARNING ] salt.loaded.int.module.smf.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'smf', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for solaris_group. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.solaris_group.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'solaris_group', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for solaris_user. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.solaris_user.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'solaris_user', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for solarisips. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.solarisips.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'solarisips', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for solarispkg. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.solarispkg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'solarispkg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for systemd. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.systemd.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'systemd', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for useradd. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.useradd.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'useradd', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for win_powercfg. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.win_powercfg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'win_powercfg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for xapi. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.xapi.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'xapi', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for xbpspkg. Module will not be loaded 'os'
[WARNING ] salt.loaded.int.module.xbpspkg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'xbpspkg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for zfs. Module will not be loaded 'kernel'
[WARNING ] salt.loaded.int.module.zfs.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'zfs', please fix this.
[ERROR   ] Failed to import utils saltminionservice, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/loader.py", line 1332, in _load_module
    ), fn_, fpath, desc)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/saltminionservice.py", line 8, in <module>
    from salt.utils.winservice import Service, instart
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/winservice.py", line 33, in <module>
    class Service(win32serviceutil.ServiceFramework):
NameError: name 'win32serviceutil' is not defined
[ERROR   ] Failed to import utils win_osinfo, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/loader.py", line 1332, in _load_module
    ), fn_, fpath, desc)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/win_osinfo.py", line 32, in <module>
    class OSVERSIONINFO(ctypes.Structure):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/win_osinfo.py", line 33, in OSVERSIONINFO
    _fields_ = (('dwOSVersionInfoSize', DWORD),
NameError: name 'DWORD' is not defined
[ERROR   ] Failed to import utils winservice, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/loader.py", line 1332, in _load_module
    ), fn_, fpath, desc)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/winservice.py", line 33, in <module>
    class Service(win32serviceutil.ServiceFramework):
NameError: name 'win32serviceutil' is not defined
Exception occurred in runner test.arg: Traceback (most recent call last):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/client/mixins.py", line 394, in _low
    with tornado.stack_context.StackContext(self.functions.context_dict.clone):
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/tornado/stack_context.py", line 135, in __enter__
    self.enter()
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/tornado/stack_context.py", line 118, in enter
    context = self.context_factory()
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/context.py", line 92, in clone
    child = ChildContextDict(parent=self, overrides=kwargs)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/site-packages/salt/utils/context.py", line 142, in __init__
    self._data[k] = copy.deepcopy(v)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/apps/unix/salt-2016.11.2/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)
TypeError: can't pickle thread.lock objects
# 

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2016.11.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.0
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.5
        libgit2: Not Installed
        libnacl: 1.5.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.13 (default, Feb  3 2017, 15:47:32)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: 0.2.4
        Tornado: 4.4.2
            ZMQ: 4.1.5
 
System Versions:
           dist: redhat 7.2 Maipo
        machine: x86_64
        release: 3.10.0-327.36.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.2 Maipo
 
# 

[gtmanfred]

How have you installed salt?

We provide packages for redhat systems, but it look like you are managing salt in opt?

https://repo.saltstack.com/#rhel

With our packages, I am unable to replicate this issue.

[root@salt salt]# salt-run test.arg 'foo'
args:
    - foo
kwargs:
    ----------

Thanks,
Daniel

[d101nelson]

@gtmanfred Because of our non-homogenous environment, there's a central build script that compiles everything from source so that everything is consistent across the board regardless of OS/platform. It's encouraging to hear that the problem is local to me. I'll kickoff a fresh compile and see if I can replicate the problem and go from there.

[twangboy]

@d101nelson The above two PR's will fix the stack traces caused by the Windows specific salt utils.

[d101nelson]

@gtmanfred This seems to have been something with corruption in our local PyPI repo. Got that reset and rebuilt salt-2016.11.2 and all seems good.

@twangboy Thanks for the heads-up on those PRs. I'll keep those in mind if I run into them.

[d101nelson]

I spoke to soon. Not realizing it, I inadvertently reinstalled 2016.3.3 and tested against that.

So, I've installed 2011.11.2 and the problem persists. I manually applied the PRs from @twangboy and that fixed the Windows related errors but the "Exception raised when processing ...",
"salt.loaded.int.module....", and 'copy.py' errors persist.

I'm now backtracking through salt releases and 2016.3.5 runners work as expected; unfortunately, I need 2016.11.x to address some beacon issues I ran into with 2016.3.x. I'm also looking at the 2016.11.2 RPM's for any clues, too.

[d101nelson]

@gtmanfred Found it. In my /etc/salt/master I have defined:

runner_dirs: 
  - /srv/runners

If I comment this out, then 'salt-run test.arg foo' runs as expected. How should I have 'runner_dirs' configured as I have custom runners.

[gtmanfred]

If you put the runners in the _runners directory in your salt file server, you can run ’salt-run saltutil.sync_runners’ and have them synced to the master extmods That is how I manage them. Daniel

…

On Mon, Feb 13, 2017 at 4:52 PM d101nelson ***@***.***> wrote: @gtmanfred <https://github.com/gtmanfred> Found it. In my /etc/salt/master I have defined: runner_dirs: - /srv/runners If I comment this out, then 'salt-run test.arg foo' runs as expected. How should I have 'runner_dirs' configured as I have custom runners. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39323 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoZeFlQ7uDfx6RIpCJgYkGYQiylyYks5rcN6-gaJpZM4L91zt> .

[d101nelson]

@gtmanfred I did what you suggested and it worked - but only once. Subsequent salt-run saltutil.sync_runners get the same error messages as above. I narrowed down the problem to the __virtual__ function in my runner. I store the local client's context in __context__ because I need access to grain and pillar data throughout my runner and __virtual__() uses this to verify that configuration data is available and returns True/False as necessary.

I've reordered the code in the runner so that __virtual__() always returns True and configuration checks are done in the runner's various functions.

A simple example runner demonstrates the problem.

import logging
import salt.client

log = logging.getLogger(__name__)

def __virtual__():
    '''
    A simple example that breaks runners
    '''
    if 'client' not in __context__:
        client = salt.client.LocalClient()
        __context__['client'] = client
    return(True)