highstate and gevent 1.1 not compatible

[jakehilton]

Just installed on a box running gevent 1.1 and the salt-call state.highstate fails with the following error:

# salt-call state.highstate
[INFO    ] Got list of available master addresses: ['smaster1-ause.xxxxx.com', 'smaster2-ause.xxxxx.com']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://top.sls'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[INFO    ] Syncing beacons for environment 'base'
[INFO    ] Loading cache from salt://_beacons, for base)
[INFO    ] Caching directory u'_beacons/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)
[INFO    ] Caching directory u'_modules/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base)
[INFO    ] Caching directory u'_states/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base)
[INFO    ] Caching directory u'_sdb/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base)
[INFO    ] Caching directory u'_grains/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base)
[INFO    ] Caching directory u'_renderers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base)
[INFO    ] Caching directory u'_returners/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/output'
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base)
[INFO    ] Caching directory u'_output/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base)
[INFO    ] Caching directory u'_utils/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[INFO    ] Syncing log_handlers for environment 'base'
[INFO    ] Loading cache from salt://_log_handlers, for base)
[INFO    ] Caching directory u'_log_handlers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base)
[INFO    ] Caching directory u'_proxy/' for environment 'base'

Passed invalid arguments: Cannot set attribute.

Usage:

    Retrieve the state data from the salt master for this minion and execute it

    test
        Notify states to execute in test-only (dry-run) mode.

        Sets the ``test`` variable in the minion ``opts`` for the duration of
        the state run.
    pillar
        Custom Pillar data can be passed with the ``pillar`` kwarg. Values
        passed here will override hard-coded Pillar values.
    queue : ``False``
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run so use this
        option sparingly.
    localconfig:
        Instead of using running minion opts, load ``localconfig`` and merge that
        with the running minion opts. This functionality is intended for using
        "roots" of salt directories (with their own minion config, pillars,
        file_roots) to run highstate out of.

    mock:
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    CLI Example:

    .. code-block:: bash

        salt '*' state.highstate

        salt '*' state.highstate whitelist=sls1_to_run,sls2_to_run
        salt '*' state.highstate exclude=sls_to_exclude
        salt '*' state.highstate exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

        salt '*' state.highstate pillar="{foo: 'Foo!', bar: 'Bar!'}"

Salt Version:
           Salt: 2015.8.5

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.22
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-77-generic
         system: Ubuntu 14.04 trusty

Just wondering if gevent 1.1 has been tested at all or will be? Downgrading to gevent 1.0.2 resolves this problem.. but unfortunately is not an option for me.

[jakehilton]

This traceback may help.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1071, in _thread_return
    return_data = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 568, in highstate
    whitelist=kwargs.get('whitelist')
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3107, in call_highstate
    self.load_dynamic(matches)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2643, in load_dynamic
    self.state.module_refresh()
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 804, in module_refresh
    reload(site)
  File "/usr/lib/python2.7/site.py", line 563, in <module>
    main()
  File "/usr/lib/python2.7/site.py", line 541, in main
    abs__file__()
  File "/usr/lib/python2.7/site.py", line 100, in abs__file__
    m.__file__ = os.path.abspath(m.__file__)
  File "/usr/local/lib/python2.7/dist-packages/gevent/__init__.py", line 87, in __setattr__
    raise TypeError("Cannot set attribute")
TypeError: Cannot set attribute

[jfindlay]

@jakehilton, thanks for reporting. Is it possible for you to narrow this down to a particular state, or does it only happen on highstate?

[jakehilton]

If I run a simple salt-call state.sls <somestate> it works fine.. it only happens when I run highstate as far as I've been able to repro.

It is really messing with my deployments because I have other processes that require the newer gevent.. but then salt breaks down. Is there another state besides highstate that I could call that would accomplish the same/similar thing?

[jfindlay]

@jakehilton, you might be able to workaround highstate in some contexts by simulating highstate with a top level state that includes all other states.

[jakehilton]

I did attempt that but it complained when putting the following type syntax in a state:

base:
  '*':
    - base
  '*services*':
    - filebeat
    - packetbeat
    - topbeat

I tried a few iterations including using import but nothing would stick.. I may have to just use some jinja logic to load those states?

[jakehilton]

I figured it out.. so for now I don't have my hair on fire.. but it sure would be nice to have this working.

Thank you!

[jakehilton]

Got bit by this again.. this time installing a simple package.

# salt-call state.sls encoder.install
[INFO    ] Got list of available master addresses: ['smaster1-ausw.xxxxx.com', 'smaster2-ausw.xxxxx.com']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** done ** u'encoder/install.sls'
[ERROR   ] You should upgrade pyOpenSSL to at least 0.14.1 to enable the use of X509 extensions
[INFO    ] Running state [traceroute] at time 18:30:42.215348
[INFO    ] Executing state pkg.installed for traceroute
[INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}\n', '-W'] in directory '/root'
[INFO    ] Executing command 'apt-get -q update' in directory '/root'
[INFO    ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'traceroute'] in directory '/root'
[INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}\n', '-W'] in directory '/root'
[INFO    ] Made the following changes:
'traceroute' changed from 'absent' to '1:2.0.20-0ubuntu0.1'


Passed invalid arguments: Cannot set attribute.

Usage:

    Execute a set list of state files from an environment.

    test
        Notify states to execute in test-only (dry-run) mode.

        Sets the ``test`` variable in the minion ``opts`` for the duration of
        the state run.
    pillar
        Custom Pillar data can be passed with the ``pillar`` kwarg. Values
        passed here will override hard-coded Pillar values.
    queue : ``False``
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run so use this
        option sparingly.
    saltenv : None
        Specify a ``file_roots`` environment.

        .. versionchanged:: 0.17.0
            Argument name changed from ``env`` to ``saltenv``.
        .. versionchanged:: 2014.7
            Defaults to None. If no saltenv is specified, the minion config will
            be checked for a saltenv and if found, it will be used. If none is found,
            base will be used.
    pillarenv : None
        Specify a ``pillar_roots`` environment. By default all pillar environments
        merged together will be used.
    concurrent:
        WARNING: This flag is potentially dangerous. It is designed
        for use when multiple state runs can safely be run at the same
        Do not use this flag for performance optimization.
    localconfig:
        Instead of using running minion opts, load ``localconfig`` and merge that
        with the running minion opts. This functionality is intended for using
        "roots" of salt directories (with their own minion config, pillars,
        file_roots) to run highstate out of.

    CLI Example:

    .. code-block:: bash

        salt '*' state.sls core,edit.vim dev
        salt '*' state.sls core exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

        salt '*' state.sls myslsfile pillar="{foo: 'Foo!', bar: 'Bar!'}"

If I run the salt call command again I don't get the error most likely because the package has been installed.

[jakehilton]

This has become an increasingly problematic issue.. to the point where it's made salt unusable for us. We have to run commands up to 7 times to finally get all packages installed/configured on the box.

Anyone looking into this by chance?

[jakehilton]

Not pretty but this will allow it to continue on its way and won't hold me up. In state.py:

def module_refresh(self):
         '''
         Refresh all the modules
         '''
         log.debug('Refreshing modules...')
         if self.opts['grains'].get('os') != 'MacOS':
             # In case a package has been installed into the current python
             # process 'site-packages', the 'site' module needs to be reloaded in
             # order for the newly installed package to be importable.
             try:
                 reload(site)
             except RuntimeError:
                 log.error('Error encountered during module reload. Modules were not reloaded.')
             except TypeError:
                 log.error('Error encountered during module reload. Modules were not reloaded.')

Just handling the exception is enough... dirty?.. maybe.. but seems to work in terms of not blowing up.

[rallytime]

@jakehilton Since your fix for this issue has been merged, can we close this issue? Or would you like to keep it open until the next release is available?

[jakehilton]

We can close this I think.. and if regresses we can re-open.

[rallytime]

Sounds good. Thanks!

[jakehilton]

Out of curiosity.. when is 2015.8.8 scheduled to land?

[rallytime]

Should be in the next couple of weeks or so.

[jakehilton]

Confirmed this looks good in 2015.8.8.