AttributeError: type object 'IOLoop' has no attribute 'initialized'

[yatahaze]

Used salt for the past 2 days, only running the highstate commands. Went back to it this morning, none of the services are working. They all give the following error:

2018-03-05 12:29:16,104 [salt.utils.process:670 ][ERROR   ][19428] An un-handled exception from the multiprocessing process 'EventPublisher-8' was caught:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/process.py", line 663, in _run
    return self._original_run()
  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 1107, in run
    self.publisher.start()
  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 517, in start
    io_loop=self.io_loop,
TypeError: add_accept_handler() got an unexpected keyword argument 'io_loop'

(Same TypeError for other processes is outputted to, just omitted them from this report for now.)

I went through everything and got everything up to the latest versions, but it was all unrelated packages.

Looking back at my logs it looks like my server had run out of memory at some point prior to this, maybe the cause of the bugs? I've also cleared the cache for the master and minion.

Versions Report

Salt Version:
           Salt: 2017.7.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 5.0
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[gtmanfred]

        Tornado: 5.0

We do not currently support tornado 5.0

https://github.com/saltstack/salt/blob/v2017.7.4/requirements/base.txt#L6

Please downgrade tornado.

[yatahaze]

Seems pip was able to install a newer version than the system and override things on me somehow. Strange! I've got it fixed now though.

[konfiot]

Is there any plan to support tornado 5.0 ? because it is now default in debian testing

[toabctl]

Is there any plan to support tornado 5.0 ? because it is now default in debian testing

and the corresponding Debian bug is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=893817

[zetneteork]

Fast work around to fix issue because of python-tornado 5.0.1 on debian-testing
https://gist.github.com/zetneteork/22a6d11b341ae88f55b95cf38445d836

[toabctl]

Fast work around to fix issue because of python-tornado 5.0.1 on debian-testing
https://gist.github.com/zetneteork/22a6d11b341ae88f55b95cf38445d836

What about python3? Also this is a problem also for other distros. So this should be fixed in salt itself...

[gtmanfred]

We will be getting tornado 5.0 support in 2017.7.6 and 2018.3.1

[toabctl]

We will be getting tornado 5.0 support in 2017.7.6 and 2018.3.1

@gtmanfred when will that happen? is there any schedule?

[gtmanfred]

Sometime after this pr is merged #47106