RuntimeError: IOLoop is already running

[spyd3r]

Description of Issue/Question

The minion function caused an exception: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1606, in _thread_return
return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "/var/cache/salt/minion/extmods/modules/io_loop_error.py", line 4, in test_func
caller = salt.client.Caller()
File "/usr/lib/python2.7/dist-packages/salt/client/init.py", line 2094, in init
self.sminion = salt.minion.SMinion(self.opts)
File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 793, in init
lambda: self.eval_master(self.opts, failed=True)
File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 440, in run_sync
self.start()
File "/usr/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 162, in start
super(ZMQIOLoop, self).start()
File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 731, in start
raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running

Setup

Create custom module io_loop_error.py for testing

def test_func():
    caller = salt.client.Caller()
    return caller.cmd('test.ping')

Steps to Reproduce Issue

Both steps are taken on a salt master

sudo salt-call io_loop_error.test_func

local:
    True

sudo salt '<minion_id>' io_loop_error.test_func
<minion_id>:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1606, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/io_loop_error.py", line 4, in test_func
        caller = salt.client.Caller()
      File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 2094, in __init__
        self.sminion = salt.minion.SMinion(self.opts)
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 793, in __init__
        lambda: self.eval_master(self.opts, failed=True)
      File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 440, in run_sync
        self.start()
      File "/usr/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 162, in start
        super(ZMQIOLoop, self).start()
      File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 731, in start
        raise RuntimeError("IOLoop is already running")
    RuntimeError: IOLoop is already running

Versions Report

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1063-aws
         system: Linux
        version: Ubuntu 16.04 xenial

[spyd3r]

Possibly related to #33697

[gtmanfred]

This is probably related to #46905

Why are you trying to initialize a new caller client inside a module?

You should already have access to all execution modules using __salt__?

def test_func():
    return __salt__['test.ping']()

[spyd3r]

Using caller is how I'd learned to do it. Switching to __salt__ fixes the problem.

[cmclaughlin]

I have the same error calling a execution module from pillar:

2018-08-07 17:20:20,906 [salt.pillar      :653 ][CRITICAL][31597] Rendering SLS 'XXXXX' failed, render error:
Jinja error: IOLoop is already running
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 425, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/XXXX/XXXXX.py", line 246, in get_user
    caller = salt.client.Caller()
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 2053, in __init__
    self.sminion = salt.minion.SMinion(self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 650, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 440, in run_sync
    self.start()
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 731, in start
    raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running

@gtmanfred I understand it might be related to issue #46905. However, I followed the suggestion there to upgrade pyzmq from 14.0.1 to 17.1.0 and I still have this problem. Do you want to reopen this and treat it as a separate problem?

[gtmanfred]

Can you show me the code you are using?

[cmclaughlin]

{% set foo = salt['my_module.my_method']('arg1', 'arg2') %}

[gtmanfred]

What is in your my_module.my_method?

[cmclaughlin]

Oh sorry, my_method was creating new caller clients. I thought I removed that... but apparently saltutil.sync_modules didn't deploy my change. Had to run highstate to get it deployed.

[gtmanfred]

👍