Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CRITICAL] Missing python-concurrent.futures leads to infinite loop error logging on salt 2018.3.3 #50241

Closed
tomlaredo opened this issue Oct 26, 2018 · 2 comments
Closed

[CRITICAL] Missing python-concurrent.futures leads to infinite loop error logging on salt 2018.3.3 #50241

tomlaredo opened this issue Oct 26, 2018 · 2 comments
Labels
Duplicate Duplicate of another issue or PR - will be closed

Comments

@tomlaredo
Copy link

tomlaredo commented Oct 26, 2018 •
edited

Description of Issue/Question

Upgrading from 2018.3.2 to 2018.3.3 on a server that doesn't have python-concurrent.futures installed on Debian 9 (stretch) makes salt-minion fail and write this error message in an infinite way, filling logs of all my servers:

Oct 26 06:25:24 backup-rbx salt-minion[31105]: [CRITICAL] Unexpected error while connecting to xx.xx.xx.xx
Oct 26 06:25:24 backup-rbx salt-minion[31105]: Traceback (most recent call last):
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 997, in _connect_minion
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yield minion.connect_master(failed=failed)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     value = future.result()
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     raise_exc_info(self._exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1021, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yielded = self.gen.throw(*exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1188, in connect_master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     master, self.pub_channel = yield self.eval_master(self.opts, self.timeout, self.safe, failed)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     value = future.result()
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     raise_exc_info(self._exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yielded = next(result)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 626, in eval_master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     pub_channel = salt.transport.client.AsyncPubChannel.factory(opts, **factory_kwargs)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/client.py", line 161, in factory
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.transport.zeromq
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/zeromq.py", line 31, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.transport.mixins.auth
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/mixins/auth.py", line 16, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 35, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.client.ssh.client
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 47, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.utils.thin
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/utils/thin.py", line 16, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import concurrent
Oct 26 06:25:24 backup-rbx salt-minion[31105]: ImportError: No module named concurrent
Oct 26 06:25:24 backup-rbx salt-minion[31105]: [INFO    ] Moving possibly failed master 193.248.36.25 to the end of the list of masters
Oct 26 06:25:24 backup-rbx salt-minion[31105]: [WARNING ] Master ip address changed from xx.xx.xx.xx to yy.yy.yy.yy
Oct 26 06:25:24 backup-rbx salt-minion[31105]: [WARNING ] Master ip address changed from xx.xx.xx.xx to zz.zz.zz.zz
Oct 26 06:25:24 backup-rbx salt-minion[31105]: [CRITICAL] Unexpected error while connecting to master.salt.rodacom.net
Oct 26 06:25:24 backup-rbx salt-minion[31105]: Traceback (most recent call last):
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 997, in _connect_minion
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yield minion.connect_master(failed=failed)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     value = future.result()
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     raise_exc_info(self._exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1021, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yielded = self.gen.throw(*exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1188, in connect_master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     master, self.pub_channel = yield self.eval_master(self.opts, self.timeout, self.safe, failed)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     value = future.result()
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     raise_exc_info(self._exc_info)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     yielded = next(result)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 626, in eval_master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     pub_channel = salt.transport.client.AsyncPubChannel.factory(opts, **factory_kwargs)
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/client.py", line 161, in factory
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.transport.zeromq
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/zeromq.py", line 31, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.transport.mixins.auth
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/transport/mixins/auth.py", line 16, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.master
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 35, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.client.ssh.client
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 47, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import salt.utils.thin
Oct 26 06:25:24 backup-rbx salt-minion[31105]:   File "/usr/lib/python2.7/dist-packages/salt/utils/thin.py", line 16, in <module>
Oct 26 06:25:24 backup-rbx salt-minion[31105]:     import concurrent
Oct 26 06:25:24 backup-rbx salt-minion[31105]: ImportError: No module named concurrent

An insteresting point is that python-concurrent.futures MIGHT be present if apt is configured to install "recommended" dependencies as tornado recommends it:

# aptitude why python-concurrent.futures
i   salt-common    Depends     python-tornado (>= 4.2.1)
i A python-tornado Recommends python-concurrent.futures

Short term solution is to manually install python-concurrent.futures.

This is a critical bug as i just lost every server due to this bug.

Steps to Reproduce Issue

On Debian 9 (stretch) with salt-minion 2018.3.2 installed and without python-concurrent.futures:
apt install salt-minion

Versions Report

# salt-call --versions-report
Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.5 
         locale: UTF-8
        machine: x86_64
        release: 3.14.32-xxxx-grs-ipv6-64
         system: Linux
        version: debian 9.5
@dmurphy18 dmurphy18 added the Duplicate Duplicate of another issue or PR - will be closed label Oct 26, 2018
@dmurphy18
Copy link
Contributor

@tomlaredo This is being handled vmware-archive/salt-pack#593

But basically stretch supplies python-tornado (not SaltStack) which has a recommends on concurrent-futures, not a depends. Salt is using the python-tornado that is supplied with the OS.

However shall look into steps to help with the issue.

@dmurphy18
Copy link
Contributor

Closing this since being address as detailed above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Duplicate Duplicate of another issue or PR - will be closed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tomlaredo @dmurphy18