Skip to content
This repository has been archived by the owner on Jun 16, 2023. It is now read-only.

Ensure python-tornado builds from SaltStack have a dependency on concurrent-futures #593

Closed
dmurphy18 opened this issue Oct 24, 2018 · 13 comments
Closed

Ensure python-tornado builds from SaltStack have a dependency on concurrent-futures #593

dmurphy18 opened this issue Oct 24, 2018 · 13 comments
Assignees
@dmurphy18
Labels
Packaging Platform TEAM Platform
Milestone
Approved

Comments

@dmurphy18
Copy link
Contributor

dmurphy18 commented Oct 24, 2018 •
edited

Check Ubuntu 16.04 esp.
@dmurphy18 dmurphy18 added this to the Approved milestone Oct 24, 2018
@dmurphy18 dmurphy18 self-assigned this Oct 24, 2018
@arminbuerkle
Copy link

We ran into this issue today as well, when we updated our salt-minions to 2017.7.8+ds-1 on debian stretch.

Highstates resulted in an exception:

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1477, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 860, in highstate
        conflict = _check_queue(queue, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 250, in _check_queue
        conflict = running(concurrent=kwargs.get('concurrent', False))
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 209, in running
        active = __salt__['saltutil.is_running']('state.*')
      File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1121, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 101, in __getitem__
        raise KeyError(key)
    KeyError: 'saltutil.is_running'

Manually installing apt-get install -y python-concurrent.futures fixed the issue.

@gtmanfred gtmanfred mentioned this issue Oct 25, 2018
Closed
@Ch3LL Ch3LL mentioned this issue Oct 25, 2018
Closed
@sblaisot
Copy link

Also seen on debian 9

@dmurphy18
Copy link
Contributor Author

@sblaisot To offer a fuller explanation, SaltStack builds python-tornado for Python 2.7 which utilizes concurrent.futures (python-futures in salt-pack) for the following OS's:
Raspbian 8
Debian 8
Ubuntu 16.04
Ubuntu 14.04

for other OS's the version of python-tornado (and its dependencies/recommends on concurrent.futures) is utilized. Also if one of the above OS's provides a version of python-tornado later then v4.2.1, then it is usual to utilize the later version provided by the OS.

Python 3 support for python-tornado is usually provided by the OS since SaltStack has a minimum requirement of python-tornado v4.2.1, and Python 3 implementations are typically using a later version.

Hope this clears the issue for Debian/Ubuntu OS's.

Python-tornado v4.2.1 is also built for RHEL 7 and 6, and Amazon Linux 1. There is a not a firm dependency for python-futures there, and unit test checks shall be conducted to see if it is required. Noting that EPEL is recently updated with a later version of python-tornado than that supplied by SaltStack.

This was referenced Oct 26, 2018
Closed
Closed
@dmurphy18
Copy link
Contributor Author

Updating Debian 9 and Ubuntu 18.04 packages to require concurrent-futures to alleviate this issue. Other platforms should be unaffected.

@lomeroe
Copy link

lomeroe commented Oct 27, 2018

@dmurphy18 I had CentOS boxes affected on fresh installs of 2018.3.3 and 2017.7.8 A manual install of python2-futures was required to get them running.

These were minimal CentOS builds with salt installed by the salt bootstrap script.

salt.utils.thin imports concurrent, which is imported by saltutil, so the python-futrues (or whatever package provides "concurrent" for a PY2 based distribution) is required to run the saltutil module

#50220 has my comments on this with debug logs/etc

@gtmanfred gtmanfred mentioned this issue Oct 29, 2018
Closed
@dmurphy18
Copy link
Contributor Author

dmurphy18 commented Oct 29, 2018 •
edited

@Iomeroe After the Debian/Ubuntu issue with python-tornado only now recommending concurrent.futures, I checked the spec file for the Redhat / Centos builds and python-futures is a Requires:, so the packaging is correct. Installing by the bootstrap should have utilized the requirements file which also includes futures.

I shall take a closer look however and unit test to see if I can duplicate your issue, since I had thought the Redhat family should have been unaffected by this. In the past, I have seen issues with minimal CentOS builds where even sudo was not part of the minimal version of CentOS (RHEL 6), and the packages available in minimal installs changed between minor versions, for example: RHEL 6.1, 6.2, 6.3. Hence the problem may be due to the minimal version of CentOS not containing all of the required packages for Salt to function rather than a script or packaging error.

Can you provide the version of minimal CentOS that you are using, or confirm that it is CentOS Linux 7.5.1804 Core (from #50220)

@dmurphy18
Copy link
Contributor Author

@Iomeroe After installing CentOS 7.5 minimal, I found that python-futures is listed as a dependency for salt:

[root@localhost ~]# yum list python*-futures
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

  • base: sjc.edge.kernel.org
  • extras: mirror.hostduplex.com
  • updates: repos.dfw.quadranet.com
    salt-latest | 2.9 kB 00:00:00
    salt-latest/x86_64/primary_db | 42 kB 00:00:00
    Available Packages
    python-futures.noarch 3.0.3-1.el7 salt-latest
    [root@localhost ~]# ypu deplist salt
    -bash: ypu: command not found
    [root@localhost ~]# yum deplist salt
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
  • base: sjc.edge.kernel.org
  • extras: mirror.hostduplex.com
  • updates: repos.dfw.quadranet.com
    package: salt.noarch 2018.3.3-1.el7
    dependency: /bin/sh
    provider: bash.x86_64 4.2.46-30.el7
    dependency: /usr/bin/env
    provider: coreutils.x86_64 8.22-21.el7
    dependency: /usr/bin/python
    provider: python.x86_64 2.7.5-69.el7_5
    dependency: PyYAML
    provider: PyYAML.x86_64 3.11-1.el7
    dependency: pciutils
    provider: pciutils.x86_64 3.5.1-3.el7
    dependency: python(abi) = 2.7
    provider: python.x86_64 2.7.5-69.el7_5
    dependency: python-crypto >= 2.6.1
    provider: python2-crypto.x86_64 2.6.1-15.el7
    provider: python-crypto.x86_64 2.6.1-2.el7
    dependency: python-futures >= 2.0
    provider: python-futures.noarch 3.0.3-1.el7
    dependency: python-jinja2
    provider: python-jinja2.noarch 2.7.2-2.el7
    dependency: python-markupsafe
    provider: python-markupsafe.x86_64 0.11-10.el7
    dependency: python-msgpack >= 0.4
    provider: python-msgpack.x86_64 0.4.6-1.el7
    dependency: python-psutil
    provider: python-psutil.x86_64 2.2.1-1.el7
    dependency: python-requests >= 1.0.0
    provider: python-requests.noarch 2.6.0-1.el7_1
    dependency: python-six
    provider: python-six.noarch 1.9.0-2.el7
    dependency: python-tornado >= 4.2.1
    provider: python-tornado.x86_64 4.2.1-3.el7
    dependency: python-tornado < 6.0
    provider: python-tornado.x86_64 4.2.1-3.el7
    dependency: python-zmq
    provider: python-zmq.x86_64 15.3.0-3.el7
    dependency: systemd-python
    provider: systemd-python.x86_64 219-57.el7_5.3
    dependency: systemd-units
    provider: systemd.x86_64 219-57.el7_5.3
    dependency: which
    provider: which.x86_64 2.20-7.el7
    dependency: yum-utils
    provider: yum-utils.noarch 1.1.31-46.el7_5
    [root@localhost ~]# yum deplist salt| less
    [root@localhost ~]# yum deplist salt-ssh
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
  • base: sjc.edge.kernel.org
  • extras: mirror.hostduplex.com
  • updates: repos.dfw.quadranet.com
    package: salt-ssh.noarch 2018.3.3-1.el7
    dependency: /usr/bin/python
    provider: python.x86_64 2.7.5-69.el7_5
    dependency: salt = 2018.3.3-1.el7
    provider: salt.noarch 2018.3.3-1.el7
    [root@localhost ~]#

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@localhost ~]#

Installing salt-ssh also install python-futures
Installed:
salt-ssh.noarch 0:2018.3.3-1.el7

Dependency Installed:
PyYAML.x86_64 0:3.11-1.el7 libsodium.x86_64 0:1.0.16-1.el7 libtomcrypt.x86_64 0:1.17-26.el7 libtommath.x86_64 0:0.42.0-6.el7
libxml2-python.x86_64 0:2.9.1-6.el7_2.3 libyaml.x86_64 0:0.1.4-11.el7_0 openpgm.x86_64 0:5.2.122-2.el7 pciutils.x86_64 0:3.5.1-3.el7
python-babel.noarch 0:0.9.6-8.el7 python-backports.x86_64 0:1.0-8.el7 python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 python-chardet.noarch 0:2.2.1-1.el7_1
python-futures.noarch 0:3.0.3-1.el7 python-ipaddress.noarch 0:1.0.16-2.el7 python-jinja2.noarch 0:2.7.2-2.el7 python-kitchen.noarch 0:1.1.1-5.el7
python-markupsafe.x86_64 0:0.11-10.el7 python-msgpack.x86_64 0:0.4.6-1.el7 python-psutil.x86_64 0:2.2.1-1.el7 python-requests.noarch 0:2.6.0-1.el7_1
python-six.noarch 0:1.9.0-2.el7 python-tornado.x86_64 0:4.2.1-3.el7 python-urllib3.noarch 0:1.10.2-5.el7 python-zmq.x86_64 0:15.3.0-3.el7
python2-crypto.x86_64 0:2.6.1-15.el7 salt.noarch 0:2018.3.3-1.el7 systemd-python.x86_64 0:219-57.el7_5.3 yum-utils.noarch 0:1.1.31-46.el7_5
zeromq.x86_64 0:4.1.4-7.el7

@dmurphy18
Copy link
Contributor Author

The updated Debian 9 and Ubuntu 18.04 packages for 2017.7.8-2 and 2018.3.3-2 have been delivered to test.

@dmurphy18
Copy link
Contributor Author

@Iomeroe If the problem persists on Redhat, please open another issue and assign it to dmurphy18 and I shall look into it further. Want a different issue since this is dealing with Ubuntu/Debian.

Thanks

@dmurphy18 dmurphy18 mentioned this issue Oct 29, 2018
Closed
@dmurphy18
Copy link
Contributor Author

The fixes are now live on repo.saltstack.com

@wwentland wwentland mentioned this issue Oct 30, 2018
Closed
@dmurphy18 dmurphy18 mentioned this issue Oct 31, 2018
Closed
@jryberg
Copy link

jryberg commented Nov 13, 2018 •
edited

Hi,

This problem exist on CentOS 7 as well but it does not seems this has been addressed? At least not for 2017.7.8

@dmurphy18, did you have another case for CentOS?

@dmurphy18
Copy link
Contributor Author

@jryberg A better place to comment would be salt-pack #598 since it is open and this issue is closed.
But examining the current Salt for 2017.7.8, it would appear that python-futures is a dependency and is provided for Centos 7.

root@localhost:~# yum deplist salt
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile

  • base: mirrors.xmission.com
  • extras: mirrors.xmission.com
  • updates: mirrors.xmission.com
    package: salt.noarch 2017.7.8-1.el7
    dependency: /bin/sh
    provider: bash.x86_64 4.2.46-30.el7
    dependency: /usr/bin/env
    provider: coreutils.x86_64 8.22-21.el7
    dependency: /usr/bin/python2
    provider: python.x86_64 2.7.5-69.el7_5
    dependency: PyYAML
    provider: PyYAML.x86_64 3.11-1.el7
    dependency: pciutils
    provider: pciutils.x86_64 3.5.1-3.el7
    dependency: python(abi) = 2.7
    provider: python.x86_64 2.7.5-69.el7_5
    dependency: python-crypto >= 2.6.1
    provider: python2-crypto.x86_64 2.6.1-15.el7
    provider: python-crypto.x86_64 2.6.1-2.el7
    dependency: python-futures >= 2.0
    provider: python-futures.noarch 3.0.3-1.el7
    dependency: python-jinja2
    provider: python-jinja2.noarch 2.7.2-2.el7
    dependency: python-markupsafe
    provider: python-markupsafe.x86_64 0.11-10.el7
    dependency: python-msgpack >= 0.4
    provider: python-msgpack.x86_64 0.4.6-1.el7
    dependency: python-psutil
    provider: python-psutil.x86_64 2.2.1-1.el7
    dependency: python-requests >= 1.0.0
    provider: python2-requests.noarch 2.20.1-2.el7
    provider: python-requests.noarch 2.6.0-1.el7_1
    dependency: python-six
    provider: python-six.noarch 1.9.0-2.el7
    dependency: python-tornado >= 4.2.1
    provider: python-tornado.x86_64 4.2.1-3.el7
    dependency: python-tornado < 6.0
    provider: python-tornado.x86_64 4.2.1-3.el7
    dependency: python-zmq
    provider: python-zmq.x86_64 15.3.0-3.el7
    dependency: systemd-python
    provider: systemd-python.x86_64 219-57.el7_5.3
    dependency: systemd-units
    provider: systemd.x86_64 219-57.el7_5.3
    dependency: which
    provider: which.x86_64 2.20-7.el7
    dependency: yum-utils
    provider: yum-utils.noarch 1.1.31-46.el7_5
    root@localhost:~# yum list python-futures
    Loaded plugins: fastestmirror, langpacks
    Loading mirror speeds from cached hostfile
  • base: mirrors.xmission.com
  • extras: mirrors.xmission.com
  • updates: mirrors.xmission.com
    Installed Packages
    python-futures.noarch 3.0.3-1.el7 @salt-2017.7
    root@localhost:# rpm -qi python-futures
    Name : python-futures
    Version : 3.0.3
    Release : 1.el7
    Architecture: noarch
    Install Date: Tue 13 Nov 2018 01:09:47 PM MST
    Group : Development/Libraries
    Size : 88366
    License : BSD
    Signature : RSA/SHA1, Wed 19 Sep 2018 03:48:25 PM MDT, Key ID 0e08a149de57bfbe
    Source RPM : python-futures-3.0.3-1.el7.src.rpm
    Build Date : Wed 19 Sep 2018 03:37:40 PM MDT
    Build Host : 7725ad0a8c1b450d8f81e3bc556f1b45
    Relocations : (not relocatable)
    URL : https://github.com/agronholm/pythonfutures
    Summary : Backport of the concurrent.futures package from Python 3.2
    Description :
    The concurrent.futures module provides a high-level interface for
    asynchronously executing callables.
    root@localhost:    #

@maltris
Copy link

maltris commented Nov 16, 2018

Not sure why this is closed, but I think it also applies to the Ubuntu 16.04 repo, which as of today, does not contain +ds-2 and does not have the correct dependencies set.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dmurphy18 dmurphy18

Labels
Packaging Platform TEAM Platform
Projects
None yet
Milestone
Approved
Development

No branches or pull requests
6 participants
@jryberg @sblaisot @lomeroe @dmurphy18 @maltris @arminbuerkle