Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt-master scheduled items ignore maxrunning. #49957

Closed
whytewolf opened this issue Oct 9, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@whytewolf
Copy link
Contributor

commented Oct 9, 2018

Description of Issue/Question

scheduled items in the master config ignore the maxrunning setting. So can have more than a single running schedule item at a time.

Setup

setup a schedule on a master that ruins a task longer than the time it is scheduled for with a maxrunning setting, or using the default of 1. Wait and than parse ps for running tasks and see that more than one /usr/bin/salt-master Maintenance Schedule <jid> is running. setup setproctitle to know what processes are running.

Steps to Reproduce Issue

Here is what I used for my configuration and test.

[root@salt00 test_orch]# cat /srv/salt/test_orch/test.sls
test_wait:
  salt.function:
    - tgt: '*'
    - name: test.sleep
    - arg:
      - 300
[root@salt00 test_orch]# cat /etc/salt/master.d/schedule.conf
schedule:
  test_sch:
    function: state.orch
    minutes: 1
    args:
      - test_orch.test

And the result

root     23908     1 23908  1    1 20:38 ?        00:00:03 /usr/bin/python /usr/bin/salt-master Maintenance Schedule 20181009203836745635
root     23974     1 23974  0    1 20:38 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_return 20181009203837752085
root     24308     1 24308  1    1 20:39 ?        00:00:02 /usr/bin/python /usr/bin/salt-master Maintenance Schedule 20181009203938188486
root     24374     1 24374  0    1 20:39 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_return 20181009203939210343
root     24710     1 24710  1    1 20:40 ?        00:00:01 /usr/bin/python /usr/bin/salt-master Maintenance Schedule 20181009204039865523
root     24776     1 24776  0    1 20:40 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_return 20181009204040859635
root     25125     1 25125  1    1 20:41 ?        00:00:00 /usr/bin/python /usr/bin/salt-master Maintenance Schedule 20181009204141665150
root     25191     1 25191  0    1 20:41 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_return 20181009204142634640

(Include debug logs if possible and relevant.)

Versions Report

all versions tested

salt --versions
Salt Version:
           Salt: 2017.7.5

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core
Salt Version:
           Salt: 2017.7.7

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

@whytewolf whytewolf added the ZD label Oct 9, 2018

@whytewolf

This comment has been minimized.

Copy link
Contributor Author

commented Oct 9, 2018

zd-2847

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2018

looks like i'm able to replicate this on the head of 2018.3 as well.

root      2680  2.1  0.3 504272 63752 ?        S    14:51   0:01 python /testing/scripts/salt-master -d Maintenance Schedule 20181010145125892161
root      2703  0.0  0.3 575744 53268 ?        S    14:51   0:00 python /testing/scripts/salt-minion -d KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_return 20181010145134591130
root      2881  4.4  0.3 505544 64672 ?        S    14:52   0:00 python /testing/scripts/salt-master -d Maintenance Schedule 20181010145234502660
root      2928  0.2  0.3 575744 53124 ?        S    14:52   0:00 python /testing/scripts/salt-minion -d KeepAlive MultiMinionProcessManager MinionProcessManager Minion._thread_ret

and when i run salt-run jobs.active it shows multiple test.sleep commands running

ping @garethgreenaway any ideas here?

@Ch3LL Ch3LL added this to the Approved milestone Oct 10, 2018

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 10, 2018

@Ch3LL will take a look.

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

Closing this out, if the problem persists please comment & we'll re-open the issue or feel free to open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.