Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scheduled job runs in endless loop when using when: and splay: together. #50162

Closed
rbthomp opened this issue Oct 22, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@rbthomp
Copy link
Contributor

commented Oct 22, 2018

Description of Issue/Question

When defining a scheduled job in a pillar and specifying when: and splay: options together the schedule job runs in an endless loop.

Setup

Pillar:

schedule:
    test_schedule:
      function: test.ping
      returner: rawfile_json
      when:
        - 6:00am
      splay: 300
      return_kwargs:
        filename: /var/log/salt/pkg_list_test

Steps to Reproduce Issue

Create scheduled job using pillar.
Commenting the pillar to any of the following will result in normal behavior.

Comment out splay:

schedule:
    test_schedule:
      function: test.ping
      returner: rawfile_json
      when:
        - 6:00am
     # splay: 300
      return_kwargs:
        filename: /var/log/salt/pkg_list_test

Change schedule to seconds from when:

schedule:
    test_schedule:
      function: test.ping
      returner: rawfile_json
      seconds: 120
 #    when:
 #       - 6:00am
      splay: 5
      return_kwargs:
        filename: /var/log/salt/pkg_list_test

Versions Report

Salt Version:
Salt: 2018.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.28.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.5.1804 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.14.4.el7.x86_64
system: Linux
version: CentOS Linux 7.5.1804 Core

Python-dateutil:
version: 1.5-7.el7

Debug Log

DEBUG ] LazyLoaded test.ping
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.32.4.180:4506', u'aes')
[DEBUG ] Re-using AsyncAuth for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.32.4.180:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.32.4.180:4506
[DEBUG ] Trying to connect to: tcp://10.32.4.180:4506
[DEBUG ] LazyLoaded config.merge
[DEBUG ] LazyLoaded afcu_diskusage.beacon
[INFO ] Running scheduled job: test_schedule (runtime missed by 42224 seconds)
[DEBUG ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG ] schedule: This job was scheduled with a max number of 1
[INFO ] Running scheduled job: pkg_list_upgrades (runtime missed by 42224 seconds)
[DEBUG ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG ] schedule: This job was scheduled with a max number of 1
[DEBUG ] Job: pkg_list_upgrades is disabled
[DEBUG ] schedule.handle_func: adding this job to the jobcache with data {u'fun_args': [], u'jid': u'20181022174344453758', u'schedule': 'test_schedule', u'pid': 32631, u'fun': 'test.ping', u'id': u'LXOT-SALTTEST.afcucorp.test'}
[DEBUG ] LazyLoaded test.ping
[DEBUG ] test.ping received for minion 'LXOT-SALTTEST.afcucorp.test'
[DEBUG ] LazyLoaded rawfile_json.returner
[DEBUG ] LazyLoaded config.option
[DEBUG ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_a4fc0abfea_pub.ipc
[DEBUG ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_a4fc0abfea_pull.ipc
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/minion/minion_event_a4fc0abfea_pull.ipc
[DEBUG ] Sending event: tag = __schedule_return; data = {u'fun_args': [], u'jid': u'req', u'return': True, u'retcode': 0, u'success': True, u'schedule': 'test_schedule', u'cmd': u'_return', u'pid': 32631, u'_stamp': '2018-10-22T23:43:44.477770', u'ret_kwargs': {'filename': '/var/log/salt/pkg_list_test'}, u'fun': 'test.ping', u'id': u'LXOT-SALTTEST.afcucorp.test'}
[DEBUG ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20181022174344453758
[DEBUG ] Minion of 'lxo-salt01.afcucorp.local' is handling event tag '__schedule_return'
[INFO ] Returning information for job: req
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.6.8.128:4506', u'aes')
[DEBUG ] Re-using AsyncAuth for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.6.8.128:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.6.8.128:4506
[DEBUG ] Trying to connect to: tcp://10.6.8.128:4506
[DEBUG ] Minion of 'lxs-salt01.afcucorp.local' is handling event tag '__schedule_return'
[INFO ] Returning information for job: req
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.32.4.180:4506', u'aes')
[DEBUG ] Re-using AsyncAuth for (u'/etc/salt/pki/minion', u'LXOT-SALTTEST.afcucorp.test', u'tcp://10.32.4.180:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.32.4.180:4506
[DEBUG ] Trying to connect to: tcp://10.32.4.180:4506
[INFO ] Running scheduled job: test_schedule (runtime missed by 42225 seconds)
[DEBUG ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG ] schedule: This job was scheduled with a max number of 1

@garethgreenaway garethgreenaway self-assigned this Oct 23, 2018

@garethgreenaway garethgreenaway added this to the Approved milestone Oct 23, 2018

@rbthomp rbthomp changed the title Scheduled job runs in endless loop when using when:, splay:, and return_kwargs. Scheduled job runs in endless loop when using when: and splay: together. Oct 23, 2018

@rbthomp

This comment has been minimized.

Copy link
Contributor Author

commented Oct 23, 2018

I've confirmed that this is happening when splay: and when: are used in conjunction. Changing the return_kwargs was only changing the output to default to /var/log/salt/events from my /var/log/salt/pkg_list_test which let me to believe it was a factor.

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

@rbthomp Thanks for the update. Looking into this now.

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

@rbthomp So far I haven't been able to reproduce this and I'm not seeing an endless loop. That being said, it does appear that splay + when it not working as expected.

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

Was finally able to reproduce this one. Seems related to specifying when as a list and using in combination with splay.

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

Closing this out, if the problem persists please comment & we'll re-open the issue or feel free to open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.