Scheduled runners not executed (for proxy minions, at least) #36021
Assignees
Labels
Bug
broken, incorrect, or confusing behavior
P4
Priority 4
Proxy-Minion
RIoT
Relates to integration with cloud providers, hypervisors, API-based services, etc.
severity-medium
3rd level, incorrect or bad functionality, confusing and lacks a work around
Milestone
Comments
|
For the mines, looking in the proxy debug logs, seems that at 12:34 the proxy was started and mines refreshed, at 12:37 executed manually
root@36netops1:~# cat /etc/salt/pillar/mine.sls
mine_functions:
testmine.timestamp: []
mine_interval: 1
proxy:
proxytype: dummy |
|
Master debug logs for the scheduler: |
|
I have also a question - there might be an overlap between an older config and the newest changes from #34446 The following line says that the proc are stored under In cachedir: /var/cache/salt/proxyWhile in the # Directory to store job and cache data
cachedir: /var/cache/salt/proxy-masterLooks like the root@36netops1:~# ls -la /var/cache/salt/proxy-master/minions/mine/
total 108
drwxr-xr-x 2 root root 32 Sep 2 12:43 .
drwxr-xr-x 538 root root 16384 Sep 2 12:03 ..
-rw------- 1 root root 72394 Sep 2 12:43 data.p
-rw-r--r-- 1 root root 40 Sep 2 12:43 mine.pWhile module files for the same minion are under @cro would you recommend using the same directory? If so, which of the config files is the most suitable to be adjusted? Thank you! |
|
After upgrade: (virtualenv)root@36netops3:~# salt --versions-report
Salt Version:
Salt: 2016.3.3-48-gd248ab0
Dependency Versions:
cffi: 1.7.0
cherrypy: Not Installed
dateutil: Not Installed
gitdb: 0.6.4
gitpython: 1.0.2
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.9 (default, Mar 1 2015, 12:57:24)
python-gnupg: 0.3.8
PyYAML: 3.12
PyZMQ: 15.4.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.4.1
ZMQ: 4.1.5
System Versions:
dist: debian 8.2
machine: x86_64
release: 4.1.3-cloudflare
system: Linux
version: debian 8.2The error appears in the event bus: (virtualenv)root@36netops3:~# salt-run state.event pretty=True
/usr/local/salt/virtualenv/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.
salt/run/20160902143454211747/ret {
"_stamp": "2016-09-02T14:34:54.387877",
"fun": "runner.dummy.test",
"jid": "20160902143454211747",
"return": "Exception occurred in runner dummy.test: Traceback (most recent call last):\n File \"/usr/local/salt/virtualenv/lib/python2.7/site-packages/salt/client/mixins.py\", line 352, in low\n data['fun_args'] = args + ([kwargs] if kwargs else [])\nTypeError: can only concatenate tuple (not \"list\") to tuple\n",
"success": false,
"user": "root"
}
salt/run/20160902143554515090/ret {
"_stamp": "2016-09-02T14:35:54.674344",
"fun": "runner.dummy.test",
"jid": "20160902143554515090",
"return": "Exception occurred in runner dummy.test: Traceback (most recent call last):\n File \"/usr/local/salt/virtualenv/lib/python2.7/site-packages/salt/client/mixins.py\", line 352, in low\n data['fun_args'] = args + ([kwargs] if kwargs else [])\nTypeError: can only concatenate tuple (not \"list\") to tuple\n",
"success": false,
"user": "root"
}
salt/run/20160902143654774890/ret {
"_stamp": "2016-09-02T14:36:54.935636",
"fun": "runner.dummy.test",
"jid": "20160902143654774890",
"return": "Exception occurred in runner dummy.test: Traceback (most recent call last):\n File \"/usr/local/salt/virtualenv/lib/python2.7/site-packages/salt/client/mixins.py\", line 352, in low\n data['fun_args'] = args + ([kwargs] if kwargs else [])\nTypeError: can only concatenate tuple (not \"list\") to tuple\n",
"success": false,
"user": "root"
}Being related to https://github.com/saltstack/salt/blob/2016.3/salt/client/mixins.py#L352 data['fun_args'] = args + ([kwargs] if kwargs else []) |
|
Changing the line in cause to: data['fun_args'] = list(args) + ([kwargs] if kwargs else [])Don't see the error anymore, but not sure the datatype should be (virtualenv)root@36netops3:~# salt-run state.event pretty=True
/usr/local/salt/virtualenv/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.
salt/auth {
"_stamp": "2016-09-02T14:48:08.013579",
"act": "accept",
"id": "mine",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6861fSbTHlhKMKuDWgNL\ncD8cp+cgN6zPk2K1CfSR63xxdWRMe1Aiy5xqpaX5+FhznZYLBj1gAwwFLs5XaH7A\n7jghEVR+cd31Jz+i2+VjYpNNrJ8if9ZZpaxBM3LCyVsXhFtcFX7IqdFSESuQfAuL\nFh6dMjpjwwRk8g3j3Dq4D7L1/NNDtQtTSFUI+oaFStbeanWN2QapP2dpYxopTwXT\nc02qS+U24DewUGGNVw3BNwj7jCetn7R/m92KoKjlDRMsrZYTNmDYsApwdU/+PX/P\n24774Du7SVXT9oHFlVilzJkJie0PnJT9aKKCKbSSFtYi3w+wNnNEsFncTmiYP1Er\niwIDAQAB\n-----END PUBLIC KEY-----",
"result": true
}
salt/run/20160902144837714970/new {
"_stamp": "2016-09-02T14:48:37.880787",
"fun": "runner.dummy.test",
"fun_args": [],
"jid": "20160902144837714970",
"user": "root"
}
salt/run/20160902144837714970/ret {
"_stamp": "2016-09-02T14:48:37.881189",
"fun": "runner.dummy.test",
"fun_args": [],
"jid": "20160902144837714970",
"return": true,
"success": true,
"user": "root"
} |
mirceaulinic
added a commit
to cloudflare/salt
that referenced
this issue
Sep 2, 2016
|
@mirceaulinic thanks for the great detailed report. Also thanks for the PR! ping @cro can you answer his questions regarding what datatype this should be? I believe the concern is to also ensure it does not regress other parts of salt outside of the proxy minion. |
Ch3LL
added
Bug
Ch3LL
added
the
severity-medium
|
Hi @Ch3LL as you pointed out, I want to make sure that with the change I propose I won't broke other things. |
|
FYI: seems to be a general regression on at least master-side schedule handling (from 2016.3.3 release) |
|
#36025 has been merged - next week will try the code from the 2016.3.3 branch and if everything is fine we can close the issue. |
|
great thanks for all your work on that one @mirceaulinic 👍 let us know how the tests go |
|
@Ch3LL seems to work fine so far - can we close this issue, or are there any other changes needed to be applied related to this? |
|
Closing now Thanks for testing! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of Issue/Question
After upgrading to Salt 2016.3.2, as per #35443 to test the code from #35178 I have noticed that the mines are not refreshed again - similarly to #32022. This time looks like the scheduling process that should refresh the mines is not executed.
Using the same test proxy config as described in #32022, section Setup:
We also have a couple of runners scheduled to be run at specific time intervals, e.g. schedule part from master config file:
They don't seem to be executed either.
Setup
For the mine refresh process, please see #32022.
To see if the runners are executed as scheduled in the master config file, a dummy runner is enough:
under
_runnersdirectory:dummy.py:
Scheduled in the master to be executed every minute:
Restart salt-master.
Steps to Reproduce Issue
In the master, looks that the error below is relevant:
Versions Report
The text was updated successfully, but these errors were encountered: