Returner configuration override options don't work for scheduled jobs (schedule module) #33868

abalashov · 2016-06-08T18:43:35Z

Description of Issue/Question

The SMTP returner works for a generic cmd.run invocation, but not for a scheduled job that wraps the same.

Setup

Just a stock master and a minion, no SLS files, pillars or environmental complexities. Basically just using Salt for remote mass-execution at this point.

Steps to Reproduce Issue

This works:

salt 'my-minion' cmd.run 'iostat' --return smtp --return_kwargs '{"to": "me@me.com", "from": "noc@me.com", "host": "10.150.20.2", "port": 25, "subject": "iostat output on my-minion"}'

It results in:

2016-06-08 14:35:56,298 [salt.minion                                          ][INFO    ][2500] User root Executing command cmd.run with jid 20160608143601572555
2016-06-08 14:35:56,313 [salt.minion                                          ][INFO    ][11096] Starting a new job with PID 11096
2016-06-08 14:35:56,315 [salt.loaded.int.module.cmdmod                        ][INFO    ][11096] Executing command 'iostat' in directory '/root'
2016-06-08 14:35:56,319 [salt.minion                                          ][INFO    ][11096] Returning information for job: 20160608143601572555

However, when I do the same using a one-time scheduled job ...

salt 'my-minion' schedule.add run_iostat function='cmd.run' job_args="['iostat']" returner=smtp return_kwargs='{"to": "me@me.com", "from": "noc@me.com", "host": "10.150.20.2", "subject": "iostat"}' once='2016-06-08T14:37:10'

It runs the job, and the output can be accessed, e.g. via salt-run jobs.lookup_jid, but blows up with an smtplib exception, per the minion log:

2016-06-08 14:37:10,186 [salt.utils.schedule                                  ][INFO    ][2500] Running scheduled job: run_iostat
2016-06-08 14:37:10,192 [salt.loaded.int.module.cmdmod                        ][INFO    ][11223] Executing command 'iostat' in directory '/root'
2016-06-08 14:37:10,205 [salt.utils.schedule                                  ][ERROR   ][11223] Unhandled exception running cmd.run
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/utils/schedule.py", line 737, in handle_func
    self.returners[ret_str](ret)
  File "/usr/lib/python2.6/site-packages/salt/returners/smtp_return.py", line 211, in returner
    server.sendmail(from_addr, to_addrs, message)
  File "/usr/lib64/python2.6/smtplib.py", line 692, in sendmail
    self.ehlo_or_helo_if_needed()
  File "/usr/lib64/python2.6/smtplib.py", line 516, in ehlo_or_helo_if_needed
    if not (200 <= self.ehlo()[0] <= 299):
  File "/usr/lib64/python2.6/smtplib.py", line 389, in ehlo
    self.putcmd(self.ehlo_msg, name or self.local_hostname)
  File "/usr/lib64/python2.6/smtplib.py", line 323, in putcmd
    self.send(str)
  File "/usr/lib64/python2.6/smtplib.py", line 315, in send
    raise SMTPServerDisconnected('please run connect() first')
SMTPServerDisconnected: please run connect() first
2016-06-08 14:37:10,208 [salt.minion                                          ][INFO    ][2500] Returning information for job: req

I should add that I ran a tcpdump on the SMTP server and saw no attempt to connect to 10.150.20.2 from the minion or the master in the second case.

Versions Report

Master:

[root@allegro-2 ~]# salt --versions-report
Salt Version:
           Salt: 2016.3.0-n/a-a8d9221

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.3.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 3.2.5

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.18.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

Minion:

[root@my-minion ~]# salt --versions-report
Salt Version:
           Salt: 2016.3.0-182-g94f98b4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.4.1
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.20.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.26.1.el6.x86_64
         system: Linux
        version: CentOS 6.7 Final

The text was updated successfully, but these errors were encountered:

abalashov · 2016-06-08T19:31:32Z

I added a bit of logging to /usr/lib/python2.6/site-packages/salt/returners/smtp_return.py and discovered that in the case of the manual cmd.run invocation...

2016-06-08 15:29:59,319 [salt.loaded.int.returner.smtp_return ][INFO ][15540] Connecting to server 10.150.20.2:25

Meanwhile, with the scheduled job:

2016-06-08 15:29:06,975 [salt.loaded.int.returner.smtp_return ][INFO ][15469] Connecting to server None:25

So, it looks like the host key in my return_kwargs to schedule.add isn't getting propagated down correctly.

Edit: actually, looks like none of the return_kwargs parameters are getting plumbed down correctly. I printed the message buffer in smtp_return.returner() and got:

Message: From: None
To: None
Date: Wed, 08 Jun 2016 15:34:14 -0400
Subject: Email from Salt

id: my-minion
function: cmd.run
function args: ['iostat']
jid: 20160608153414946310

abalashov · 2016-06-08T19:52:10Z

I think I'm getting closer to the root of the problem. return_kwargs is what's used everywhere else in the code base, but returner_kwargs is the key sought in utils/schedule.py:handle_func():

            if data_returner or self.schedule_returner:
                if 'returner_config' in data:
                    ret['ret_config'] = data['returner_config']
                if 'returner_kwargs' in data:
                    ret['ret_kwargs'] = data['returner_kwargs']

This has no presentation elsewhere in the codebase:

[root@209-133-220-127 salt]# pwd
/usr/lib/python2.6/site-packages/salt
[root@209-133-220-127 salt]# fgrep -HR returner_kwargs .
Binary file ./utils/schedule.pyc matches

abalashov · 2016-06-08T19:55:53Z

Yep, fixed by:

-                if 'returner_kwargs' in data:
-                    ret['ret_kwargs'] = data['returner_kwargs']
+                if 'return_kwargs' in data:
+                    ret['ret_kwargs'] = data['return_kwargs']

Will submit a PR.

abalashov · 2016-06-08T20:15:07Z

returner_config vs. return_config needs to be similarly changed.

…ation attributes 'return_config' and 'return_kwargs'. They are specified as 'return_config' and 'return_kwargs' in the documentation, and those are the member names used elsewhere in the code base. However, this code previously referenced 'returner_kwargs' and 'returner_config', so returners from scheduled jobs were not appropriately called in situations in which configuration parameters to override default minion config needed to be provided to the schedule module. Resolves issue saltstack#33868 (saltstack#33868).

Ch3LL · 2016-06-09T19:20:33Z

@abalashov thanks for the PR! Its always great to see a fix for an issue 👍 We will close this once the PR has been merged.

stale · 2018-05-20T19:43:46Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

abalashov changed the title ~~SMTP returner doesn't work for scheduled jobs~~ Returner configuration override options don't work for scheduled jobs (schedule module) Jun 8, 2016

abalashov mentioned this issue Jun 9, 2016

utils/schedule.py:handle_func() - Fix for accessing returner configur… #33912

Merged

Ch3LL added Bug broken, incorrect, or confusing behavior P3 Priority 3 Returners RIoT Relates to integration with cloud providers, hypervisors, API-based services, etc. severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around labels Jun 9, 2016

Ch3LL added this to the Approved milestone Jun 9, 2016

stale bot added the stale label May 20, 2018

stale bot closed this as completed May 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Returner configuration override options don't work for scheduled jobs (schedule module) #33868

Returner configuration override options don't work for scheduled jobs (schedule module) #33868

abalashov commented Jun 8, 2016 •

edited

Loading

abalashov commented Jun 8, 2016 •

edited

Loading

abalashov commented Jun 8, 2016

abalashov commented Jun 8, 2016

abalashov commented Jun 8, 2016

Ch3LL commented Jun 9, 2016

stale bot commented May 20, 2018

Returner configuration override options don't work for scheduled jobs (schedule module) #33868

Returner configuration override options don't work for scheduled jobs (schedule module) #33868

Comments

abalashov commented Jun 8, 2016 • edited Loading

Description of Issue/Question

Setup

Steps to Reproduce Issue

Versions Report

Master:

Minion:

abalashov commented Jun 8, 2016 • edited Loading

abalashov commented Jun 8, 2016

abalashov commented Jun 8, 2016

abalashov commented Jun 8, 2016

Ch3LL commented Jun 9, 2016

stale bot commented May 20, 2018

abalashov commented Jun 8, 2016 •

edited

Loading

abalashov commented Jun 8, 2016 •

edited

Loading