/usr/bin/salt-call network.interfaces not working in v.17 like it did in v.16.4 in a cronjob

[shantanub]

This is a really weird problem and something I've usually only seen with cronjobs when an interactive shell is required but unavailable due to the nature of cron.

I have the following call that I execute as a part of a script that sits in a cronjob run by root:

/usr/bin/salt-call network.interfaces --output json --log-level=quiet --local

This call produces valid output when called interactively in a normal shell as root but produces nothing when the same call is made from within a cronjob.

non-interactive cronjob output:

*/6 * * * * /usr/bin/salt-call network.interfaces --output json --log-level=quiet --local > /tmp/salty.log


# cat /tmp/salty.log 
{
    "local": {}
}

interactive shell output:
# /usr/bin/salt-call network.interfaces --output json --log-level=quiet --local
{
    "local": {
        "lo": {
            "hwaddr": "00:00:00:00:00:00",
            "up": true,
            "inet": [
                {  
                    "broadcast": null,
                    "netmask": "255.0.0.0",
                    "label": "lo",
                    "address": "127.0.0.1"
                }  
            ],     
            "inet6": [
                {  
                    "prefixlen": "128",
                    "address": "::1"
                }  
            ]      
        },         
        "em2": {   
            "hwaddr": "78:2b:cb:50:b3:3b",
            "up": false
        },         
        "em4": {   
            "hwaddr": "78:2b:cb:50:b3:3f",
            "up": false
        },         
        "em1": {   
            "hwaddr": "78:2b:cb:50:b3:39",
            "up": false
        },         
        "em3": {   
            "hwaddr": "78:2b:cb:50:b3:3d",
            "up": false
        },         
<snip> ips / bonds removed </snip>
    }              
}

I haven't made any substantive changes to this shell script or call but this script broke upon upgrading to v.17. Is this a bug? Or is there some other way I can poll and pull networking information out? I need mac-addresses, interface names (virtual and physical) and IPs.

Thanks.

[shantanub]

On a related note, fetching the ip_interfaces grain also does not appear to work from within a cronjob:

/usr/bin/salt-call grains.get ip_interfaces --output json --log-level=quiet --local

cronjob:

*/6 * * * * /usr/bin/salt-call grains.get ip_interfaces --output json --log-level=quiet --local > /tmp/toosalty.log

recorded output from cronjob:

# cat /tmp/toosalty.log 
{
    "local": {}
}

output when not called from cronjob:

# /usr/bin/salt-call grains.get ip_interfaces --output json --log-level=quiet --local

{
    "local": {
        "p4p1": [], 
        "p4p2": [], 
        "bond0.<abc>": [
            "<ip removed>"
        ], 
        "bond0": [], 
        "lo": [
            "127.0.0.1"
        ], 
        "p2p1": [], 
        "p1p1": [], 
        "bond0.<xyz>": [
            "<ip removed>"
        ], 
        "p1p2": [], 
        "em4": [], 
        "em1": [], 
        "em3": [], 
        "em2": [], 
        "p2p2": []
    }
}

This is specific to this grain and function. I'm able to fetch any other system grain (or grain I create) without issue. Is this due to the dynamic nature of interface handling?

[shantanub]

One more finding... I'm able to get my scripts to run properly when using salt's internal scheduler (both "grains.get ip_interfaces" and "network.interfaces" return the appropriate network configuration output ). I'm guessing that this is the preferred approach to running these types of scripts over using cron?

Is support for cron (external schedulers) being deprecated?

[basepi]

Are you using 0.17.1 or 0.17.0? This may be related to a change we made for 0.17.1.

[shantanub]

This stopped working in v.17.0. I just upgraded to v.17.1 (I have another thread started with the problems I've encountered there). Most of my minions are still running v.17.0 but the master and 3 of minions are now on v.17.1.

The behavior is exactly the same with regards to this problem with both versions at least as far as I can tell.

[basepi]

Thanks for the update. We'll get this fixed.

[terminalmage]

I just tested this on a debian wheezy minion and was not able to reproduce the problem. I tested both with 0.17.1 and the latest from git.

What OS and version is your minion running?

[shantanub]

@terminalmage: This is definitely not working on both my 17.0 and 17.1 minions. My minions are all either RHEL5 or RHEL6.

Versions reports:

[root@salt-master ~]# salt --versions-report
           Salt: 0.17.1
         Python: 2.6.6 (r266:84292, May 27 2013, 05:35:12)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
         PyYAML: 3.10
          PyZMQ: 2.2.0.1
            ZMQ: 3.2.4

[root@salt-minion-00 ~]# salt-call --versions-report
           Salt: 0.17.1
         Python: 2.6.6 (r266:84292, May 27 2013, 05:35:12)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
         PyYAML: 3.10
          PyZMQ: 2.2.0.1
            ZMQ: 3.2.4

[root@salt-minion-01 ~]# salt-call --versions-report
           Salt: 0.17.1
         Python: 2.6.8 (unknown, Nov  7 2012, 14:47:45)
         Jinja2: unknown
       M2Crypto: 0.21.1
 msgpack-python: 0.1.12
   msgpack-pure: Not Installed
       pycrypto: 2.3
         PyYAML: 3.08
          PyZMQ: 2.1.9
            ZMQ: 2.2.0

[root@salt-master ~]# salt -v "test*" cmd.run "salt-call --versions-report"
Executing job with jid 20131101104101341413
-------------------------------------------

test:
               Salt: 0.17.0
             Python: 2.6.8 (unknown, Nov  7 2012, 14:47:45)
             Jinja2: unknown
           M2Crypto: 0.21.1
     msgpack-python: 0.1.12
       msgpack-pure: Not Installed
           pycrypto: 2.3
             PyYAML: 3.08
              PyZMQ: 2.1.9
                ZMQ: 2.2.0

OSs:

[root@salt-master ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.4 (Santiago)
[root@salt-master ~]# uname -a
Linux salt-master 2.6.32-358.23.2.el6.x86_64 #1 SMP Sat Sep 14 05:32:37 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@salt-master ~]# salt -v "salt-minion*" cmd.run "cat /etc/redhat-release"
Executing job with jid 20131101103754461004
-------------------------------------------

salt-minion-01:
    Red Hat Enterprise Linux Server release 5.10 (Tikanga)
salt-minion-00:
    Red Hat Enterprise Linux Server release 6.4 (Santiago)

[root@salt-master ~]# salt -v "salt-minion*" cmd.run "uname -a"
Executing job with jid 20131101103825319367
-------------------------------------------

salt-minion-01:
    Linux salt-minion-01 2.6.18-371.1.2.el5 #1 SMP Mon Oct 7 16:34:35 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
salt-minion-00:
    Linux salt-minion-00 2.6.32-358.23.2.el6.x86_64 #1 SMP Sat Sep 14 05:32:37 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@salt-master ~]# salt -v "test" cmd.run "uname -a"
Executing job with jid 20131101104148102150
-------------------------------------------

test:
    Linux test 2.6.18-371.el5 #1 SMP Thu Sep 5 21:21:44 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@salt-master ~]# salt -v "test" cmd.run "cat /etc/redhat-release"
Executing job with jid 20131101104209967583
-------------------------------------------

test:
    Red Hat Enterprise Linux Server release 5.10 (Tikanga)

I have this working with salt's internal scheduler now instead.

[terminalmage]

@shantanub What shell does the root user use on your master? I'm still unable to reproduce this and want to try to more closely match your setup

[shantanub]

@terminalmage bash. We have a pretty plain vanilla RHEL5/6 setup outside of bonding/vlan tagging.

You can't replicate this with rhel5 or 6? I just upgraded to salt v.17.2 and have the same issue with network.interfaces lookups (and grains.get ip_interfaces) via cron.

This works perfectly fine with salt's built in scheduler.

Why do you think the shell might be a factor? What's fundamentally different when I use salt's scheduler vs. cron when I make the same dump of networking info? And maybe just as importantly, what's changed? This can't just be an artifact of cron if I had working code that no longer works with minion version upgrades.

I've been able to move on since I've moved the reporting info into a state that's called hourly through highstate (via pillar and salt's internal scheduler), though this is pretty weird.

[terminalmage]

I was just looking for other possible differences between your setup and mine. I'll continue to try to replicate this, thanks.

[terminalmage]

OK, I was able to reproduce this on CentOS 5. The issue lies in the helper function salt.utils.which(), I think. The default PATH in Cent5 is /usr/bin:/bin, and the function that gets interface info invokes salt.utils.which() to see if the ip command (from iproute2) is available, and failing that, if ifconfig is available. These both happen to reside in /sbin, which is not in the default PATH. salt.utils.which() will try a "default" path (based off of the PATH used in busybox), but only if there is no PATH environment variable found. The solution, it seems, will be to use that "default" PATH as a fallback if the target binary is not found in PATH discovered by checking os.environ.

Working on this and will test and let you know what I find.

[terminalmage]

@shantanub I submitted a fix for this in #9148, but this may not actually be merged, because there's an argument to be made for the user keeping track of their own PATH.

Can you try specifying a PATH in your crontab?

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
* * * * *      /usr/bin/salt-call network.interfaces --output json --log-level=quiet --local > /tmp/salty.log

[shantanub]

@terminalmage Looks like the path was it. Thanks. I've always specified everything in our cronjobs with absolute paths so I didn't notice this with my other jobs.

[DmitryKuzmenko]

Removed the extra logic from which(). It confuses salt and users. If it's needed to run salt from crontab there are two cases. The first one proposed by @terminalmage above. It specifies PATH for all cron tasks. The second one specifies PATH for the only needed task (salt):

* * * * *      PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin /usr/bin/salt-call network.interfaces --output json --log-level=quiet --local > /tmp/salty.log