pkg.latest completely broken on FreeBSD in salt-ssh 2016.3

[djneades]

Description

I am running salt-ssh on Mac OS X 10.11.5 against a FreeBSD 10.3-RELEASE-p3 host. pkg.latest appears to be completely broken on FreeBSD in the Salt 2016.3 release. For example, applying the following state:

curl: # ftp/curl
  pkg.latest: []

results in this stack trace:

----------
          ID: curl
    Function: pkg.latest
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/state.py", line 1703, in call
                  **cdata['kwargs'])
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/loader.py", line 1649, in wrapper
                  return f(*args, **kwargs)
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/states/pkg.py", line 1552, in latest
                  **kwargs)
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/modules/pkgng.py", line 281, in latest_version
                  if salt.utils.compare_versions(_get_pkgng_version(jail, chroot), '>=', '1.6.0'):
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/modules/pkgng.py", line 100, in _get_pkgng_version
                  return __salt__['cmd.run']([_pkg(jail, chroot), '--version']).strip()
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/modules/cmdmod.py", line 852, in run
                  bg=bg)
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/modules/cmdmod.py", line 471, in _run
                  proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs)
                File "/var/tmp/.djn_1fe71c_salt/py2/salt/utils/timed_subprocess.py", line 41, in __init__
                  self.process = subprocess.Popen(args, **kwargs)
                File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
                  errread, errwrite)
                File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
                  raise child_exception
              AttributeError: 'list' object has no attribute 'rfind'
     Started: 09:28:34.900231
    Duration: 24.996 ms
     Changes:
----------

Other pkg.latest states fail in a similar fashion. Using package origins (rather than names) does not make any difference. (pkg.installed appears to work, however.)

The states were working perfectly with Salt 2015.8.10. The only change is upgrading to Salt 2016.3.

Versions Report

Here is the versions report on the OS X machine on which I am running salt-ssh:

$ salt --versions-report
Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.11 (default, Mar 22 2016, 22:17:11)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist:
        machine: x86_64
        release: 15.5.0
         system: Darwin
        version: 10.11.5 x86_64

[Ch3LL]

@dneades looks like i'm able to replicate this on a freebsd10 box:

ch3ll-freebsd:
----------
          ID: curl
    Function: pkg.latest
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/var/tmp/.root_881c5e_salt/py2/salt/state.py", line 1703, in call
                  **cdata['kwargs'])
                File "/var/tmp/.root_881c5e_salt/py2/salt/loader.py", line 1649, in wrapper
                  return f(*args, **kwargs)
                File "/var/tmp/.root_881c5e_salt/py2/salt/states/pkg.py", line 1552, in latest
                  **kwargs)
                File "/var/tmp/.root_881c5e_salt/py2/salt/modules/pkgng.py", line 281, in latest_version
                  if salt.utils.compare_versions(_get_pkgng_version(jail, chroot), '>=', '1.6.0'):
                File "/var/tmp/.root_881c5e_salt/py2/salt/modules/pkgng.py", line 100, in _get_pkgng_version
                  return __salt__['cmd.run']([_pkg(jail, chroot), '--version']).strip()
                File "/var/tmp/.root_881c5e_salt/py2/salt/modules/cmdmod.py", line 852, in run
                  bg=bg)
                File "/var/tmp/.root_881c5e_salt/py2/salt/modules/cmdmod.py", line 471, in _run
                  proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs)
                File "/var/tmp/.root_881c5e_salt/py2/salt/utils/timed_subprocess.py", line 41, in __init__
                  self.process = subprocess.Popen(args, **kwargs)
                File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
                  errread, errwrite)
                File "/usr/local/lib/python2.7/subprocess.py", line 1335, in _execute_child
                  raise child_exception
              AttributeError: 'list' object has no attribute 'rfind'
     Started: 23:25:07.388126
    Duration: 45.832 ms
     Changes:   

To ensure this is only freebsd I was able to run this against a centos6 machine just fine. This is limited to freebsd.

[Ch3LL]

also verified this is not exclusive to salt-ssh. occurs on the minion when salt is installed on freebsd

[djneades]

@Ch3LL Thank you for confirming this issue and providing the additional information, that’s very helpful.

[Ch3LL]

@dneades no problem :) Looks like we definitely need to get this one fixed. Thanks for reporting it. much appreciated

[djneades]

@Ch3LL, yes I agree it needs to be fixed :-) I’ve changed all my pkg.latest rules to use pkg.installed as a temporary workaround, but that’s not ideal.

[amontalban]

@Ch3LL I think the problem was introduced in f4fb771 by @terminalmage.

Check this:

[INFO    ] Executing command ['pkg', 'info', '-ao'] in directory '/root'
[INFO    ] Executing command [['pkg'], '--version'] in directory '/root'

As you can see the first command looks good, however second is ['pkg'] and therefore it fails.

[terminalmage]

Fixed in #33648. You can work around this until we release 2016.3.1 by downloading this file and placing it into salt://_modules/, then running salt -G 'kernel:FreeBSD' saltutil.sync_modules to sync the patched file to your minions.

Once your FreeBSD minions have been updated to 2016.3.1 you can remove that file and re-run that sync_modules command to revert to using the copy of pkgng.py which is distributed with Salt.

[amontalban]

Hi @terminalmage,

Thank you very much for the prompt fix, I think it fixes the original issue. I was going with the same approach and I found this that I'm not sure if it is correct:

[root@devserver ~]# salt-call --local pkg.latest_version bash
[CRITICAL] Failed to load grains defined in grain file disks.disks in function <function disks at 0x80c3afe60>, error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/salt/loader.py", line 724, in grains
    ret = fun()
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 31, in disks
    return _freebsd_geom()
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 107, in _freebsd_geom
    parse_geom_attribs(device)
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 102, in parse_geom_attribs
    if tmp[_geomconsts.ROTATIONRATE] == 0:
KeyError: 'rotationrate'
[INFO    ] Executing command ['pkg', 'info', '-ao'] in directory '/root'
[INFO    ] Executing command ['pkg', '--version'] in directory '/root'
[INFO    ] Executing command ['pkg', 'search', '-S', 'name', '-Q', 'version', '-e', '-q', 'bash'] in directory '/root'
local:

As you can see this is returning empty string, however if I apply the following patch:

@@ -307,7 +308,7 @@ def latest_version(*names, **kwargs):
             if not installed:
                 ret[name] = pkgver
             else:
-                if not any(
+                if any(
                     (salt.utils.compare_versions(ver1=x,
                                                  oper='>=',
                                                  ver2=pkgver)

And execute same command:

[root@devserver ~]# salt-call --local pkg.latest_version bash
[CRITICAL] Failed to load grains defined in grain file disks.disks in function <function disks at 0x80c3afe60>, error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/salt/loader.py", line 724, in grains
    ret = fun()
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 31, in disks
    return _freebsd_geom()
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 107, in _freebsd_geom
    parse_geom_attribs(device)
  File "/usr/local/lib/python2.7/site-packages/salt/grains/disks.py", line 102, in parse_geom_attribs
    if tmp[_geomconsts.ROTATIONRATE] == 0:
KeyError: 'rotationrate'
[INFO    ] Executing command ['pkg', 'info', '-ao'] in directory '/root'
[INFO    ] Executing command ['pkg', '--version'] in directory '/root'
[INFO    ] Executing command ['pkg', 'search', '-S', 'name', '-Q', 'version', '-e', '-q', 'bash'] in directory '/root'
local:
    4.3.42_1

I'm seeing the same behaviour with aptpkg.py so not sure if I'm interpreting this right.

Thanks once again for the quick fix and help!

[amontalban]

@terminalmage please disregard my last message I read the docs and clearly says:

If the latest version of a given package is already installed, an empty string will be returned for that package.

@cedwards Can you add this patch to FreeBSD package as a workaround until 2016.3.1 is launched?

Thank you!

[amontalban]

Hi @terminalmage

I'm seeing some odd behaviour when running highstate on a second run on FreeBSD with the following state:

p5-MIME-Lite:
  pkg.latest

The first time it works ok but on subsequent runs I see this error:

[ERROR   ] LooseVersion instance has no attribute 'version'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/salt/utils/__init__.py", line 2294, in version_cmp
    distutils.version.LooseVersion(pkg2):
  File "/usr/local/lib/python2.7/distutils/version.py", line 300, in __cmp__
    return cmp(self.version, other.version)
AttributeError: LooseVersion instance has no attribute 'version'

Based on my prior comment #33529 (comment) this is caused because the returned version is empty and then https://github.com/saltstack/salt/blob/v2016.3.1/salt/states/pkg.py#L1597 fails because we try to do this in distutils/version.py with other being empty:

def __cmp__ (self, other):
    if isinstance(other, StringType):
        other = LooseVersion(other)

    return cmp(self.version, other.version)

What do you think?

Thanks!

[terminalmage]

I think that this traceback seems unrelated. Can you please open a new issue?

[terminalmage]

Actually, nevermind. I've already fixed this in #34429.

[amontalban]

Wooot! That was a super fast fix thanks a lot @terminalmage!