Error when running salt-run jobs.lookup_jid with master_job_cache: pgjsonb

[PierreR]

Description of Issue/Question

When running salt-run jobs.lookup_jid 20170412195438001065, I have got this error:

[ 0 ] root # salt-run jobs.lookup_jid 20170412195438001065
[INFO    ] Using default for returner.pgjsonb port
Exception occurred in runner jobs.lookup_jid: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 395, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/runners/jobs.py", line 126, in lookup_jid
    display_progress=display_progress
  File "/usr/lib/python2.7/site-packages/salt/runners/jobs.py", line 196, in list_job
    ret.update(_format_jid_instance(jid, job))
  File "/usr/lib/python2.7/site-packages/salt/runners/jobs.py", line 561, in _format_jid_instance
    ret = _format_job_instance(job)
  File "/usr/lib/python2.7/site-packages/salt/runners/jobs.py", line 538, in _format_job_instance
    ret = {'Function': job.get('fun', 'unknown-function'),
AttributeError: 'str' object has no attribute 'get'
[INFO    ] Using default for returner.pgjsonb port
[INFO    ] Runner completed: 20170413093238280005

Setup

On a salt master, connected to remote minions with this config:

[ 0 ] root # cat /etc/salt/master
fileserver_backend:
  - roots

file_roots:
  base:
    - /srv/salt

log_level: info

master_job_cache: pgjsonb

returner.pgjsonb.host: xxx.prd.srv.lan
returner.pgjsonb.user: xxx
returner.pgjsonb.pass: xxxx
returner.pgjsonb.db: xxx
returner.pgjsonb.port: 5432

How to reproduce this error

I am not sure how to reproduce this error. I have a similar setting on another master and it works fine there (exactly the same salt report version, same postgresql, only the name of the database differs).

But what I do on the faulty server is the following:

1. salt --async '*' test.ping
2. taking the received jid let say xxx, do 'salt-run jobs.lookup_jid xxx'

Versions Report

Salt Version:
           Salt: 2016.11.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.10.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

[PierreR]

As a note, I have deleted all records in jids and salt_return in the postgresql database (returner) to be sure the error does not comes from a bogus 'pgjsonb' entry.

[gtmanfred]

Can you double check that you don't have any extra pgjsonb.py files in like /var/cache/salt/master/extmods/returners ?

I am unfortunately unable to replicate this issue.

[root@salt salt]# salt \* test.ping --async
Executed command with job ID: 20170413171056078391
[root@salt salt]# salt-run jobs.lookup_jid 20170413171056078391
salt.c.gtmanfred-1263.internal:
    True
[root@salt salt]# salt-run jobs.list_job 20170413171056078391
Arguments:
Function:
    test.ping
Result:
    ----------
    salt.c.gtmanfred-1263.internal:
        ----------
        _stamp:
            2017-04-13T17:10:56.167512
        cmd:
            _return
        fun:
            test.ping
        fun_args:
        id:
            salt.c.gtmanfred-1263.internal
        jid:
            20170413171056078391
        retcode:
            0
        return:
            True
        success:
            True
StartTime:
    2017, Apr 13 17:10:56.078391
Target:
    *
Target-type:
    glob
User:
    root
jid:
    20170413171056078391

If you can try cleaning salt off the server which is failing and reinstalling salt?

Thanks,
Daniel

[Poil]

Hi,

I've just tried this

curl -sSi http://salt.xxxx:8000  -b ~/cookies.txt -d fun='jobs.lookup_jid' -d jid='20180822171348900609' -d client='runner'

When the returner pgjsonb is enable I have the same traceback

{"return": ["Exception occurred in runner jobs.lookup_jid: Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/salt/client/mixins.py\", line 387, in _low\n    data['return'] = self.functions[fun](*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/salt/runners/jobs.py\", line 128, in lookup_jid\n    display_progress=display_progress\n  File \"/usr/lib/python2.7/site-packages/salt/runners/jobs.py\", line 199, in list_job\n    ret.update(_format_jid_instance(jid, job))\n  File \"/usr/lib/python2.7/site-packages/salt/runners/jobs.py\", line 573, in _format_jid_instance\n    ret = _format_job_instance(job)\n  File \"/usr/lib/python2.7/site-packages/salt/runners/jobs.py\", line 550, in _format_job_instance\n    ret = {'Function': job.get('fun', 'unknown-function'),\nAttributeError: 'str' object has no attribute 'get'\n"]}

When disabling the returner I have no more problem

Best regards,

# salt --versions-report
Salt Version:
           Salt: 2018.3.2
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.3
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core

Returner config

event_return:
  - pgjsonb

returner.pgjsonb.host: 'tlsslt-p01-pg.xxxxx'
returner.pgjsonb.user: 'salt'
returner.pgjsonb.pass: 'xxxxxxx'
returner.pgjsonb.db: 'salt'
returner.pgjsonb.port: 5432
master_job_cache: pgjsonb

[Poil]

Note we are both on CentOS 7, could be a bug with python 2.7.5 ?

I think you should do like Puppet, a package that include the full Python environment, no dependency with the Python of the OS, also Jinja on CentOS7 it's a old one (2.7.x), I've upgrade it to 2.10

[Poil]

It's working with package from http://repo.saltstack.com/py3/redhat/7/x86_64/latest/

[gtmanfred]

So, is this still an issue? or can it be closed?

[Poil]

Issue with Python 2.7.5

[gtmanfred]

I was unable to replicate it on centos 7 with python 2.7.5 here

#40673 (comment)

[Poil]

I confirm that when I switch back to python 2.7.5 it's no more working for me.
Perhaps it's a salt component that create this error ?

In my formula I use these resources, git.latest, file (managed, recurse, directory, serialize, replace, append, symlink), ini.options_present, ssh_known_hosts.present, cmd (run, script), service.running, pkgrepo.(managed, installed), module.run (cp.push), pkg.installed, user.present, archive.extracted

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.