2016.3.4 redis_returner causes Maintenance tasks to spit errors

[pcn]

Description of Issue/Question

Using the redis returner as the default job cache, my master log as a lot of these:

2016-12-07 22:16:20,039 [salt.utils.process][ERROR   ] An un-handled exception from the multiprocessing process 'Maintenance-20' was caught:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/process.py", line 623, in _run
    return self._original_run()
  File "/usr/lib/python2.7/dist-packages/salt/master.py", line 237, in run
    salt.daemons.masterapi.clean_old_jobs(self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/daemons/masterapi.py", line 187, in clean_old_jobs
    mminion.returners[fstr]()
  File "/usr/lib/python2.7/dist-packages/salt/returners/redis_fix.py", line 297, in clean_old_jobs
    serv.delete(*to_remove)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 421, in delete
    return self.execute_command('DEL', *names)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 361, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 371, in parse_response
    response = connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 311, in read_response
    raise response
ResponseError: wrong number of arguments for 'del' command

Versions Report

$ salt --versions-report
Salt Version:
           Salt: 2016.3.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 2.4.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-100-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Minions are 2015.8.7 in case that's a likely cause.

[Ch3LL]

@pcn are you using custom code?

I don't see this file in the code: /usr/lib/python2.7/dist-packages/salt/returners/redis_fix.py

[brejoc]

I'm seeing the same error with the redis master job cache. Addition to the masters config looks like this:

master_job_cache: redis
redis.db: '0'
redis.host: '<redis_hostname>'
redis.port: 6379

$ salt --versions-report
Salt Version:
           Salt: 2016.11.1
 
Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: debian 8.7 
        machine: x86_64
        release: 4.4.0-59-generic
         system: Linux
        version: debian 8.7 

The python-redis has been installed via default packages:

$ dpkg -l | grep redis
ii  libhiredis0.10:amd64       0.11.0-4                  amd64        minimalistic C client library for Redis
ii  python-hiredis             0.1.4-1                   amd64        redis protocol reader for Python using hiredis
ii  python-redis               2.10.1-1                  all          Persistent key-value database with network interface (Python library)

Here is the error message:

2017-01-27 11:29:08,640 [salt.utils.process][ERROR   ][14998] An un-handled exception from the multiprocessing process 'Maintenance-17' was caught:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/process.py", line 644, in _run
    return self._original_run()
  File "/usr/lib/python2.7/dist-packages/salt/master.py", line 236, in run
    salt.daemons.masterapi.clean_old_jobs(self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/daemons/masterapi.py", line 193, in clean_old_jobs
    mminion.returners[fstr]()
  File "/usr/lib/python2.7/dist-packages/salt/returners/redis_return.py", line 230, in clean_old_jobs
    serv.delete(*to_remove)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 768, in delete
    return self.execute_command('DEL', *names)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 529, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 541, in parse_response
    response = connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 550, in read_response
    raise response
ResponseError: wrong number of arguments for 'del' command

[brejoc]

This also happens with the latest version of python-redis installed via pip: redis==2.10.5

[brejoc]

I also updated redis to version 3.2.6 to make sure there is no API incompatibility between python-redis and redis. Made no difference.

[brejoc]

Think this issue might be related to this redis/redis-py#513. When clean_old_jobs() is executed and no old jobs are there to be cleaned, an empty list is passed to serv.delete(). This is then causing ResponseError: wrong number of arguments for 'del' command. Perhaps we should just check if the list to_remove is empty before invoking serv.delete().

[brejoc]

I've just discovered, that this has been fixed in 54325cf and should be merged to 2016.11 in GH-38610. I'm using 2016.11.1 with this installation. Shouldn't this already be in this release?

[brejoc]

Seems like this has landet in Release v2016.11.2. I'd suggest, that this issue is beeing closed.

[Ch3LL]

Perfect. Thanks for all your investigation on this one closing now

[pcn]

Will this be backported to 2016.3.X?

[Ch3LL]

@pcn just requested that its backported to 2016.3

[pcn]

It looks like that's done in #39299 ?

[Ch3LL]

yep :)