DISCARD without MULTI

[jkarneges]

Hey folks,

I came across a weird issue today where one of our processes started raising ResponseError: DISCARD without MULTI when calling execute() on a pipeline object. As far as I can tell, this error should be impossible to receive since redis-py is supposed to take care of any discard/unwatching when using pipeline objects. This suggests a bug in redis-py not properly cleaning up after itself, although I've been reading the code and I don't see yet how this is possible. I also don't know how to reproduce the bug.

ERROR 2016-10-29 04:34:32.807 error expiring
Traceback (most recent call last):
  File "dmserver.py", line 1477, in push_in_worker
    redis_ops.cq_expire_items(channel_key)
  File "redis_ops.py", line 129, in cq_expire_items
    pipe.execute()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2626, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2521, in _execute_transaction
    self.immediate_execute_command('DISCARD')
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2461, in immediate_execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2584, in parse_response
    self, connection, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 582, in read_response
    raise response
ResponseError: DISCARD without MULTI

I'm using Redis 3.0.6 and redis-py 2.10.5.

Anyone seen anything like this before and have any suggestions?

[andymccurdy]

Hey Justin, thanks for the report. I haven't heard of this happening before, and as you mentioned, I don't see how it's possible either.

Are you running in a threaded environment by chance? In case you weren't aware, it's not safe to share pipeline instances between threads.

[jkarneges]

I am using a threaded environment. The Redis object is shared, but the pipeline is not.

Here's a simplified version of my code:

import redis

g_redis = redis.Redis(host="localhost", port=6379)

def cq_expire_items(channel_key):
    r = g_redis
    while True:
        with r.pipeline() as pipe:
            try:
                # ... several calls to pipe.watch here ...
                # ... some read calls here: pipe.lindex, pipe.get ...
                # ... also some logic to break out (this loop repeats until satisfied) ...
                pipe.multi()
                # ... some write calls here: pipe.lpop, pipe.set, pipe.hdel, pipe.set, pipe.zadd ...
                pipe.execute()
                # note: don't break on success
            except redis.WatchError:
                continue

[andymccurdy]

Based on your stack track, Redis is returning an EXECABORT error. That typically means there was a syntactical error in one of the commands within the MULTI/EXEC block.

Is this error happening every time you run your code now? If so, it's almost certainly one of your commands.

If it's only happening some of the time, perhaps there's an issue with the data that's being sent (like a key name or value).

You could try running MONITOR within redis-cli while you run the Python code to figure out the command causing EXECABORT to be raised.

[jkarneges]

It's very rare. This code has been running for around a year and I've only seen this issue a few times, for just some mins at a time. Interestingly, when I saw it the other day it occurred on several of our server nodes within the span of a few hours.

I may have just figured it out. We're using maxmemory and likely hit the max, since we also saw a few MISCONF errors recently.
https://stackoverflow.com/questions/19581059/misconf-redis-is-configured-to-save-rdb-snapshots

Perhaps in an out-of-memory situation, non-pipelined commands give MISCONF, but pipelined commands cause EXECABORT? I'll see if I can reproduce this using a low maxmemory setting.

That said, regardless of how EXECABORT is occurring, probably redis-py shouldn't be trying to do a DISCARD.

[jkarneges]

Here we go. It's not maxmemory, but the inability to write to disk. I changed the permissions on disk to prevent Redis from being able to write and was able to reproduce MISCONF on individual commands. As predicted, this also causes pipelines to EXECABORT.

import redis

r = redis.Redis()

with r.pipeline() as pipe:
        pipe.watch('foo')
        pipe.multi()
        pipe.set('foo', 'bar')
        pipe.execute()

Traceback (most recent call last):
  File "discardwithoutmulti.py", line 9, in <module>
    pipe.execute()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2626, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2521, in _execute_transaction
    self.immediate_execute_command('DISCARD')
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2461, in immediate_execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2584, in parse_response
    self, connection, command_name, **options)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 582, in read_response
    raise response
redis.exceptions.ResponseError: DISCARD without MULTI

[dlcron]

I've also had this issue recently.

Let my summarise it just if anyone is also struggling.

You clearly cannot do the following:

> redis_db.sadd('123', *[])
*** redis.exceptions.ResponseError: wrong number of arguments for 'sadd' command

But following is allowed:

> pipe = redis_db.pipeline()
> pipe.multi()
> pipe.sadd('123', *[])
> pipe.execute()
*** redis.exceptions.ResponseError: DISCARD without MULTI

Long story short, if you are using pipeline, each of the command is queued. If one of the queued command results with an error, you might get DISCARD without MULTI. In my case arguments to sadd were computed dynamically and sometimes ended up as an empty list. This was clearly an issue in my application logic.

Since we keep the commands in the stack we could handle this specific case by introspecting the stack arguments and throw an error as soon as we add something there. In my case it is possible to know this sooner, without calling execute on pipeline.

> [..]/redis/client.py(2879)execute()
  return execute(conn, stack, raise_on_error)  # <---- we have queued commands in stack

What do you guys think about this?

[dlcron]

This would make sense as redis-cli throws any syntax errors as soon as they arise:

127.0.0.1:6379> MULTI
OK
127.0.0.1:6379> sadd 1
(error) ERR wrong number of arguments for 'sadd' command

[github-actions]

This issue is marked stale. It will be closed in 30 days if it is not updated.

[andymccurdy]

This was fixing in redis-py 3.5.0. Forgot to close this issue when the error was resolved.