New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calling a task returns a TypeError in 3.1.19 with redis as a broker. #2903

Closed
ryechus opened this Issue Nov 2, 2015 · 23 comments

Comments

Projects
None yet
@ryechus

ryechus commented Nov 2, 2015

When I try to call a task like the add task from the documentation I get the following traceback.

Traceback (most recent call last):
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/app/trace.py", line 283, in trace_task
    uuid, retval, SUCCESS, request=task_request,
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/backends/base.py", line 257, in store_result
    request=request, **kwargs)
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/backends/base.py", line 491, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/backends/redis.py", line 160, in set
    return self.ensure(self._set, (key, value), **retry_policy)
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/backends/redis.py", line 149, in ensure
    **retry_policy
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/kombu/utils/__init__.py", line 243, in retry_over_time
    return fun(*args, **kwargs)
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/celery/backends/redis.py", line 169, in _set
    pipe.execute()
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/redis/client.py", line 2572, in execute
    self.shard_hint)
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/redis/connection.py", line 874, in get_connection
    connection = self.make_connection()
  File "/Users/michael/.virtualenvs/wavcrate2/lib/python2.7/site-packages/redis/connection.py", line 883, in make_connection
    return self.connection_class(**self.connection_kwargs)
TypeError: __init__() got an unexpected keyword argument 'socket_connect_timeout'

This just started happening when I upgraded to 3.1.19 from 3.1.18. The issue is fixed if I rollback to 3.1.18.

@msull

This comment has been minimized.

msull commented Nov 3, 2015

Having the same issue -- this occurs for me when running with a connection to a redis server with a unix socket. The problem seems to stem from a recent change here:

https://github.com/celery/celery/blob/master/celery/backends/redis.py#L243

socket_connect_timeout is now always passed to ConnectionPool, and the underlying redis connection. When running redis with a unix socket, that underlying connection is redis.connection.UnixDomainSocketConnection which does not define the socket_connect_timeout argument.

@alexhayes

This comment has been minimized.

alexhayes commented Nov 10, 2015

I can also confirm this is happening on 3.1.19 however I'm using RabbitMQ as broker and Redis just as a result backend with a socket.

@rjagadeeswaran

This comment has been minimized.

rjagadeeswaran commented Nov 11, 2015

Any Updates on this ? I have the same config as you guys . Redis backend .

UPDATE:
Installing the redis bundle that comes with celery solved the issue for me !! http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html

mlevogiannis added a commit to mlevogiannis/demos-voting that referenced this issue Nov 24, 2015

settings.py,__init__.py: update celery configuration
- register 'custom-json' as kombu serializer
- set 'custom-json' as the default celery serializer
- fix the result backend configuration
- import settings from django.conf (it allows using different settings modules)
- celery 3.1.19 is NOT supported due to a bug in the redis backend (only affects UNIX-domain sockets, see: celery/celery#2903)

chriskuehl added a commit to ocf/ocfweb that referenced this issue Dec 7, 2015

@BrunoDesthuilliers

This comment has been minimized.

BrunoDesthuilliers commented Dec 8, 2015

WorksForMe with redis==2.10.3 (as a backend at least)

@ask

This comment has been minimized.

Member

ask commented Dec 8, 2015

It looks like the socket_connect_timeout was first introduced in redis 2.10.0

ask added a commit that referenced this issue Dec 8, 2015

@ask ask closed this Dec 11, 2015

kevcampb added a commit to kevcampb/celery that referenced this issue Dec 14, 2015

Fixing outstanding issue for celery#2903
The change to fix redis was misnamed. Changing to the correct parameter name
seems to make the fix work.
@chisleu

This comment has been minimized.

chisleu commented Jan 12, 2016

Is kevcampb's patch still going to be merged? The issue is still in 3.1.19.

I'm downgrading to 3.1.17 for now. (edit: done, and working now, although my issue #2995 is present in 3.1.17 as well w/ redis server 2.8.18)

@ask

This comment has been minimized.

Member

ask commented Jan 12, 2016

It's fixed in the 3.1 branch, just a matter of releasing a new version

xrg added a commit to xrg/demos that referenced this issue Feb 11, 2016

settings.py,__init__.py: update celery configuration
- register 'custom-json' as kombu serializer
- set 'custom-json' as the default celery serializer
- fix the result backend configuration
- import settings from django.conf (it allows using different settings modules)
- celery 3.1.19 is NOT supported due to a bug in the redis backend (only affects UNIX-domain sockets, see: celery/celery#2903)
@ryechus

This comment has been minimized.

ryechus commented Mar 7, 2016

Hi @ask,
I am still having the problem using celery 3.1.21, redis 3.0.0, and redis(python) 2.10.5. The traceback is identical to the first one I posted except the line numbers have changed.

@lddubeau

This comment has been minimized.

lddubeau commented Apr 7, 2016

This problem is still very much alive with the following combination of packages:

  • redis-py 2.10.5
  • celery 3.1.23
  • The redis server is 3.0.6

@msull put the finger on the immediate cause of the exception: redis.connection.UnixDomainSocketConnection does not allow socket_connect_timeout. The problem is that the method _create_client() always sets socket_connect_timeout even if has not been specified anywhere. If the code that calls it, leave it unspecified, then this method will set socket_connect_timeout to None but setting it to None is not the same thing as not setting it at all.

I've modified _create_client to this, and no longer get an error:

    def _create_client(self, **params):
        def normalize(param):
            value = params.get(param)
            if value is not None:
                params[param] = float(value)

        normalize('socket_timeout')
        normalize('socket_connect_timeout')
        return self._new_redis_client(**params)

(The above code has not been thoroughly tested.)

@ryechus

This comment has been minimized.

ryechus commented Apr 8, 2016

Haven't tried your patch but I can also confirm that none of the above fixes have made this problem go away for me either. I have been quietly using celery 3.1.18. Can we open this ticket again?

-Mike

On Apr 7, 2016, at 10:21 AM, Louis-Dominique Dubeau notifications@github.com wrote:

This problem is still very much alive with the following combination of packages:

redis-py 2.10.5
celery 3.1.23
The redis server is 3.0.6
@msull put the finger on the immediate cause of the exception: redis.connection.UnixDomainSocketConnection does not allow socket_connect_timeout. The problem is that the method _create_client() always sets socket_connect_timeout even if has not been specified anywhere. If the code that calls it, leave it unspecified, then this method will set socket_connect_timeout to None but setting it to None is not the same thing as not setting it at all.

I've modified _create_client to this, and no longer get an error:

def _create_client(self, **params):
    def normalize(param):
        value = params.get(param)
        if value is not None:
            params[param] = float(value)

    normalize('socket_timeout')
    normalize('socket_connect_timeout')
    return self._new_redis_client(**params)

(The above code has not been thoroughly tested.)


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

@chaoliu1024

This comment has been minimized.

chaoliu1024 commented Apr 14, 2016

  • celery 3.1.18
  • redis-py 2.10.5
  • redis server 3.0.5 (as a broken)

and also have this bug.

@luk

This comment has been minimized.

luk commented Apr 15, 2016

Hi,
I experienced a similar error, however in this case, the issue was kombu (check your stacktrace). I fixed it by pinning it to 3.0.34. There is also a bug at kombu: celery/kombu#576

These are the packages, that i use:

  • celery==3.1.18
  • django-celery==3.1.17
  • django-redis==4.4.1
  • kombu==3.0.34
  • redis==2.10.5

Best,
Lukas

@ask

This comment has been minimized.

Member

ask commented Apr 15, 2016

python redis 2.10.5 supports the argument, so I have no idea why you would receive this error if using that version. Using the latest kombu version and both (celery 3.1.18 / 3.1.23) sending tasks works for me with redis 2.10.5.

3.1.18 is not the latest celery, and if you upgrade kombu you should also upgrade celery.

@lddubeau

This comment has been minimized.

lddubeau commented Apr 15, 2016

@ask You say "python redis 2.10.5 supports the argument". Well, yes it supports it, but it does not support it for every class of connection possible. Celery (and probably Kombu too) just passes socket_connect_timeout to redis-py irrespective of the class of connection being created.

As I've pointed out in this comment redis.connection.UnixDomainSocketConnection in redis-py 2.10.5 does not support socket_connect_timeout. This is the init for that class:

class UnixDomainSocketConnection(Connection):
    description_format = "UnixDomainSocketConnection<path=%(path)s,db=%(db)s>"

    def __init__(self, path='', db=0, password=None,
                 socket_timeout=None, encoding='utf-8',
                 encoding_errors='strict', decode_responses=False,
                 retry_on_timeout=False,
                 parser_class=DefaultParser, socket_read_size=65536):

There is no socket_connect_timeout defined there.

raphaelm added a commit to pretix/kombu that referenced this issue May 9, 2016

ask added a commit to celery/kombu that referenced this issue May 26, 2016

@linpingta

This comment has been minimized.

linpingta commented Jun 24, 2016

For my case, it's related with redis version.

celery == 3.1.22
redis (python) == 2.9.1
redis-server == 2.8.17

Just upgrade redis (python) version from 2.9 to 2.10, it's ok now.

source code in connection.py is different from 2.9 to 2.10, it has socket_connect_timeout inside Connection in 2.10 while not in 2.9, please check the source code if interested.

Hope it helps.

@rockallite

This comment has been minimized.

rockallite commented Aug 24, 2016

If you use Celery 3.1.23 + redis-py 2.10.5 with Redis UNIX socket as the broker, you still get the exception TypeError: __init__() got an unexpected keyword argument 'socket_connect_timeout'. No easy fix yet, unless we monkey-patch Celery, kombu and/or redis-py library. For now, I can only give up using UNIX socket and use TCP connection with Redis instead.

@lenzls

This comment has been minimized.

lenzls commented Sep 1, 2016

the problem stil persists. I use a Redis UNIX socket as the broker.

celery==3.1.23
redis==2.10.5

Any suggestions?

@ghost

This comment has been minimized.

ghost commented Oct 7, 2016

This issue is still happening with latest versions:
redis server 3.2.3
celery 3.1.24
redis 2.10.5
kombu 3.0.37

The issue is the same mentioned before, in file celery/backends/redis.py, the _create_client method passes the socket_connect_timeout parameter no matter of the connection_class type.
Method _detect_client_capabilities doesn't consider the connection_class type neither, and UnixDomainSocketConnection still doesn't support that parameter.

Also, this error happens while saving the result for the task, so you won't see it if CELERY_IGNORE_RESULT is set to True.

@gremmie

This comment has been minimized.

gremmie commented Oct 13, 2016

The issue was fixed for me when I updated to kombu 3.0.36

mlevogiannis added a commit to mlevogiannis/demos-voting that referenced this issue Oct 23, 2016

settings.py,__init__.py: update celery configuration
- register 'custom-json' as kombu serializer
- set 'custom-json' as the default celery serializer
- fix the result backend configuration
- import settings from django.conf (it allows using different settings modules)
- celery 3.1.19 is NOT supported due to a bug in the redis backend (only affects UNIX-domain sockets, see: celery/celery#2903)
@lockie

This comment has been minimized.

lockie commented Oct 28, 2016

@gremmie actually, that's a downgrade, but yeah, that helps.

@sherzinger

This comment has been minimized.

sherzinger commented Nov 22, 2016

The issue is not fixed for me.
redis (2.10.5)
celery (4.0.0)
kombu (4.0.0)

I use a Redis UNIX socket as the result backend. The broker is a rabbitmq server.

"Invalid Arguments for task test.add: __init__() got an unexpected keyword argument 'socket_connect_timeout'"

.> transport: amqp://guest:**@localhost:5672//
.> results: socket:///var/folders/sy/4v6z8kl93tz1h6y_wygzkh6m0000gn/T/tmplau5pf6x/redis.socket

@markwingerd

This comment has been minimized.

markwingerd commented Mar 2, 2017

I've been getting this same issue as well with redis 3.2.7 and celery 3.1.20

@markwingerd

This comment has been minimized.

markwingerd commented Mar 2, 2017

It looks like this error can be fixed by adding socket_connect_timeout=None to the __init__ method arguments in redis/connection.py. I found this file in /usr/local/lib/python2.7/dist-packages/redis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment