New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError | 'NoneType' object has no attribute 'callback' #1089

Closed
AndreCimander opened this Issue Feb 2, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@AndreCimander

AndreCimander commented Feb 2, 2018

  • gevent version: 1.3a1
  • Python version: Python 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609] on linux2
  • Operating System: Ubuntu 16.04.3 LTS

Description:

Might be a fault on Celery's or redis-py's end, I reverted back to 1.2.2 and everything runs "smooth" again (as far as you can say Celery runs smoothly - we are SO looking forward to the replacement system).

AttributeError: 'NoneType' object has no attribute 'callback'
  File "celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "core/celery/monitoring/tasks.py", line 24, in celery_worker_heartbeat_crawler_instagram_daily
    CeleryMonitoring.set_heartbeat_for_queue("crawler_instagram_daily")
  File "core/celery/monitoring/utils.py", line 47, in set_heartbeat_for_queue
    cls(queue).set_heartbeat()
  File "core/celery/monitoring/utils.py", line 26, in set_heartbeat
    cache.set(self.cache_key, timezone.now(), self.cache_max_ttl)
  File "django_redis/cache.py", line 33, in _decorator
    return method(self, *args, **kwargs)
  File "django_redis/cache.py", line 68, in set
    return self.client.set(*args, **kwargs)
  File "django_redis/client/default.py", line 141, in set
    return client.set(nkey, nvalue, nx=nx, px=timeout, xx=xx)
  File "redis/client.py", line 1172, in set
    return self.execute_command('SET', *pieces)
  File "redis/client.py", line 666, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "redis/connection.py", line 1000, in get_connection
    connection.disconnect()
  File "redis/connection.py", line 579, in disconnect
    self._sock.shutdown(socket.SHUT_RDWR)
  File "gevent/_socket2.py", line 491, in shutdown
    self.hub.cancel_wait(self._read_event, cancel_wait_ex)
  File "gevent/hub.py", line 678, in cancel_wait
    if watcher.callback is not None:

Worker start up command: worker -c 4 --without-gossip --without-mingle -Ofair -P gevent

@AndreCimander AndreCimander changed the title from AttributeError to AttributeError | 'NoneType' object has no attribute 'callback' Feb 2, 2018

@jamadden

This comment has been minimized.

Member

jamadden commented Feb 2, 2018

Thanks for your report. I believe this happens when a closed socket is then shutdown. I'm trying to find the code-path in the redis client that can get in that state in order to confirm, but I'm having trouble matching line numbers up. Can you please tell me what version of the redis client you're using?

@jamadden jamadden closed this in 24e7e81 Feb 2, 2018

@jamadden

This comment has been minimized.

Member

jamadden commented Feb 2, 2018

I was able to confirm my hypothesis and make a fix, but I'm still curious about how the redis connection got in that state and would like to examine the version you're using.

@AndreCimander

This comment has been minimized.

AndreCimander commented Feb 6, 2018

Hey there,

sorry for the delayed answer, one single point of failure decided to go south and it was a little ... hectic so to say.

We are currently using a forked redis-py version (smartfile/redis-py@f934896) that contains the following PR andymccurdy/redis-py#886, so this could the culprit.

Celery with Py2.7, RabbitMQ and threaded workers is quite a ride :-)

@jamadden

This comment has been minimized.

Member

jamadden commented Feb 6, 2018

Thanks for the info! That patch could explain it, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment