Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kombu 4.3.0: Unrecoverable error: ValueError('file descriptor cannot be a negative integer (-1) #1006

Closed
petersanchez opened this issue Feb 12, 2019 · 7 comments · Fixed by #1011
Closed

Kombu 4.3.0: Unrecoverable error: ValueError('file descriptor cannot be a negative integer (-1) #1006

petersanchez opened this issue Feb 12, 2019 · 7 comments · Fixed by #1011
Labels
Component: Redis Broker Issue Type: Bug Report

Comments

@petersanchez
Copy link

petersanchez commented Feb 12, 2019 •
edited
Loading

4.3.0 seems to have broken our Celery install. Downgrading to 4.2.2-post1 resolves it.

Redis backend

redis==2.10.6
Django==2.1.5
celery==4.2.1

OS: FreeBSD 12 p3.

Strange thing is, on my Ubuntu 18.04 laptop, the same requirements, and it works as expected.

I also completely flushed the redis db to ensure it wasn't some stuck task or odd issue causing it. Same thing. Only downgrading resolved it.

[2019-02-11 15:28:19,386: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-02-11 15:28:19,389: DEBUG/MainProcess] | Worker: Building graph...
[2019-02-11 15:28:19,390: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2019-02-11 15:28:19,411: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-02-11 15:28:19,411: DEBUG/MainProcess] | Consumer: Building graph...
[2019-02-11 15:28:19,429: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2019-02-11 15:28:19,459: DEBUG/MainProcess] | Worker: Starting Hub
[2019-02-11 15:28:19,459: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,459: DEBUG/MainProcess] | Worker: Starting Pool
[2019-02-11 15:28:19,619: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,619: DEBUG/MainProcess] | Worker: Starting Beat
[2019-02-11 15:28:19,623: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,624: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-02-11 15:28:19,625: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-02-11 15:28:19,642: INFO/MainProcess] Connected to redis://localhost:6379/0
[2019-02-11 15:28:19,643: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,643: DEBUG/MainProcess] | Consumer: Starting Events
[2019-02-11 15:28:19,659: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,659: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-02-11 15:28:19,659: INFO/MainProcess] mingle: searching for neighbors
[2019-02-11 15:28:19,677: CRITICAL/MainProcess] Unrecoverable error: ValueError('file descriptor cannot be a negative integer (-1)')
Traceback (most recent call last):
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 40, in start
    self.sync(c)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
    replies = self.send_hello(c)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 143, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 95, in _request
    timeout=self.timeout, reply=True,
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 454, in broadcast
    limit, callback, channel=channel,
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer, retry=True,
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 248, in producer_or_acquire
    yield producer
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/messaging.py", line 239, in __exit__
    self.release()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
    self.release(R)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pools.py", line 74, in release
    resource.__connection__.release()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
    self.release(R)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 134, in release
    self.close_resource(resource)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 950, in close_resource
    resource._close()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
    self._do_close_self()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 328, in _do_close_self
    self.maybe_close_channel(self._default_channel)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 320, in maybe_close_channel
    channel.close()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
    self._disconnect_pools()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
    async_pool.disconnect()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
    connection.disconnect()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
    channel._on_connection_disconnect(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
    self.connection.cycle._on_connection_disconnect(connection)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
    self.poller.unregister(connection._sock)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
    self._quick_unregister(fd)
ValueError: file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:19,690: DEBUG/MainProcess] | Worker: Closing Hub...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Pool...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Beat...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Consumer...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Stopping Beat...
[2019-02-11 15:28:19,692: INFO/MainProcess] beat: Shutting down...
[2019-02-11 15:28:19,692: DEBUG/MainProcess] | Worker: Stopping Pool...
[2019-02-11 15:28:20,726: DEBUG/MainProcess] | Worker: Stopping Hub...
[2019-02-11 15:28:20,726: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2019-02-11 15:28:20,728: WARNING/MainProcess] Traceback (most recent call last):
[2019-02-11 15:28:20,728: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
[2019-02-11 15:28:20,728: WARNING/MainProcess] self.blueprint.start(self)
[2019-02-11 15:28:20,729: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
[2019-02-11 15:28:20,729: WARNING/MainProcess] step.start(parent)
[2019-02-11 15:28:20,729: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
[2019-02-11 15:28:20,730: WARNING/MainProcess] return self.obj.start()
[2019-02-11 15:28:20,730: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
[2019-02-11 15:28:20,730: WARNING/MainProcess] blueprint.start(self)
[2019-02-11 15:28:20,730: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
[2019-02-11 15:28:20,731: WARNING/MainProcess] step.start(parent)
[2019-02-11 15:28:20,731: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 40, in start
[2019-02-11 15:28:20,731: WARNING/MainProcess] self.sync(c)
[2019-02-11 15:28:20,731: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
[2019-02-11 15:28:20,731: WARNING/MainProcess] replies = self.send_hello(c)
[2019-02-11 15:28:20,732: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
[2019-02-11 15:28:20,732: WARNING/MainProcess] replies = inspect.hello(c.hostname, our_revoked._data) or {}
[2019-02-11 15:28:20,732: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 143, in hello
[2019-02-11 15:28:20,732: WARNING/MainProcess] return self._request('hello', from_node=from_node, revoked=revoked)
[2019-02-11 15:28:20,733: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 95, in _request
[2019-02-11 15:28:20,733: WARNING/MainProcess] timeout=self.timeout, reply=True,
[2019-02-11 15:28:20,733: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 454, in broadcast
[2019-02-11 15:28:20,733: WARNING/MainProcess] limit, callback, channel=channel,
[2019-02-11 15:28:20,734: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
[2019-02-11 15:28:20,734: WARNING/MainProcess] serializer=serializer)
[2019-02-11 15:28:20,734: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 290, in _publish
[2019-02-11 15:28:20,734: WARNING/MainProcess] serializer=serializer, retry=True,
[2019-02-11 15:28:20,735: WARNING/MainProcess] File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
[2019-02-11 15:28:20,735: WARNING/MainProcess] next(self.gen)
[2019-02-11 15:28:20,735: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 248, in producer_or_acquire
[2019-02-11 15:28:20,735: WARNING/MainProcess] yield producer
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/messaging.py", line 239, in __exit__
[2019-02-11 15:28:20,736: WARNING/MainProcess] self.release()
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
[2019-02-11 15:28:20,736: WARNING/MainProcess] self.release(R)
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pools.py", line 74, in release
[2019-02-11 15:28:20,737: WARNING/MainProcess] resource.__connection__.release()
[2019-02-11 15:28:20,737: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
[2019-02-11 15:28:20,737: WARNING/MainProcess] self.release(R)
[2019-02-11 15:28:20,737: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 134, in release
[2019-02-11 15:28:20,738: WARNING/MainProcess] self.close_resource(resource)
[2019-02-11 15:28:20,738: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 950, in close_resource
[2019-02-11 15:28:20,738: WARNING/MainProcess] resource._close()
[2019-02-11 15:28:20,738: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
[2019-02-11 15:28:20,739: WARNING/MainProcess] self._do_close_self()
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 328, in _do_close_self
[2019-02-11 15:28:20,739: WARNING/MainProcess] self.maybe_close_channel(self._default_channel)
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 320, in maybe_close_channel
[2019-02-11 15:28:20,739: WARNING/MainProcess] channel.close()
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
[2019-02-11 15:28:20,740: WARNING/MainProcess] self._disconnect_pools()
[2019-02-11 15:28:20,740: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
[2019-02-11 15:28:20,740: WARNING/MainProcess] async_pool.disconnect()
[2019-02-11 15:28:20,740: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
[2019-02-11 15:28:20,741: WARNING/MainProcess] connection.disconnect()
[2019-02-11 15:28:20,741: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
[2019-02-11 15:28:20,741: WARNING/MainProcess] channel._on_connection_disconnect(self)
[2019-02-11 15:28:20,741: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
[2019-02-11 15:28:20,742: WARNING/MainProcess] self.connection.cycle._on_connection_disconnect(connection)
[2019-02-11 15:28:20,742: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
[2019-02-11 15:28:20,742: WARNING/MainProcess] self.poller.unregister(connection._sock)
[2019-02-11 15:28:20,742: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
[2019-02-11 15:28:20,743: WARNING/MainProcess] self._quick_unregister(fd)
[2019-02-11 15:28:20,743: WARNING/MainProcess] ValueError
[2019-02-11 15:28:20,743: WARNING/MainProcess] :
[2019-02-11 15:28:20,743: WARNING/MainProcess] file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:20,743: WARNING/MainProcess] During handling of the above exception, another exception occurred:
[2019-02-11 15:28:20,743: WARNING/MainProcess] Traceback (most recent call last):
[2019-02-11 15:28:20,743: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/bin/celery", line 11, in <module>
[2019-02-11 15:28:20,744: WARNING/MainProcess] sys.exit(main())
[2019-02-11 15:28:20,744: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
[2019-02-11 15:28:20,745: WARNING/MainProcess] _main()
[2019-02-11 15:28:20,745: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
[2019-02-11 15:28:20,757: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2019-02-11 15:28:20,757: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
[2019-02-11 15:28:20,758: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2019-02-11 15:28:20,758: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
[2019-02-11 15:28:20,766: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2019-02-11 15:28:20,766: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
[2019-02-11 15:28:20,767: WARNING/MainProcess] return self.execute(command, argv)
[2019-02-11 15:28:20,767: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
[2019-02-11 15:28:20,767: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2019-02-11 15:28:20,767: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
[2019-02-11 15:28:20,769: WARNING/MainProcess] return self(*args, **options)
[2019-02-11 15:28:20,769: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/base.py", line 238, in __call__
[2019-02-11 15:28:20,769: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2019-02-11 15:28:20,769: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
[2019-02-11 15:28:20,770: WARNING/MainProcess] worker.start()
[2019-02-11 15:28:20,770: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 210, in start
[2019-02-11 15:28:20,770: WARNING/MainProcess] self.stop(exitcode=EX_FAILURE)
[2019-02-11 15:28:20,771: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 253, in stop
[2019-02-11 15:28:20,771: WARNING/MainProcess] self._shutdown(warm=True)
[2019-02-11 15:28:20,771: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 268, in _shutdown
[2019-02-11 15:28:20,771: WARNING/MainProcess] self.blueprint.stop(self, terminate=not warm)
[2019-02-11 15:28:20,772: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 177, in stop
[2019-02-11 15:28:20,772: WARNING/MainProcess] self.on_stopped()
[2019-02-11 15:28:20,772: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 164, in on_stopped
[2019-02-11 15:28:20,773: WARNING/MainProcess] self.consumer.shutdown()
[2019-02-11 15:28:20,773: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 352, in shutdown
[2019-02-11 15:28:20,773: WARNING/MainProcess] self.blueprint.shutdown(self)
[2019-02-11 15:28:20,773: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 159, in shutdown
[2019-02-11 15:28:20,774: WARNING/MainProcess] self.send_all(parent, 'shutdown')
[2019-02-11 15:28:20,774: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 151, in send_all
[2019-02-11 15:28:20,774: WARNING/MainProcess] fun(parent, *args)
[2019-02-11 15:28:20,774: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/events.py", line 69, in shutdown
[2019-02-11 15:28:20,794: WARNING/MainProcess] self._close(c)
[2019-02-11 15:28:20,794: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/events.py", line 63, in _close
[2019-02-11 15:28:20,794: WARNING/MainProcess] ignore_errors(c, dispatcher.connection.close)
[2019-02-11 15:28:20,794: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/common.py", line 295, in ignore_errors
[2019-02-11 15:28:20,796: WARNING/MainProcess] return fun(*args, **kwargs)
[2019-02-11 15:28:20,796: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 372, in release
[2019-02-11 15:28:20,796: WARNING/MainProcess] self._close()
[2019-02-11 15:28:20,796: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
[2019-02-11 15:28:20,797: WARNING/MainProcess] self._do_close_self()
[2019-02-11 15:28:20,797: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 331, in _do_close_self
[2019-02-11 15:28:20,797: WARNING/MainProcess] self.transport.close_connection(self._connection)
[2019-02-11 15:28:20,797: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 953, in close_connection
[2019-02-11 15:28:20,799: WARNING/MainProcess] channel.close()
[2019-02-11 15:28:20,799: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
[2019-02-11 15:28:20,799: WARNING/MainProcess] self._disconnect_pools()
[2019-02-11 15:28:20,800: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
[2019-02-11 15:28:20,800: WARNING/MainProcess] async_pool.disconnect()
[2019-02-11 15:28:20,800: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
[2019-02-11 15:28:20,801: WARNING/MainProcess] connection.disconnect()
[2019-02-11 15:28:20,801: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
[2019-02-11 15:28:20,802: WARNING/MainProcess] channel._on_connection_disconnect(self)
[2019-02-11 15:28:20,802: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
[2019-02-11 15:28:20,803: WARNING/MainProcess] self.connection.cycle._on_connection_disconnect(connection)
[2019-02-11 15:28:20,803: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
[2019-02-11 15:28:20,803: WARNING/MainProcess] self.poller.unregister(connection._sock)
[2019-02-11 15:28:20,803: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
[2019-02-11 15:28:20,804: WARNING/MainProcess] self._quick_unregister(fd)
[2019-02-11 15:28:20,804: WARNING/MainProcess] ValueError
[2019-02-11 15:28:20,804: WARNING/MainProcess] :
[2019-02-11 15:28:20,804: WARNING/MainProcess] file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:20,804: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
 
 -------------- celery@dev01 v4.2.1 (windowlicker)
---- **** ----- 
--- * ***  * -- FreeBSD-12.0-RELEASE-p3-amd64-64bit-ELF 2019-02-11 15:28:19
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         grinch:0x802663c18
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . comments.tasks.process_comment_mentions
  . grinch.tasks.log_invoice_action
  . grinch.tasks.run_maintenance_tasks
  . grinch.tasks.run_orgs_tasks
  . grinch.tasks.send_activation_reminders
  . grinch.tasks.send_email_broadcast
  . grinch.tasks.send_invoice_client_submit_email
  . grinch.tasks.send_simple_email
  . grinch.tasks.send_status_update_csv_email
  . grinch.tasks.update_files_permissions
  . payments.tasks.process_payments
  . payments.tasks.send_client_payment_invoice_email
  . payments.tasks.send_invoice_payment_refund_email
  . payments.tasks.send_org_payment_invoice_email
  . web.core.celery.debug_task
@thedrow thedrow mentioned this issue Feb 13, 2019
Closed
@georgepsarakis
Copy link
Contributor

@petersanchez could you please try upgrading the redis client library? Could this error be related to the issue described here redis/redis-py@4e1e748 ?

@petersanchez
Copy link
Author

Same issue with redis==3.1.0 and Kombu==4.3.0.

Downgraded Kombu==4.2.2-post1, keeping the updated redis, and it works as expected.

@georgepsarakis
Copy link
Contributor

I found this change that possibly explains the issue.

For reference, all changes between 4.2.2-post1 and 4.3.0: v4.2.2-post1...v4.3.0

@emord emord mentioned this issue Feb 18, 2019
Closed
lithammer added a commit to lithammer/kombu that referenced this issue Feb 21, 2019
@lithammer
Revert celery#954 and update redis-py to at least 3.2.0
706f9f0 
This reverts celery#954, and bumps the required redis-py dependency to 3.2.0
to include this fix:

redis/redis-py@4e1e748

Fixes celery#1006
lithammer added a commit to lithammer/kombu that referenced this issue Feb 21, 2019
@lithammer
Port improved connection disconnect from redis-py 3.2.0
43c897c 
Port the fixes of `Connection.disconnect` from upstream redis-py v3.2.0:

redis/redis-py@4e1e748

Fixes celery#1006
This was referenced Feb 21, 2019
Merged
Closed
@lithammer
Copy link
Contributor

I come bearing two solution proposals for this, #1011 and #1012.

auvipy pushed a commit that referenced this issue Feb 22, 2019
@lithammer @auvipy
Revert #954 and update redis-py to at least 3.2.0 (#1011)
eb6e4c8 
This reverts #954, and bumps the required redis-py dependency to 3.2.0
to include this fix:

redis/redis-py@4e1e748

Fixes #1006
@trunet
Copy link

trunet commented Feb 27, 2019

is there any ETA for 4.3.1 with this fix?

@trunet
Copy link

trunet commented Mar 4, 2019

replying to myself: 4.4.0 is released and it fixed this issue for me. Thanks!

@petersanchez
Copy link
Author

Confirmed 4.4.0, redis==3.2.0 fixes this for me as well.

@thedrow thedrow mentioned this issue Mar 12, 2019
Closed
This was referenced Mar 12, 2019
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Redis Broker Issue Type: Bug Report
Projects
None yet
Milestone
No milestone
5 participants
@thedrow @lithammer @petersanchez @trunet @georgepsarakis