Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TaskManager: RabbitMQ outages requires a James restart #4081

Closed
Arsnael opened this issue Nov 24, 2020 · 1 comment
Closed

TaskManager: RabbitMQ outages requires a James restart #4081

Arsnael opened this issue Nov 24, 2020 · 1 comment
Assignees
Labels

Comments

@Arsnael
Copy link
Member

Arsnael commented Nov 24, 2020

Steps to reproduce:

  • Stop rabbitMQ
  • Interact for a while with RabbitMQ
  • Restart rabbit.

Observation: RabbitMQ does not automatically heal!

A James restartis being required.

2020-11-24 05:01:03.164 [error] <0.758.0> Channel error on connection <0.730.0> (192.168.0.5:55846 -> 192.168.0.3:5672, vhost: '/', user: 'guest'), channel 1:
operation basic.consume caused a channel exception not_found: no queue 'eventbus-b0ce1474-c6e5-4117-a611-2a62d1e11ec3' in vhost '/'
2020-11-24 05:01:03.192 [error] <0.854.0> Channel error on connection <0.730.0> (192.168.0.5:55846 -> 192.168.0.3:5672, vhost: '/', user: 'guest'), channel 14:
operation basic.consume caused a channel exception not_found: no queue 'terminationSubscriber6557b9f9-812c-42e8-9020-0b6646731398' in vhost '/'
2020-11-24 05:01:03.200 [error] <0.862.0> Channel error on connection <0.730.0> (192.168.0.5:55846 -> 192.168.0.3:5672, vhost: '/', user: 'guest'), channel 16:
operation basic.consume caused a channel exception not_found: no queue 'taskManagerCancelRequestsQueue89eddcec-542c-4237-b379-656a692af78e' in vhost '/'
2020-11-24 05:01:03.426 [info] <0.910.0> accepting AMQP connection <0.910.0> (192.168.0.5:55848 -> 192.168.0.3:5672)

This, is due to the use of exclusive queue, that do not gets re-created upon restoration of the connection.

@Arsnael Arsnael added the bug label Nov 24, 2020
@Arsnael Arsnael added this to the into-production-kanban milestone Nov 24, 2020
@Arsnael Arsnael changed the title RebbitMQ outages requires a James restart RabbitMQ outages requires a James restart Nov 24, 2020
@chibenwa chibenwa changed the title RabbitMQ outages requires a James restart TaskManager: RabbitMQ outages requires a James restart Nov 24, 2020
@chibenwa
Copy link
Member

Maybe we should have a "Recovery callback" should be implemented as part of SimpleConnectionPool.

We can use this callback to recreate the queues when we restore a session.

Work on this should be handled as a draft. QA should be performed on DEV before a merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants