Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: RabbitMQ - re-register message consumer? #329

Closed
ml-eds opened this issue Apr 25, 2019 · 10 comments

Comments

@ml-eds
Copy link

commented Apr 25, 2019

Hello,

thanks a lot for creating this great software.

I have following scenario:

  • RabbitMQ with several microservices
  • Microservice with CAP starts before RabbitMQ is ready
  • When RabbitMQ comes up a few seconds later, then publishing events works correctly, but subscription (call of methods annotated with CapSubscribe) to messages does not work
  • After restart of microservice both publishing and subscribe work

My guess is that CAP does not try to re-register message consumers in case of unavailable and later available again RabbitMQ? Is there any way to manually trigger re-registering of message consumers in mircoservice?

This only seems to happen during startup of CAP empowered microservice and non-ready RabbitMQ. When RabbitMQ is available during microservice start everything works fine. I then also killed and restarted RabbitMQ, which also results in (still) working publishing and consuming of messages by microservice.

This might be a non-issue if CAP users are expected to guarantee availability of RabbitMQ before CAP microservices start.

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

Hello @ml-eds

CAP will register the consumer to RabbitMQ at the startup, and if can't find the RabbitMQ at startup, the registration queue to RabbitMQ will faild.

By default, the consumer queue registered by CAP to RabbitMQ is persistent, so I recommend that you start the microservice once before it is ready used to production, which will guarantees the message queue with persistence in RabbitMQ and the message will not be lost.

@ml-eds

This comment has been minimized.

Copy link
Author

commented Apr 25, 2019

Hello,

thanks for your really quick answer. In my scenario the persistent queue had already been created before via microservice in RabbitMQ. Then I turn off RabbitMQ and microservice. Then I start microservice again and RabbitMQ with a delay so that it is available after the microservice. Publishing then works, consuming messages not.

I think this is not an entirely impossible scenario, when one assumes flapping availability of microservice and RabbitMQ.

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

Hello, I just tested the scene you described and it works fine.

    1. Start microservices.
    1. Close RabbitMQ.
    1. Send a message, the message will save to database table and status will be Failed.
    1. Start RabbitMQ.
    1. 5-1 The CAP retry process will resend the message from message table to RabbitMQ.
    1. 5-2 RabbitMQ will auto reconnect and start the registered consumer (This seems to be done internally by the RabbitMQ client driver).
    1. The consumer received the message.

The CAP retry process will retry the message sent 4 minutes ago, so you can adjust the send time of the failed message forward for a period of time in order to trigger the retry in time.

@ml-eds

This comment has been minimized.

Copy link
Author

commented Apr 25, 2019

Hello,

thanks again for your quick reply. My steps are

  • i. Start RabbitMQ
  • ii. Start microservices, ensure persistent queue is created
  • iii. Close RabbitMQ and microservices
  • iv. Start microservices without starting RabbitMQ
  • v. AFTER all microservices have started: Start RabbitMQ
  • vi. Send a message, publish succeeds (entry in database has succeeded status and I can see queued message in RabbitMQ management console)
  • vii. Message is not received by consumer, even after waiting for a couple of minutes
  • viii. Restart microservice which consumes messages: Message is received

So step vii is the problem and viii seems to be the only workaround, as far as I can tell.

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@ml-eds
I reproduce the problem you described , which is a Bug for CAP, I think CAP needs to be aware that RabbitMQ is available and re-register the consumer, I will try to fix it, thanks very much for your report

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 26, 2019

Hello @ml-eds

I have fixed this bug in 2.5.1-preview-73031417, if broker is unreachable, we will try to re-register consumer to broker every 30 seconds , you can get it from NuGet and test it.

Thanks

@ml-eds

This comment has been minimized.

Copy link
Author

commented Apr 26, 2019

Hello,

thank your very much for this fast update!

I updated to 2.5.1-preview-73031417 and re-tested several times. I am sorry to say - but behavior is exactly the same as before. Re-registering after 30 seconds does not happen for RabbitMQ.

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 26, 2019

Hello,

My test is working fine. I recorded a video(https://youtu.be/9nJDMar1ABU) showing my test process.

What have I missed?

@ml-eds

This comment has been minimized.

Copy link
Author

commented Apr 26, 2019

Hello,

thanks for digging into it. At about 1:10-1:50 of the video you reproduce the steps iii., iv. and v. of my comment above (#329 (comment)) and it works in your setup. I can clearly see the consumer re-registers.

In my setup each service runs in a Docker container on macOS. Repeating steps iii., iv. and v. does not result in re-registering here. Perhaps there is a Docker networking issue interfering. Any idea on how to get more information what is going wrong here? Is it possible to increase log-level for CAP when using a nuget package or do I have to build the project myself to do so?

I will re-test again in a week and give feedback again.

@ml-eds

This comment has been minimized.

Copy link
Author

commented May 2, 2019

I re-tested, here is my feedback:

It does not work when RabbitMQ and subscribing/publishing service are both run as Docker containers configured via docker-compose.yml and when both are started at the same time via docker-compose up. If RabbitMQ comes up after the service, re-registering does not happen.

If I start each container manually, one after the other, re-registering works. My guess is that this is caused by a bug in communicating network state changes between containers. Thus it is a Docker issue. (Tested with Version Docker Desktop for Mac version 2.0.0.3 (31259), Docker Engine: 18.09.2).

Regarding CAP I can confirm that it works. Thank you very much for your immediate response and super-good support!

@ml-eds ml-eds closed this May 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.