-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#323 RabbitMQ transport is sending events too slow #324
Conversation
…er transports in debug mode in order to find problem in transport rate.
…onsumers by creating additional processes.
@@ -53,7 +53,8 @@ def __init__(self, beaver_config, queue_consumer_function, callback, logger=None | |||
self._folder = self._beaver_config.get('path') | |||
self._last_file_mapping_update = {} | |||
self._logger = logger | |||
self._proc = None | |||
self._number_of_consumer_processes = int(self._beaver_config.get('number_of_consumer_processes')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All this logic should also be set in the tail_manager and tail module.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@josegonzalez
Thanks, missed it.
I added it to worker/tail_manager.py but I don't see anything related in worker/tail.py. Can you focus on where should I add the change in worker/tail.py?
…onsumers by creating additional processes while running with --experimental flag
@josegonzalez I verified that the changes in the last commit are working well on my machine also with the --experimental flag but I see that the "Travis CI Build" is failing with "No output has been received in the last 10 minutes" - Is there a problem in the code or it is a problem with the test? |
…nnection instead of BlockingConnection for better performance.
1. Add clear debug prints with queue size (one print every 1000 items in order not to hurt performance) 2. If main queue is empty keep running and do nothing 3. In case of a timeout from main queue restart queue
…function to print connection errors.
#323 RabbitMQ transport is sending events too slow
Fixes #265 |
Fixes #264 |
This fix includes two part:
Please let me know if you have any questions or remarks - from the tests I ran in my environment using the rabbitMQ trasport (sends around 6500 log entries per second), increasing the number of consumers really helped.