RabbitMQ transport is sending events too slow

[tomkregenbild]

I encountered a problem that when I deliver large amounts of log files from one server to RabbitMQ, I don't get all the events and in some cases I have to restart Beaver in order for it to start sending the events again.

After playing with the run_queue.py file I managed to print the number transports to RabbitMQ vs the number of events in the internal Queue of beaver and I see that the internal queue is feeling up pretty quickly with a ratio of about 1:4 of values entering the Queue and values transported to RabbitMQ.

Is there a way to improve the performance here? maybe increase the number of thread dealing with the transport?

[josegonzalez]

Maybe don't use a blocking connection?

[tomkregenbild]

@josegonzalez Sounds like a good idea - I'll try to implement it

[timstoop]

@tomkregenbild Did this solve your problem? I'd like to know if I should test your changes as well, since we're more or less running into this as well.

[tomkregenbild]

@timstoop adding more cosumer processes did help us a lot

the parameter is "number_of_consumer_processes" - also running with --debug will allow you to see the queue sizes and number of transported events so you'll be able to analyze your problem.

[timstoop]

@tomkregenbild You did not get a benefit from no longer using Blocking Connections? Just asking, since this exact problem is also discussed in #265, hence I thought it might make a large difference.

[tomkregenbild]

@timstoop from what I saw it was a little better but nothing too extraordinary.

[JamieCressey]

Duplicate of #265